FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models

In one sentence

Summary

The paper addresses the challenge of applying large language models to complex financial domains, where dense numerical data, domain-specific jargon, and visual elements like charts require robust multimodal understanding. Standard LLMs often struggle with financial reasoning and are prone to hallucinations, limiting their reliability for decision-making. The authors propose FinTral, a family of models built on Mistral-7B, designed to integrate textual, numerical, tabular, and image data. The core innovation lies in a comprehensive training pipeline that includes domain-specific pretraining on a curated 20-billion-token dataset, instruction fine-tuning, and alignment via Direct Preference Optimization (DPO) using AI feedback. Additionally, the model is enhanced with vision capabilities via a CLIP encoder and equipped with external tools and Retrieval-Augmented Generation (RAG) to handle quantitative tasks and out-of-domain queries effectively. The experimental framework centers on FinSet, a novel benchmark comprising nine tasks across 25 datasets, including sentiment analysis, named entity recognition, number understanding, stock movement prediction, credit scoring, and a unique financial hallucination analysis. The training data includes a 20-billion-token pretraining corpus derived from SEC filings, news, and social media, alongside instruction tuning data and AI-generated preference pairs. The evaluation compares FinTral variants against baselines like ChatGPT-3.5 and GPT-4. Key results show that FinTral-DPO-T&R, which combines DPO alignment with tools and retrieval, outperforms ChatGPT-3.5 in all tasks and surpasses GPT-4 in five out of nine text-based tasks. The model also demonstrates strong performance in chart understanding and significantly reduces hallucinations, achieving a 97% hallucination index compared to GPT-4's 98%. The findings highlight that integrating retrieval and tool-use mechanisms is critical for financial LLMs to achieve high accuracy in numerical and factual tasks. FinTral shows potential for real-time financial analysis, document summarization, and risk assessment. However, the paper acknowledges limitations, including the model's domain-specific nature, which may limit generalizability outside finance. The reliance on static pretraining data means the model may not capture real-time market dynamics without continuous updates. Furthermore, the computational cost of training and the potential for bias in public data sources are noted. The work provides a valuable resource for the community through the open-source release of the FinTral models and the FinSet benchmark, facilitating further research into multimodal financial AI and hallucination mitigation.

Research problem

Financial large language models face significant challenges due to the complexity of financial documents, which contain dense numerical information, domain-specific jargon, and visual elements like charts. Standard LLMs often lack the necessary domain knowledge and are prone to hallucinations, reducing their reliability for financial decision-making. The paper addresses the need for a multimodal model that can accurately process text, numbers, tables, and images while minimizing errors in financial reasoning.

Core contributions

• Introduction of FinTral, a multimodal financial LLM based on Mistral-7B, capable of processing text, numerical, tabular, and visual data.
• Development of FinSet, the largest financial LLM benchmark featuring nine tasks and 25 datasets, including the first evaluation of financial hallucinations.
• A comprehensive training pipeline involving domain-specific pretraining, instruction fine-tuning, and alignment via Direct Preference Optimization (DPO) using AI feedback.
• Enhancement of FinTral with vision capabilities (FinTral-VL) and external tools/retrieval (FinTral-DPO-T&R) to improve performance on quantitative and complex tasks.
• Demonstration that FinTral-DPO-T&R outperforms ChatGPT-3.5 in all tasks and surpasses GPT-4 in five out of nine text-based tasks.

Method

• Base model: Mistral-7B-v0.1, selected for its BPE tokenizer suitable for numerical tasks.
• Domain-specific pretraining: 20 billion tokens of financial data using Flash Attention 2 and LoRA.
• Instruction fine-tuning: QLoRA applied to linear layers for efficient fine-tuning on 226k instructions.
• Alignment: Direct Preference Optimization (DPO) using AI feedback generated by GPT-4.
• Multimodal integration: CLIP vision encoder and 2-layer MLP visual abstractor for image processing.
• Tool usage: Integration of Python functions (Add, Subtract, Multiply) for quantitative reasoning.
• Retrieval-Augmented Generation (RAG): BGE models used for retrieving relevant financial documents.

Results & findings

• FinTral-DPO-T&R outperforms ChatGPT-3.5 in all nine evaluated tasks.
• FinTral-DPO-T&R surpasses GPT-4 in five out of nine text-based tasks, including sentiment analysis and named entity recognition.
• The model achieves a 97% hallucination index on financial term definitions, close to GPT-4's 98%.
• FinTral-VL performs competitively on chart understanding tasks, with a score of 0.75 on FinVQA.
• Integration of tools and retrieval significantly boosts performance on numerical understanding and firm disclosure tasks.
• Human evaluation confirms FinTral-DPO-T&R produces more correct and satisfying responses than ChatGPT on financial term definitions.

Data & experiments

• Pretraining dataset: 20 billion tokens from C4, news, SEC filings, and social media, cleaned and deduplicated.
• Instruction tuning data: 226,300 financial instructions from various sources, including GPT-3.5/4 interactions.
• AI feedback data: 43,000 preference pairs generated by comparing GPT-4 outputs with FinMA-7B and LLaMA-7B outputs.
• Visual data: Multimodal instruction datasets including ChartQA, FinVis-PT, and FinVis-IT for chart understanding.
• Evaluation benchmark: FinSet with nine tasks (sentiment, NER, number understanding, summarization, stock prediction, credit scoring, firm disclosure, hallucination) across 25 datasets.
• Baselines: Comparison against Llama-2, Mistral, FinMA, Vicuna, ChatGPT-3.5, and GPT-4.

Limitations & caveats

• Domain-specific adaptation: FinTral may not perform well outside the financial domain due to its specialized training.
• Real-time data dependency: Predictive accuracy relies on the timeliness of retrieved data, which may lag in rapidly changing markets.
• Maintenance requirements: Continuous updates are needed to keep the model relevant with evolving financial regulations and data.
• Computational cost: Training and inference require significant computational resources, impacting energy efficiency.
• Data bias: Publicly available data sources may contain biases that are inherited by the model.
• Hallucination reduction: While improved, the model still exhibits some hallucinations, as evidenced by the 97% hallucination index.

Curator notes

Discovery status: round1_promoted_seed_for_round2. Source links: A Comprehensive Review of Generative AI in Finance; A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges; Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges; BloombergGPT: A Large Language Model for Finance; ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering; DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning; Deficiency