The List / Multimodal and Multilingual Finance / paper
Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications
Abstract
Financial LLMs hold promise for advancing financial tasks and domain-specific applications. However, they are limited by scarce corpora, weak multimodal capabilities, and narrow evaluations, making them less suited for real-world application. To address this, we introduce \textit{Open-FinLLMs}, the first open-source multimodal financial LLMs designed to handle diverse tasks across text, tabular, time-series, and chart data, excelling in zero-shot, few-shot, and fine-tuning settings. The suite includes FinLLaMA, pre-trained on a comprehensive 52-billion-token corpus; FinLLaMA-Instruct, fine-tuned with 573K financial instructions; and FinLLaVA, enhanced with 1.43M multimodal tuning pairs for strong cross-modal reasoning. We comprehensively evaluate Open-FinLLMs across 14 financial tasks, 30 datasets, and 4 multimodal tasks in zero-shot, few-shot, and supervised fine-tuning settings, introducing two new multimodal evaluation datasets. Our results show that Open-FinLLMs outperforms afvanced financial and general LLMs such as GPT-4, across financial NLP, decision-making, and multi-modal tasks, highlighting their potential to tackle real-world challenges. To foster innovation and collaboration across academia and industry, we release all codes (https://anonymous.4open.science/r/PIXIU2-0D70/B1D7/LICENSE) and models under OSI-approved licenses.
In one sentence
Summary
The paper addresses the limitations of existing financial LLMs, which suffer from scarce corpora, weak multimodal capabilities, and narrow evaluations. The authors introduce Open-FinLLMs, a suite of models designed to handle diverse financial data types including text, tabular, time-series, and chart data. The suite comprises FinLLaMA, a foundational model pre-trained on a 52-billion-token corpus; FinLLaMA-Instruct, fine-tuned with 573K financial instructions; and FinLLaVA, enhanced with 1.43M multimodal tuning pairs for cross-modal reasoning. This approach aims to bridge the gap between textual and structured financial data, enabling more comprehensive financial knowledge capture. The methodology involves a structured training pipeline. FinLLaMA is built on LLaMA3-8B and pre-trained on a mix of financial papers, conference calls, reports, indicators, news, historical data, and SEC filings, mixed with general domain data to prevent catastrophic forgetting. FinLLaMA-Instruct uses parameter-efficient fine-tuning with LoRA on a curated instruction dataset. FinLLaVA employs a two-stage multimodal alignment and supervised fine-tuning process using CLIP and LLaVA-1.5 frameworks, incorporating chart and tabular image-text pairs. Evaluations cover 14 financial tasks across 30 datasets and 4 multimodal tasks in zero-shot, few-shot, and fine-tuning settings. Results demonstrate that Open-FinLLMs outperform advanced financial and general LLMs, including GPT-4, across financial NLP, decision-making, and multimodal tasks. FinLLaMA excels in zero-shot and few-shot settings, while FinLLaMA-Instruct surpasses GPT-4 on three key financial analysis tasks. FinLLaVA achieves state-of-the-art performance on tabular and chart benchmarks, outperforming commercial models like GPT-4o and Gemini-1.5-pro. The models show strong potential for real-world applications such as portfolio optimization, trend analysis, and financial reporting, though the paper notes that trading performance was evaluated using a specific agent framework (FinMem) rather than direct market execution.
Research problem
Existing financial LLMs are limited by scarce domain-specific corpora, weak multimodal capabilities (lacking support for tabular, time-series, and chart data), and narrow evaluations focused primarily on text-based NLP tasks. These limitations hinder their applicability to real-world financial challenges that require complex cross-modal reasoning and decision-making.
Core contributions
- Introduction of Open-FinLLMs, a suite of open-source multimodal financial LLMs including FinLLaMA, FinLLaMA-Instruct, and FinLLaVA.
- Development of a 52-billion-token pre-training corpus combining text, tabular, and time-series data from diverse high-quality financial sources.
- Creation of a 1.43M multimodal instruction dataset incorporating chart images, tabular layouts, and text for cross-modal reasoning.
- Comprehensive evaluation across 14 financial tasks, 30 datasets, and 4 multimodal tasks, introducing two new multimodal benchmarks (ChartBench and TableBench).
- Demonstration of superior performance over general LLMs (GPT-4) and specialized financial LLMs (BloombergGPT) in zero-shot, few-shot, and fine-tuning settings.
Method
- Continual pre-training of LLaMA3-8B on a 52B-token mixed corpus of financial and general data.
- Parameter-efficient instruction fine-tuning using LoRA (r=64, alpha=128) with INT4 quantization.
- Multimodal alignment using CLIP vision encoder and MLP projector to map visual features to LLM embedding space.
- Two-stage supervised fine-tuning for multimodal capabilities: alignment stage followed by SFT with LLaVA-1.5 framework.
- Evaluation using zero-shot, few-shot, and supervised fine-tuning protocols across diverse financial benchmarks.
- Trading performance assessment using the FinMem agent framework with memory modules for short, mid, and long-term information.
Results & findings
- FinLLaMA outperforms LLaMA3-8B, LLaMA3.1-8B, and BloombergGPT in zero-shot and few-shot financial NLP tasks.
- FinLLaMA-Instruct surpasses GPT-4 on three key financial analysis tasks: sentiment analysis, named entity recognition, and numerical understanding.
- FinLLaVA achieves state-of-the-art performance on TableBench, outperforming GPT-4o and Gemini-1.5-pro.
- FinLLaMA shows strong generalization in few-shot settings, achieving an F1 score of 82.10 on NER tasks.
- In trading decision-making, FinLLaMA achieves a Sharpe Ratio over 1 with lower volatility and drawdown compared to baselines.
- The model demonstrates robustness in handling multimodal financial data, particularly in interpreting complex charts and tables.
- FinLLaMA-Instruct outperforms Palmyra-Fin-70B-32K in 4 out of 6 domain-specific tasks despite being a smaller model.
Data & experiments
- Pre-training corpus: 52 billion tokens from financial papers, conference calls, reports, indicators, news, historical data, and SEC filings.
- Instruction tuning: 573K samples from FLUPE, finred, MathInstruct, and Sujet-Finance-Instruct.
- Multimodal data: 1.43M pairs including images, charts, and tabular layouts from sources like ALLaVA-4V, ChartQA, and SynthTabNet.
- Evaluation: 14 financial tasks across 30 datasets, including sentiment analysis, NER, credit scoring, and fraud detection.
- Multimodal benchmarks: ChartBench (350 instances) and TableBench (450 questions) introduced for this study.
- Trading evaluation: Single-asset trading using FinMem agent framework on OHLCV, news, and SEC filing data for COIN, GOOG, NIO, and TSLA.
Limitations & caveats
- Trading performance was evaluated using an agent framework (FinMem) rather than direct market execution, which may not reflect real-world slippage or liquidity constraints.
- The pre-training corpus excludes web data due to noise, potentially limiting exposure to some real-time financial information.
- The multimodal dataset relies on synthetic or filtered data from general sources, which may not fully capture the complexity of all financial charts and tables.
- Evaluation of trading performance is limited to a few stocks (COIN, GOOG, NIO, TSLA) and a specific time period.
- The paper does not discuss the computational costs or energy consumption of training such large multimodal models.
Curator notes
Discovery status: round1_promoted_seed_for_round2. Source links: A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges; Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges; BloombergGPT: A Large Language Model for Finance; ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering; Exploring Large Language Models for Financial Applications: Techniques, Performance, and Challenges with FinMA; Exploring the Synergy of Quantitative Factor