PIXIU: A Comprehensive Benchmark, Instruction Dataset and Large Language Model for Finance

In one sentence

Summary

The paper addresses the lack of open-source, instruction-following large language models tailored for the financial domain, where existing models like BloombergGPT are proprietary or lack instruction-tuning capabilities. The authors aim to democratize financial AI by creating a comprehensive framework that includes a domain-specific model, a large-scale instruction dataset, and a holistic evaluation benchmark. This positioning fills a critical gap in open-source research, enabling reproducible and transparent development of financial AI tools that can handle diverse tasks beyond simple text classification. The core contribution is the PIXIU framework, which comprises three main components: FinMA, an LLaMA-based LLM fine-tuned with multi-task instruction data; FIT, a 136K-sample instruction dataset covering five tasks (sentiment analysis, headline classification, NER, question answering, and stock movement prediction) across various data modalities including text, tables, and time-series; and FLARE, an evaluation benchmark integrating these tasks. The authors conduct extensive experiments comparing FinMA against proprietary models like GPT-4 and BloombergGPT, as well as other open-source LLMs, using zero-shot and few-shot evaluation protocols on the FLARE benchmark datasets. Results show that FinMA significantly outperforms competitors on most financial NLP tasks, such as sentiment analysis and headline classification, proving the value of domain-specific instruction tuning. However, FinMA struggles with complex quantitative reasoning in financial question answering and stock movement prediction, largely due to the backbone LLaMA's limitations in mathematics and the inherent difficulty of the prediction tasks. The study highlights that while instruction tuning improves general financial language understanding, it does not fully resolve challenges in numerical reasoning or market prediction, suggesting future work should focus on enhancing these specific capabilities.

Research problem

The financial domain lacks open-source, instruction-following large language models capable of handling diverse financial tasks. Existing solutions are either proprietary (e.g., BloombergGPT) or limited to small-scale pre-trained models without instruction-tuning capabilities. This gap hinders the open development of financial AI and the evaluation of LLMs on critical financial tasks like prediction and complex reasoning.

Core contributions

• Introduction of FinMA, the first open-source, instruction-following financial LLM based on LLaMA.
• Creation of FIT, a large-scale multi-task and multi-modal instruction tuning dataset with 136K samples.
• Development of FLARE, a comprehensive evaluation benchmark covering both financial NLP and stock movement prediction tasks.
• Extensive empirical comparison of FinMA against proprietary models (GPT-4, BloombergGPT) and other open-source LLMs.
• Open-sourcing of the model, dataset, benchmark, and code to facilitate future financial AI research.

Method

• Instruction construction: Domain experts wrote 10 diverse instructions per dataset to convert raw data into instruction-tuning samples.
• Model training: Fine-tuned LLaMA-7B and LLaMA-30B using AdamW optimizer with specific learning rates and batch sizes.
• Multi-task learning: Combined NLP and prediction tasks in the instruction dataset to improve generalization.
• Evaluation protocol: Used zero-shot for most tasks and few-shot for specific baselines to ensure fair comparison.
• Data modalities: Integrated text, tables (from earnings reports), and time-series data (historical prices) into the instruction format.

Results & findings

• FinMA-30B outperforms GPT-4 by 10% F1 score on the FPB sentiment analysis dataset.
• FinMA significantly outperforms BloombergGPT on FPB (37% F1 improvement) and Headline classification.
• FinMA struggles with financial question answering (FinQA/ConvFinQA), underperforming GPT-4 and BloombergGPT due to LLaMA's lack of mathematical pre-training.
• All LLMs, including FinMA and GPT-4, show limited performance on stock movement prediction tasks, with low MCC scores.
• FinMA-7B-full, fine-tuned on both NLP and prediction tasks, achieves the best performance on the ACL18 stock prediction dataset.
• Model size (7B vs 30B) has less impact on NLP tasks than the quality of instruction data, but larger models perform better on complex QA tasks.

Data & experiments

• FIT dataset: 136,609 instruction samples derived from 9 open-source datasets across 5 tasks.
• FLARE benchmark: Includes 6 datasets for NLP tasks (FPB, FiQA-SA, Headline, NER, FinQA, ConvFinQA) and 3 for stock prediction (BigData22, ACL18, CIKM18).
• Experiments compare FinMA-7B, FinMA-30B, and FinMA-7B-full against GPT-4, ChatGPT, BloombergGPT, BLOOM, OPT-66B, and Vicuna-13B.
• Evaluation metrics include Accuracy, F1, Exact Match, and Matthews Correlation Coefficient (MCC).
• Testing protocols include zero-shot and few-shot (5-shot/20-shot) evaluations.

Limitations & caveats

• FinMA-30B was not fine-tuned on the full dataset due to computational constraints.
• The model struggles with complex quantitative reasoning tasks like financial question answering.
• Performance on stock movement prediction is limited, with low MCC scores across all models.
• The instruction data relies on handcrafted prompts, which may limit diversity and generalizability.
• The maximum input size is constrained by the backbone LLaMA model's context window.

Curator notes

Discovery status: round2_promoted. Source links: FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning; From Scores to Skills: A Cognitive Diagnosis Framework for Evaluating Financial Large Language Models; Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance - A Case Study in Finance; No Language is an Island: Unifying Chinese and English in Financial Large Language Models, Instruction Data, and Benchmarks; Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Risk