PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance

In one sentence

Summary

The paper addresses the lack of open-source, instruction-tuned large language models for finance, noting that existing models like BloombergGPT are proprietary or lack instruction-following capabilities. The authors aim to democratize financial AI by creating a comprehensive framework that includes a domain-specific LLM, a large-scale instruction tuning dataset, and a holistic evaluation benchmark covering both natural language understanding and financial prediction tasks. This positioning fills a critical gap in the open-source ecosystem, enabling reproducible research and development in financial AI without relying on closed proprietary models. The core contribution is the PIXIU framework, which comprises three main components: the FIT dataset, the FinMA model, and the FLARE benchmark. FIT is a multi-task, multi-modal instruction tuning dataset containing 136,609 samples across five tasks: financial sentiment analysis, news headline classification, named entity recognition, financial question answering, and stock movement prediction. The authors fine-tuned LLaMA-7B and LLaMA-30B to create FinMA. FLARE serves as the evaluation benchmark, integrating six NLP datasets and three stock movement prediction datasets. Experiments compared FinMA against proprietary models like GPT-4, ChatGPT, and BloombergGPT, using metrics such as F1, accuracy, and Matthews correlation coefficient across zero-shot and few-shot settings. Results indicate that FinMA significantly outperforms general LLMs and BloombergGPT on most financial NLP tasks, such as sentiment analysis and headline classification, demonstrating the value of domain-specific instruction tuning. However, FinMA underperforms on financial question answering tasks requiring complex numerical reasoning, a limitation attributed to the LLaMA backbone's lack of mathematical pre-training. Additionally, all models, including FinMA, showed limited performance on stock movement prediction, highlighting the difficulty of this task. The study concludes that while domain adaptation improves NLP performance, challenges remain in quantitative reasoning and prediction, suggesting future work should focus on enhancing these capabilities through specialized training data and model architectures.

Research problem

The financial domain lacks publicly available, instruction-tuned large language models and comprehensive evaluation benchmarks that include both NLP and prediction tasks. Existing solutions are either proprietary (e.g., BloombergGPT) or limited to small-scale pre-trained models without instruction-following capabilities. This paper addresses the need for open-source tools to advance financial AI research and application.

Core contributions

• Introduction of FIT, the first multi-task and multi-modal financial instruction tuning dataset with 136K samples.
• Development of FinMA, the first open-source, instruction-following financial LLM based on LLaMA.
• Creation of FLARE, a holistic evaluation benchmark covering financial NLP and stock movement prediction.
• Comprehensive empirical analysis comparing FinMA against proprietary models like GPT-4 and BloombergGPT.
• Open-sourcing of the model, dataset, benchmark, and code to facilitate future financial AI research.

Method

• Instruction construction: Domain experts wrote 10 diverse instructions per dataset.
• Data assembly: Combined raw financial data with instructions to create FIT.
• Model fine-tuning: Supervised fine-tuning of LLaMA-7B and LLaMA-30B using AdamW optimizer.
• Multi-task learning: Simultaneous training on NLP and prediction tasks for FinMA-7B-full.
• Evaluation protocol: Zero-shot and few-shot testing on FLARE benchmark datasets.
• Comparison baseline: Benchmarking against proprietary and open-source general LLMs.

Results & findings

• FinMA-30B outperforms GPT-4 by 10% F1 on FPB sentiment analysis.
• FinMA significantly beats BloombergGPT on FPB (37% F1 improvement) and Headline classification.
• FinMA struggles with financial QA (FinQA/ConvFinQA) due to LLaMA's weak quantitative reasoning.
• All LLMs show limited performance on stock movement prediction, with near-zero MCC on most datasets.
• FinMA-7B-full (fine-tuned on all tasks) performs best on ACL18 stock prediction among open models.
• Model size (7B vs 30B) has less impact on NLP tasks than instruction data quality.
• Proprietary models like GPT-4 still lead in complex numerical reasoning tasks.

Data & experiments

• FIT dataset: 136,609 samples across sentiment analysis, headline classification, NER, QA, and stock prediction.
• FLARE benchmark: 9 datasets including FPB, FiQA-SA, Headline, NER, FinQA, ConvFinQA, BigData22, ACL18, CIKM18.
• Models evaluated: FinMA-7B, FinMA-30B, FinMA-7B-full, BloombergGPT, GPT-4, ChatGPT, BLOOM, OPT-66B, Vicuna-13B.
• Experimental setup: Zero-shot and few-shot (5-shot/20-shot) evaluations on FLARE tasks.
• Metrics: Accuracy, F1, Exact Match, and Matthews Correlation Coefficient (MCC) for prediction tasks.

Limitations & caveats

• FinMA-30B was not fine-tuned on the full dataset due to computational constraints.
• Model struggles with quantitative reasoning, limiting performance on financial QA tasks.
• Stock movement prediction performance is limited across all evaluated LLMs.
• Instruction data relies on human-written prompts, potentially limiting diversity.
• Input length is constrained by the LLaMA backbone's maximum context window.
• Potential for misuse in spreading financial misinformation is acknowledged.

Curator notes

Important open-source financial LLM ecosystem paper.