FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets

In one sentence

Summary

This paper addresses the challenge of adapting open-source large language models to financial domains by proposing a structured instruction tuning paradigm. The authors argue that while closed-source models like BloombergGPT exist, open-source alternatives offer transparency and cost-efficiency. The research positions itself as a comprehensive benchmarking scheme that evaluates not just final performance, but the entire pipeline of integrating LLMs with financial datasets. It emphasizes reproducibility and the ability to handle diverse financial tasks, ranging from basic entity recognition to complex relation extraction, thereby filling a gap in understanding how different base models behave under domain-specific fine-tuning. The methodology involves instruction tuning six 7B-parameter open-source models (Llama2, Falcon, BLOOM, MPT, ChatGLM2, Qwen) using Low-Rank Adaptation (LoRA) on a cost-effective budget of approximately $300. The experimental design follows a three-phase progression: task-specific tuning on Sentiment Analysis, Headline Classification, Named Entity Recognition, and Relation Extraction; multi-task tuning combining these datasets; and zero-shot evaluation on unseen sentiment tasks. Data sources include FPB, FiQA-SA, TFNS, NWGI, NER, Headline, and FinRED. The study employs F1-scores as the primary metric, analyzing performance gains and degradations across phases to understand task interference and generalization capabilities. Key findings reveal that Llama2 and MPT achieved the best overall performance in task-specific and multi-task settings, with MPT showing significant improvements in information extraction tasks after multi-task training. However, zero-shot generalization was poor for most models, with ChatGLM2 and Falcon performing best, likely due to their chat-centric pre-training. A major limitation identified is the trade-off between specialization and generalization, where multi-task tuning sometimes degraded classification performance. The paper highlights that hallucination and task interference remain critical hurdles, suggesting that while instruction tuning is effective for specific financial NLP tasks, robust zero-shot financial reasoning requires further architectural or data-centric innovations.

Research problem

The paper addresses the difficulty of effectively integrating open-source large language models into financial contexts due to domain-specific terminology and the high cost of training. It seeks to determine how instruction tuning can efficiently adapt these models for various financial NLP tasks while maintaining transparency and reproducibility. The core problem is evaluating the trade-offs between task-specific specialization, multi-task versatility, and zero-shot generalization in a cost-effective manner.

Core contributions

• Proposes a three-phase instruction tuning paradigm (task-specific, multi-task, zero-shot) for open-source LLMs in finance.
• Introduces a cost-effective benchmarking scheme using LoRA, completing training for six models for under $300.
• Provides a comprehensive evaluation of six 7B-parameter open-source models across four financial NLP tasks.
• Demonstrates that multi-task tuning significantly improves information extraction (Relation Extraction) but may degrade classification performance.
• Highlights ChatGLM2's superior zero-shot generalization capabilities compared to other base models.

Method

• Task-Specific Instruction Tuning: Fine-tuning models on individual financial NLP tasks using LoRA.
• Multi-Task Instruction Tuning: Jointly training models on a combined dataset of SA, HC, NER, and RE tasks.
• Zero-Shot Evaluation: Testing models on unseen sentiment datasets after training on classification tasks.
• Task Reformulation: Converting NER and RE into classification tasks (NER(CLS), RE(CLS)) to improve zero-shot generalization.
• Dynamic Task Weighting: Oversampling smaller datasets to balance multi-task learning.
• LoRA Adaptation: Using low-rank matrices to update projection layers in attention modules.
• Cost-Effective Training: Utilizing FP16 precision and gradient accumulation to minimize GPU hours.

Results & findings

• Llama2 achieved the second-best average ranking across all tasks, showing strong versatility.
• MPT showed the most significant improvement in Relation Extraction (+35.8%) after multi-task tuning.
• BLOOM excelled in information extraction tasks but performed poorly in classification tasks.
• Multi-task tuning generally caused a slight decline in Sentiment Analysis performance for most models.
• ChatGLM2 outperformed others in zero-shot sentiment analysis, likely due to chat-centric pre-training.
• Falcon also demonstrated strong zero-shot generalization capabilities.
• Qwen showed consistent improvement in Sentiment Analysis across all sub-datasets during multi-task tuning.
• Total training cost for the entire benchmark was approximately $302.40.

Data & experiments

• Datasets: FPB, FiQA-SA, TFNS, NWGI (Sentiment); Headline (Classification); NER; FinRED (Relation Extraction).
• Models: Llama2-7B, Falcon-7B, BLOOM-7.1B, MPT-7B, ChatGLM2-6B, Qwen-7B.
• Method: Low-Rank Adaptation (LoRA) with rank 8 and alpha 32.
• Phases: Task-specific tuning, multi-task joint tuning, and zero-shot evaluation on unseen sentiment data.
• Metrics: Entity-level F1 for NER/RE, classification F1 for SA/HC.
• Hardware: 4 RTX 3090 GPUs, FP16 precision.

Limitations & caveats

• Zero-shot generalization was poor for most models, indicating limited transferability to unseen tasks.
• Multi-task tuning sometimes degraded performance on classification tasks due to task interference.
• The study is limited to 7B-parameter models, so results may not scale linearly to larger models.
• Hallucination remains a risk, particularly in zero-shot settings, though mitigated by task reformulation.
• The benchmark focuses on NLP tasks; it does not evaluate performance on numerical or tabular financial data.
• ChatGLM2's strong zero-shot performance may be biased towards its chat-centric pre-training data.

Curator notes

Discovery status: round1_promoted_seed_for_round2. Source links: A Survey of Large Language Models in Finance: FinLLMs; Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges; BloombergGPT: A Large Language Model for Finance; BondBERT: What we learn when assigning sentiment in the bond market; CN-Buzz2Portfolio: A Chinese-Market Dataset and Benchmark for LLM-Based Macro and Sector Asset Allocation from Daily Trending Financial News; Combining Financial Data and News Articles for Stock Price Movement Prediction Using Lar