A Comparative Analysis of Instruction Fine-Tuning Large Language Models for Financial Text Classification

In one sentence

Summary

This study addresses the challenge of adapting general-domain Large Language Models to specialized financial text classification tasks, where technical terminology and complex reasoning often cause performance degradation. The authors investigate the efficacy of instruction fine-tuning smaller-scale models, specifically Mistral-7B, Llama3-8B, and Phi-3-mini, comparing both base and instruction-tuned variants. The research aims to determine if efficient, resource-light fine-tuning can match or exceed proprietary models like GPT-4 while maintaining generalization capabilities on unseen financial tasks, thereby offering a scalable alternative to training from scratch or using massive proprietary models. The experimental design involves fine-tuning models on four core financial classification tasks: sentiment analysis (using Financial PhraseBank and FiQA-SA), news headline classification (Headline-Dir), relation extraction (FinRED), and hawkish-dovish policy classification (FOMC). The authors employ Low-Rank Adaptation (LoRA) for efficient training and evaluate performance using accuracy and weighted F1 scores. To assess generalization, they test zero-shot performance on three unseen tasks: argument unit classification (FinArg), deal completeness in M&A news (M&A dataset), and causal classification in SEC filings (FinCausual). Additionally, they implement model merging using the MergeKit framework, combining single-task fine-tuned models with vanilla instruction models to preserve zero-shot capabilities. Results indicate that instruction fine-tuning yields substantial improvements over zero-shot baselines, with fine-tuned models often outperforming GPT-4 on specific tasks like relation extraction and hawkish-dovish classification. However, base model fine-tuning suffered from significant catastrophic forgetting on unseen tasks, whereas instruction-tuned models remained more robust. Model merging successfully mitigated this degradation, with the merged Mistral-7B model exceeding its original zero-shot accuracy on several unseen datasets. The study concludes that instruction fine-tuning combined with model merging is a highly effective, resource-efficient strategy for deploying domain-specific LLMs in finance, though performance on long-tail relation extraction tasks remains a challenge.

Research problem

General-domain Large Language Models often struggle with the technical and specialized nature of financial texts, leading to suboptimal performance in tasks like sentiment analysis, relation extraction, and policy classification. The study addresses the need for efficient domain adaptation methods that do not require the massive computational resources of training from scratch or using proprietary models, while also mitigating the catastrophic forgetting that typically occurs when fine-tuning on narrow datasets.

Core contributions

• Comprehensive comparison of instruction fine-tuning for smaller open-source LLMs (Mistral-7B, Llama3-8B, Phi-3-mini) on financial text classification.
• Demonstration that instruction-tuned models are more robust to fine-tuning than base models, preserving better zero-shot generalization.
• Application of model merging via MergeKit to mitigate catastrophic forgetting, significantly enhancing performance on unseen financial tasks.
• Empirical evidence that fine-tuned smaller models can outperform proprietary models like GPT-4 on specific financial classification tasks.
• Analysis of in-context learning limitations, showing that increasing few-shot examples can degrade performance in small instruct models.

Method

• Instruction fine-tuning using HuggingFace Transformers with Adam-w optimizer and LoRA (rank 32, alpha 46).
• Quantization techniques to reduce computational cost during training.
• Single-task fine-tuning where each model is trained on one specific financial dataset.
• Multi-task fine-tuning using a combined dataset of all four financial tasks.
• Model merging using MergeKit with Task Arithmetic, combining single-task fine-tuned models with vanilla instruct models at equal weights.
• Evaluation of zero-shot, one-shot, five-shot, and ten-shot in-context learning capabilities.

Results & findings

• Instruction fine-tuning significantly improved task-specific performance across all models compared to zero-shot baselines.
• Base model fine-tuning led to greater performance degradation on unseen tasks compared to instruction-tuned models.
• Model merging using MergeKit enhanced zero-shot performance on unseen tasks, with Mistral-7B exceeding its original accuracy on some datasets.
• Fine-tuned models outperformed GPT-4 on relation extraction and hawkish-dovish classification tasks.
• Increasing the number of in-context examples (few-shot) degraded performance in small instruct models after a certain threshold.
• FinRED relation extraction remained challenging due to its large label space (29 classes), even after fine-tuning.
• Merged models balanced task-specific performance with generalization, avoiding the severe forgetting seen in single-task fine-tuned models.

Data & experiments

• Four training datasets: Financial PhraseBank, FiQA-SA (sentiment), Headline-Dir (news headlines), FinRED (relation extraction), and FOMC (hawkish-dovish classification).
• Three unseen evaluation datasets: FinArg-AUC-T1 (argument units), M&A (deal completeness), and FinCausual’20-T1 (causal classification).
• Experiments include zero-shot, few-shot (1, 5, 10 examples), single-task fine-tuning, multi-task fine-tuning, and model merging.
• Evaluation metrics include accuracy and weighted F1 score across all tasks.
• Baselines include vanilla instruct models, specialized financial models (FinMA-7B, AdaptLLM-7B), and GPT-4.

Limitations & caveats

• Model merging was not performed for Phi-3-mini due to language model head incompatibility.
• Performance on relation extraction (FinRED) remained low due to the large number of label classes.
• The study focuses on text classification; numerical reasoning and complex financial calculations are not addressed.
• Generalization to other financial domains beyond text classification (e.g., time-series forecasting) is not evaluated.
• The effectiveness of model merging may vary across different model architectures and sizes.

Curator notes

Discovery status: round2_promoted. Source links: Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks; Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models; FinBERT: A Pretrained Language Model for Financial Communications; FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets; InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning; Unlocking Data Value in Finance: A Study on Dist