Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models

In one sentence

Summary

Financial sentiment analysis is critical for market insight but remains challenging due to the need for precise numerical interpretation and deep contextual comprehension, which general-purpose LLMs often lack. This paper addresses these gaps by proposing Instruct-FinGPT, a method that transforms standard classification datasets into instruction-tuning formats. By fine-tuning the general-purpose LLaMA-7B model on a small subset of financial data, the approach leverages the model's inherent reasoning capabilities to better handle financial jargon, numeric values, and ambiguous contexts, aiming to bridge the performance gap between general LLMs and domain-specific models. The methodology involves formatting financial sentiment data into instruction-response pairs and supervised fine-tuning LLaMA-7B using sequence-to-sequence loss. The training data combines the Twitter Financial News dataset and the FiQA dataset, totaling over 10,000 samples. The model is evaluated on multiple benchmarks, including a custom numerical sensitivity dataset and a contextual understanding dataset, alongside the Financial PhraseBank. Baselines include FinBERT, ChatGPT (GPT-3.5 and GPT-4.0), and the base LLaMA-7B model. Evaluation metrics focus on accuracy and F1-score, with specific analysis on how models handle numeric fluctuations and implicit sentiment cues. Results demonstrate that Instruct-FinGPT-7B consistently outperforms FinBERT and ChatGPT across all datasets, achieving an accuracy of 0.880 on the Twitter validation set compared to FinBERT's 0.725. The model shows particular strength in numerical sensitivity, correctly interpreting sentiment from EPS and credit data where FinBERT fails. It also excels in contextual understanding, identifying nuanced sentiments in ambiguous news. However, the model still struggles with certain complex contextual examples, such as distinguishing between IPO price and market price movements. The study highlights that instruction tuning is a cost-effective alternative to training large domain-specific models, offering superior zero-shot generalization to unseen financial texts.

Research problem

General-purpose LLMs struggle with financial sentiment analysis due to insensitivity to numerical values and insufficient grasp of financial context. Existing domain-specific models like FinBERT require substantial labeled data and lack the reasoning capabilities of larger models. The paper addresses how to enable LLMs to accurately interpret numerical data and understand implicit context in financial texts using minimal supervised data.

Core contributions

• Proposes Instruct-FinGPT, an instruction-tuned LLaMA-7B model for financial sentiment analysis.
• Demonstrates that transforming classification tasks into instruction-tuning tasks improves LLM performance.
• Shows superior performance over FinBERT and ChatGPT in numerical sensitivity and contextual understanding.
• Validates the effectiveness of instruction tuning with a small amount of task-specific data.
• Provides a cost-effective alternative to training large proprietary financial models like BloombergGPT.

Method

• Instruction tuning: Transforming classification labels into natural language instructions.
• Supervised fine-tuning: Using sequence-to-sequence loss on LLaMA-7B.
• Data formatting: Combining human-written instructions with input text and output labels.
• Output mapping: Mapping generated text to sentiment labels (positive, negative, neutral).
• Evaluation: Comparing against FinBERT, ChatGPT, and base LLaMA-7B on multiple benchmarks.
• Training parameters: AdamW optimizer, learning rate 1e-5, batch size 32, 10 epochs.

Results & findings

• Instruct-FinGPT-7B achieves 0.880 accuracy on Twitter validation, outperforming FinBERT (0.725) and LLaMA-7B (0.54).
• F1-score of 0.841 for Instruct-FinGPT-7B vs. 0.668 for FinBERT on the same dataset.
• Model excels in numerical sensitivity, correctly interpreting EPS and credit data where FinBERT fails.
• Superior contextual understanding, identifying nuanced sentiments in ambiguous news articles.
• Zero-shot evaluation on Financial PhraseBank shows Instruct-FinGPT-7B (0.76 accuracy) outperforms ChatGPT-3.5 (0.64) and LLaMA-7B (0.60).
• Training cost is estimated at less than $300, significantly lower than proprietary models.
• Model still struggles with complex contextual examples, such as distinguishing IPO price from market price.

Data & experiments

• Training data: Twitter Financial News dataset (9,540 samples) and FiQA dataset (961 samples).
• Evaluation datasets: Twitter validation set, custom numerical sensitivity dataset (117 samples), contextual understanding dataset (20 samples), and Financial PhraseBank (4,840 samples).
• Baselines: FinBERT, ChatGPT (GPT-3.5 and GPT-4.0), and base LLaMA-7B.
• Metrics: Accuracy and F1-score for sentiment classification.
• Experimental design: Supervised fine-tuning of LLaMA-7B using instruction-tuned data with sequence-to-sequence loss.

Limitations & caveats

• Model still struggles with complex contextual examples, such as distinguishing between IPO price and market price.
• Evaluation is limited to English-language financial news and social media.
• Instruction tuning requires careful prompt engineering and data formatting.
• Performance on zero-shot generalization is good but not perfect, indicating potential overfitting to instruction formats.
• The study does not explore the impact of model size beyond LLaMA-7B.

Curator notes

Discovery status: round2_promoted. Source links: A Comprehensive Review of Generative AI in Finance; Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models; FinBERT: Financial Sentiment Analysis with Pre-trained Language Models; FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets; FinLlama: Financial Sentiment Classification for Algorithmic Trading Applications; Financial Report Chunking for Effective Retrieval Augmented Generation; Learning to Trade Like an Expert: Cog