Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models

In one sentence

Summary

Financial sentiment analysis is critical for investment decision-making, yet traditional NLP models lack generalization, and direct application of Large Language Models (LLMs) suffers from misaligned pre-training objectives and insufficient context in brief financial news. The authors address these challenges by proposing a framework that integrates instruction tuning with retrieval-augmented generation (RAG). This approach aligns LLMs to predict specific sentiment labels while enriching short input queries with relevant background information from external sources, thereby improving prediction reliability and accuracy in complex financial scenarios. The method involves two main components: an instruction-tuned LLM module and a RAG module. The instruction tuning uses a small dataset of 10,501 samples from Twitter Financial News and FiQA, formatted with human-written instructions to fine-tune open-source models like Llama-7B. The RAG module retrieves context from verified sources such as Bloomberg, Reuters, and Seeking Alpha using a two-step process: multi-source querying followed by similarity-based filtering using the Szymkiewicz-Simpson coefficient. Experiments evaluate the model on the Financial PhraseBank and Twitter validation sets, comparing it against baselines including FinBERT, ChatGPT, and LLaMA, using accuracy and F1 score as metrics. Results show that the proposed method achieves a 15% to 48% performance gain in accuracy and F1 score over baselines, with the instruction-tuned Llama-7B outperforming ChatGPT 4.0 and FinBERT. The RAG module further enhances performance by resolving ambiguities in short news snippets, as demonstrated in case studies where retrieved context clarified sentiment. However, the approach relies exclusively on textual similarity for retrieval, potentially overlooking macroeconomic timing and microeconomic operational data, which limits its holistic view of financial sentiment and suggests future work should incorporate structured economic indicators.

Research problem

Financial sentiment analysis is essential for valuation and investment but is hindered by the limited generalization of traditional NLP models and the contextual deficiencies of direct LLM application. Specifically, LLMs struggle with the discrepancy between their causal language modeling pre-training and the classification task of sentiment prediction, as well as the brevity of financial news which lacks sufficient background context for accurate interpretation.

Core contributions

• Introduction of a novel retrieval-augmented LLM framework tailored for financial sentiment analysis.
• Development of an instruction-tuning method using a small, curated dataset to align LLMs with sentiment prediction tasks.
• Implementation of a multi-source RAG module that retrieves and filters context from verified financial sources.
• Demonstration of significant performance improvements over traditional models and general-purpose LLMs like ChatGPT and LLaMA.
• Provision of a case study illustrating how RAG resolves ambiguity in short financial news snippets.

Method

• Instruction tuning of Llama-7B using 10 human-written instructions and 10,501 financial sentiment samples.
• Causal language modeling fine-tuning with AdamW optimizer and DeepSpeed for efficiency.
• Multi-source knowledge querying using APIs from Bloomberg, Reuters, Seeking Alpha, and social media.
• Similarity-based retrieval using the Szymkiewicz-Simpson coefficient to filter relevant context snippets.
• Output mapping to sentiment classes (positive, negative, neutral) by checking for keyword presence in generated text.
• Evaluation using accuracy and F1 score on standard financial sentiment benchmarks.

Results & findings

• The proposed model achieves 0.758 accuracy and 0.739 F1 on Financial PhraseBank, outperforming ChatGPT 4.0 (0.643 Acc, 0.511 F1).
• On Twitter Validation, the model reaches 0.863 accuracy and 0.811 F1, surpassing ChatGPT 4.0 (0.788 Acc, 0.652 F1).
• Adding RAG further improves performance, with the final model achieving 0.881 accuracy and 0.842 F1 on Twitter Validation.
• The performance gain ranges from 15% to 48% in accuracy and F1 score compared to baselines.
• Case studies show RAG correctly identifies positive sentiment in ambiguous headlines by retrieving analyst upgrade information.
• Instruction tuning alone provides a significant boost over zero-shot LLMs, highlighting the importance of task alignment.

Data & experiments

• Training data combines Twitter Financial News (9,540 samples) and FiQA (961 samples) for a total of 10,501 samples.
• Testing datasets include Twitter Financial News validation (2,388 samples) and Financial PhraseBank (4,840 samples).
• Baselines include FinBERT, BloombergGPT, ChatGPT 4.0, Llama-7B, and ChatGLM2-6B.
• Experiments measure accuracy and F1 score on both instruction-tuned models and models with/without RAG.
• Evaluation includes ablation studies to isolate the impact of instruction tuning and RAG components.

Limitations & caveats

• Reliance on textual similarity for retrieval may miss crucial macroeconomic or microeconomic context not present in text.
• The approach does not incorporate structured financial data or timing information, limiting holistic analysis.
• Performance may vary across different financial domains or languages not covered in the training data.
• The retrieval process depends on the availability and quality of external API sources.
• Instruction tuning requires a small amount of labeled data, which may be costly to produce for niche financial tasks.

Curator notes

Discovery status: round1_promoted_seed_for_round2. Source links: A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges; AI in Investment Analysis: Large Language Models for Equity Stock Ratings; Adaptive Financial Sentiment Analysis for NIFTY 50 via Instruction-Tuned LLMs ,RAG and Reinforcement Learning Approaches; BloombergGPT: A Large Language Model for Finance; Combining Financial Data and News Articles for Stock Price Movement Prediction Using Large Language Models; Deficiency of Large Language Models in Finance: