Self-explanatory and Retrieval-augmented LLMs for Financial Sentiment Analysis

In one sentence

Summary

Financial sentiment analysis (FSA) faces challenges due to the brevity of financial texts and the implicit nature of sentiment, which often leads to inconsistent predictions by Large Language Models (LLMs). The paper addresses this by proposing FLEX (Financial Language Enhancement with Guided LLM Execution), a system that retrieves qualitative knowledge from an LLM to enrich input sentences. This enrichment makes sentences more knowledge-dense and explicit, preserving original meaning while clarifying financial concepts. The core problem is bridging the gap between general LLM capabilities and the specific, context-heavy requirements of financial text interpretation. The FLEX framework operates in two phases: MAKEUP and MAKEUP SELECTION. In the MAKEUP phase, an LLM generates multiple candidate sentences enriched with relevant financial context. The MAKEUP SELECTION phase then filters these candidates using a novel logic combining semantic similarity (via cosine similarity of embeddings) and perplexity (measuring naturalness and clarity). This selection process mitigates hallucinations and ensures the enriched sentence is both semantically faithful and more interpretable. The method is evaluated on three benchmark datasets: Financial PhraseBank (FPB), FiQA, and SEntFiN, using Mistral 7B for generation and prediction, and DistilBERT for fine-tuning baselines. Experimental results demonstrate that FLEX consistently improves FSA accuracy compared to baselines without enrichment. For decoder-only models like Mistral, FLEX boosts zero-shot accuracy from 75.7% to 83.6% and few-shot accuracy from 78.5% to 86.0%. For encoder-only models like DistilBERT, fine-tuning on FLEX-enriched data increases average accuracy from 83.4% to 91.0%. The ablation study confirms that both semantic similarity and perplexity checks are crucial for optimal performance. The approach is particularly effective for short, information-scarce texts like news headlines, making it a valuable tool for downstream financial applications where interpretability and accuracy are paramount.

Research problem

Financial sentiment analysis (FSA) is hindered by the brevity of financial texts (e.g., news flashes, tweets) and the implicit nature of sentiment, which general LLMs struggle to interpret consistently. The mismatch between LLM pre-training objectives and FSA goals leads to unreliable predictions. This paper addresses the need for a method to enrich financial sentences with explicit, domain-specific knowledge to improve sentiment prediction accuracy and model interpretability.

Core contributions

• Proposes FLEX, a novel retrieval-augmented framework that enriches financial sentences with qualitative knowledge using LLMs.
• Introduces a dual-criteria selection algorithm combining semantic similarity and perplexity to mitigate LLM hallucinations and ensure meaningful enrichment.
• Demonstrates significant accuracy improvements in FSA across zero-shot, few-shot, and fine-tuning settings on three major benchmarks.
• Provides open-source code for the FLEX framework to promote reproducibility and further research in financial NLP.

Method

• Uses Mistral 7B-Instruct models to generate multiple enriched sentence candidates (MAKEUP phase).
• Employs sentence-transformers (all-mpnet-base-v2) for semantic similarity calculation via cosine similarity.
• Uses the evaluate library with openai-gpt for perplexity measurement.
• Selection function maximizes semantic similarity while minimizing perplexity, subject to thresholds.
• Evaluates performance using accuracy metrics on three distinct FSA benchmarks.
• Compares against zero-shot, few-shot, and fine-tuned baselines without enrichment.

Results & findings

• FLEX improves Mistral 7B zero-shot accuracy from 75.7% to 83.6% on average across datasets.
• FLEX improves Mistral 7B few-shot accuracy from 78.5% to 86.0% on average.
• DistilBERT fine-tuned on FLEX-enriched data achieves 91.0% average accuracy, up from 83.4%.
• The FiQA dataset showed the most significant gain for DistilBERT, rising from 71% to 91.6%.
• Ablation study shows that removing either semantic or perplexity checks degrades performance.
• Case studies confirm FLEX correctly interprets implicit sentiments like 'net loss narrowed' as positive.
• FLEX effectively handles informal language, rewriting tweets into clear financial statements.
• The method preserves original meaning while adding necessary financial context.

Data & experiments

• Evaluated on three standard FSA datasets: Financial PhraseBank (FPB), FiQA Task 1, and SEntFiN 1.0.
• Experiments include zero-shot and few-shot prediction using Mistral 7B, and fine-tuning using DistilBERT.
• Ablation study isolates the impact of semantic similarity and perplexity checks in the selection process.
• Case studies qualitatively analyze how FLEX clarifies implicit sentiment in specific financial contexts.
• Metrics used are accuracy for sentiment classification tasks.

Limitations & caveats

• The enrichment process relies on LLMs, which may still introduce subtle biases or errors despite the selection criteria.
• The method is evaluated on three specific datasets; generalizability to other financial text types (e.g., earnings calls) is not tested.
• The selection thresholds (omega) are set based on average sentence length, which may not be optimal for all contexts.
• Computational overhead is increased due to the generation and selection of multiple candidates.
• The study focuses on sentiment classification; other FSA tasks like aspect-based sentiment are not explored.

Curator notes

Discovery status: round2_promoted. Source links: Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks; Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models; FinBERT: A Pretrained Language Model for Financial Communications