An Evaluation of Reasoning Capabilities of Large Language Models in Financial Sentiment Analysis

In one sentence

Summary

This study addresses the gap in understanding how Large Language Models reason about specific financial attributes during sentiment analysis, a task distinct from general sentiment analysis due to its reliance on quantitative data, temporal context, and comparative benchmarks. The authors identify six key financial attributes—semantic, numerical, temporal, comparative, causal, and risk factors—and propose a Financial Attribute Prompting (FAP) framework to explicitly guide LLMs in considering these dimensions. The research aims to determine whether LLMs possess intrinsic reasoning capabilities for these attributes or require structured prompting to achieve competent performance in financial contexts. The experimental design employs the FAP framework on two benchmark datasets: PhraseBank (financial news) and Twitter Financial News. The study evaluates zero-shot performance of GPT-3.5, PaLM-2, and GPT-4 against lexicon-based and supervised baselines like FinBERT. An ablation study isolates the impact of each financial attribute prompt to measure reasoning deficiencies. Results show that while FAP significantly boosts accuracy for PaLM-2 and GPT-3.5, allowing them to surpass unsupervised baselines, it reveals critical weaknesses. Specifically, removing numerical and comparative prompts causes sharp performance drops, indicating these models struggle to interpret quantitative metrics and relative performance comparisons without explicit instruction. The findings highlight that LLMs do not inherently reason about financial nuances; they require structured guidance to process numerical and comparative data effectively. While causal reasoning shows some intrinsic capability in GPT-3.5, other attributes like temporal and risk factors yield inconsistent results across datasets, suggesting sensitivity to annotation criteria. The paper concludes that current LLMs lack the structural thinking framework of human analysts for FSA. This has implications for deploying LLMs in finance, as reliance on zero-shot capabilities may lead to errors in interpreting complex financial texts, necessitating robust prompting strategies or fine-tuning for reliable sentiment analysis.

Research problem

The paper investigates the extent to which Large Language Models can effectively reason about specific financial attributes when performing Financial Sentiment Analysis (FSA). It seeks to identify which financial attributes (semantic, numerical, temporal, comparative, causal, risk) are critical for FSA and evaluate whether LLMs possess intrinsic reasoning capabilities for these attributes or rely on explicit prompting.

Core contributions

• Identification of six key financial attributes (semantic, numerical, temporal, comparative, causal, risk) that convey sentiment in financial texts.
• Development of the Financial Attribute Prompting (FAP) framework, a structured prompting method to guide LLMs through attribute-specific reasoning steps.
• Comprehensive empirical evaluation of LLM reasoning capabilities (GPT-3.5, PaLM-2, GPT-4) on FSA benchmarks using ablation studies.
• Demonstration that LLMs lack inherent reasoning for numerical and comparative attributes, requiring explicit prompts to achieve competitive performance.
• Comparison of LLM performance against traditional lexicon-based and supervised baselines, showing FAP-enhanced LLMs can outperform unsupervised methods.

Method

• Literature review to identify six critical financial attributes for sentiment conveyance.
• Design of six specific prompts: Semantic Overview, Numerical, Temporal, Comparative, Causal, and Risk/Uncertainty.
• Integration of Chain-of-Thought prefix ('Analyze... step-by-step') and suffix ('Positive/Neutral/Negative') to structure output.
• Ensemble of all six attribute prompts in the FAP framework.
• Ablation study: Iteratively removing each attribute prompt to measure its individual impact on accuracy.
• Evaluation on PhraseBank and Twitter Financial News datasets using accuracy and macro-F1.

Results & findings

• FAP significantly improves accuracy for PaLM-2 and GPT-3.5, enabling them to surpass unsupervised lexicon-based baselines.
• PaLM-2 and GPT-3.5 show severe weaknesses in numerical and comparative reasoning; removing these prompts causes sharp accuracy drops.
• GPT-4 achieves high accuracy (0.96 on PhraseBank-100%) but shows marginal improvement with FAP, suggesting stronger intrinsic reasoning.
• Causal reasoning is relatively strong in GPT-3.5 but weaker in PaLM-2, as indicated by performance changes when causal prompts are removed.
• Temporal and risk prompts yield inconsistent results across datasets, possibly due to annotation subjectivity.
• Vanilla PaLM-2 and GPT-3.5 perform worse than state-of-the-art lexicon method FinSenticNet without FAP.

Data & experiments

• Datasets: PhraseBank (financial news with 100% and 50% annotator agreement) and Twitter Financial News dataset (11,932 tweets).
• Models: GPT-3.5, PaLM-2, and GPT-4 evaluated in zero-shot settings with and without FAP.
• Baselines: Lexicon-based methods (FinSenticNet, LM, SMSL) and supervised models (FinBERTa, FinBERTb, ULMFit, LPS, HSC).
• Metrics: Accuracy and macro-averaged F1-score.
• Ablation study: Removing individual financial attribute prompts to assess their specific contribution to model performance.

Limitations & caveats

• Evaluation limited to zero-shot settings; fine-tuning effects are not explored.
• Inconsistent results for temporal and risk prompts may be due to annotation subjectivity rather than model capability.
• Study focuses on sentiment polarity; does not address intensity or aspect-based sentiment.
• Datasets are relatively small compared to large-scale financial corpora.
• Results may not generalize to other financial tasks like stock prediction or risk assessment.

Curator notes

Discovery status: round3_promoted. Source links: BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs; FinBERT : A Large Language Model for Extracting Information from Financial Text†