Reasoning or Overthinking: Evaluating Large Language Models on Financial Sentiment Analysis

In one sentence

Summary

This study investigates whether explicit reasoning improves large language model (LLM) performance on financial sentiment analysis, a task grounded in human perception and intuition. The authors challenge the prevailing assumption that more reasoning universally enhances LLM decisions, positing that for subjective financial tasks, fast, intuitive judgment may better match human annotators than slow, deliberative processes. They evaluate three proprietary models (GPT-4o, GPT-4.1, o3-mini) and two finetuned baselines (FinBERT-Prosus, FinBERT-Tone) on the Financial PhraseBank dataset using zero-shot settings. The experimental design compares four prompting paradigms: No-CoT (direct classification), CoT-Short, CoT-Long, and LIRA (label first, then explanation), mapping these to cognitive System 1 and System 2 theories to assess alignment with human sentiment labels. The evaluation focuses on macro F1 scores across varying levels of inter-annotator agreement and linguistic complexity. Results indicate that No-CoT prompting consistently yields the highest performance, with GPT-4o achieving the best overall alignment. Introducing explicit reasoning via CoT strategies degrades performance, particularly in low-ambiguity cases. The reasoning-optimized model o3-mini performs the worst, generating verbose outputs that correlate negatively with accuracy. LIRA prompting outperforms forward-chained CoT, suggesting that post-hoc rationalization aligns better with human interpretation than pre-commitment reasoning. Finetuned models like FinBERT-Prosus perform well in-domain but generalize poorly to complex or ambiguous out-of-domain text compared to zero-shot general-purpose LLMs. The findings suggest that reasoning can introduce 'overthinking,' causing models to misalign with human heuristic judgments. Longer completion tokens are negatively correlated with performance, indicating that verbose reasoning dilutes predictive accuracy. The study highlights that while reasoning might help in highly ambiguous scenarios, it generally harms performance on clear-cut financial sentiment tasks. Limitations include the focus on zero-shot settings and the use of human-annotated sentiment rather than market-impact data, meaning the results reflect alignment with human perception rather than predictive power for stock movements. The work advises matching prompting strategies to task nature, favoring intuitive approaches for sentiment classification.

Research problem

The paper addresses whether explicit reasoning mechanisms, such as Chain-of-Thought prompting or reasoning-optimized model architectures, improve or degrade the alignment of large language models with human-annotated financial sentiment labels. It specifically investigates if 'System 2' deliberative reasoning introduces unnecessary complexity or 'overthinking' that harms performance on subjective, perception-based tasks compared to 'System 1' intuitive judgment.

Core contributions

• Empirical evaluation of zero-shot LLM performance on financial sentiment analysis using diverse prompting strategies.
• Demonstration that direct, non-reasoning prompts (No-CoT) outperform Chain-of-Thought methods in aligning with human sentiment labels.
• Identification of 'overthinking' as a failure mode where verbose reasoning degrades accuracy on subjective financial text.
• Analysis of the impact of inter-annotator agreement and linguistic complexity on model reasoning efficacy.
• Comparison of reasoning-optimized models (o3-mini) against general-purpose models, showing inferior performance in this domain.

Method

• Zero-shot evaluation of proprietary and finetuned models on Financial PhraseBank.
• Comparison of four prompting strategies: No-CoT, CoT-Short, CoT-Long, and LIRA.
• Stratification of analysis by inter-annotator agreement levels (50-65% to 100%).
• Stratification by linguistic complexity using Flesch-Kincaid readability scores.
• Analysis of completion token counts to correlate verbosity with performance.
• Confusion matrix analysis to identify specific misclassification patterns (e.g., neutral vs. positive).

Results & findings

• GPT-4o with No-CoT prompting achieved the highest macro F1 score across all configurations.
• CoT prompting consistently degraded performance compared to No-CoT for GPT-4o and GPT-4.1.
• o3-mini, optimized for reasoning, yielded the lowest performance and highest token counts, indicating overthinking.
• LIRA prompting outperformed forward-chained CoT, suggesting post-hoc explanation is more aligned with human judgment.
• Performance declined monotonically with increasing completion token length, reinforcing the overthinking hypothesis.
• FinBERT-Prosus performed best in-domain but generalized poorly to complex/out-of-domain text compared to zero-shot LLMs.
• Reasoning did not improve robustness to linguistic complexity; performance dropped for all models on high-complexity sentences.

Data & experiments

• Dataset: Financial PhraseBank, containing 4,845 sentences annotated by finance-savvy individuals with inter-annotator agreement levels.
• Models: GPT-4o, GPT-4.1, o3-mini, FinBERT-Prosus, and FinBERT-Tone.
• Prompting Paradigms: No-CoT, CoT-Short, CoT-Long, and LIRA (Label-then-Reasoning).
• Metrics: Macro F1 score to handle class imbalance, analyzed across agreement levels and readability quartiles.
• Experimental Design: Zero-shot evaluation to isolate intrinsic model capabilities without in-context learning bias.

Limitations & caveats

• Evaluation is limited to zero-shot settings; few-shot or fine-tuning effects are not explored.
• Dataset reflects human perception of sentiment, not actual market impact or stock price movements.
• Results may not generalize to other financial tasks requiring logical deduction or arithmetic reasoning.
• Proprietary models were evaluated at specific snapshot versions, limiting reproducibility.
• The study focuses on short news sentences, not longer disclosures or complex reports.

Curator notes

Discovery status: round3_promoted_seed_for_round4. Source links: Assessing Look-Ahead Bias in Stock Return Predictions Generated by GPT Sentiment Analysis; Can Large Language Models beat wall street? Evaluating GPT-4’s impact on financial decision-making with MarketSenseAI; Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learning; FinBERT : A Large Language Model for Extracting Information from Financial Text†