AI in Investment Analysis: Large Language Models for Equity Stock Ratings

In one sentence

Summary

The paper addresses the challenge of automating equity stock ratings, a core task in investment analysis traditionally performed by human analysts who face data overload and cognitive biases. The authors position Large Language Models (LLMs) as a cost-effective, consistent alternative capable of processing multimodal financial data. The research problem centers on evaluating whether instruction-based LLMs can match or exceed the predictive accuracy of professional analysts when provided with structured financial inputs, specifically focusing on the impact of different data modalities such as fundamentals, news summaries, and sentiment scores on rating accuracy. The methodology employs GPT-4-32k (v0613) with a September 2021 training cutoff to prevent data leakage, applied to S&P 500 constituents from January 2022 to June 2024. The experimental design compares five methods: a Vanilla baseline using technical indicators, News (summaries), Sentiment (pre-computed scores), Fundamentals (quarterly 10-K/10-Q metrics), and Fundamentals + Sentiment. The model uses Chain-of-Thought prompting to generate ordinal ratings (Strong Sell to Strong Buy) and price targets. Evaluation is conducted using Mean Absolute Error (MAE) against forward returns (1, 3, 6, 12, 18 months), quantifying accuracy by comparing predicted ratings to the actual return quintiles of the market. Results indicate that the Fundamentals + Sentiment method achieves the lowest MAE (1.417), outperforming both the Vanilla baseline (1.447) and traditional analysts (1.570). While news summaries improve short-term (1-month) prediction, they introduce positive bias and do not enhance long-term accuracy compared to sentiment scores, which reduce token usage without performance loss. The study highlights that LLMs are more accurate in short-term horizons, whereas analysts perform relatively better over longer periods. Limitations include the exclusion of qualitative factors like earnings call transcripts and the sensitivity of forward-return evaluation to market volatility, suggesting that while LLMs offer a robust framework, they currently lack the nuanced qualitative judgment of human experts.

Research problem

The paper investigates whether instruction-based Large Language Models can effectively automate the generation of equity stock ratings, a critical but labor-intensive task in financial services. It specifically addresses the limitations of traditional analyst methods, such as data overload, inconsistency, and bias, by exploring how LLMs can leverage diverse data modalities to improve prediction accuracy and consistency.

Core contributions

• Demonstrates the novel application of GPT-4-32k for predicting ordinal stock ratings, a previously under-explored area in LLM finance research.
• Provides a reproducible framework for generating stock ratings using multimodal data (fundamentals, news, sentiment) without fine-tuning.
• Evaluates the impact of different data modalities, finding that financial fundamentals significantly enhance accuracy while raw news text introduces bias.
• Establishes that LLM-based ratings outperform traditional Wall Street analyst ratings when measured by forward return accuracy.
• Identifies that pre-computed sentiment scores are more efficient and less biased than raw news summaries for long-term predictions.

Method

• Used GPT-4-32k with a system prompt defining the persona of a financial analyst and defining rating scales.
• Implemented Chain-of-Thought (CoT) prompting to encourage reasoning before generating final ratings.
• Incorporated Chain of Verification (CoVE) to check date consistency and prevent hallucinations in time horizons.
• Structured input data with textual information (news/fundamentals) followed by numerical tables to optimize LLM ingestion.
• Computed forward returns and assigned ground-truth ratings based on return quintiles relative to the market and sector.
• Evaluated performance using Mean Absolute Error (MAE) to account for the ordinal nature of rating errors.

Results & findings

• The Fundamentals + Sentiment method achieved the lowest Mean Absolute Error (1.417), outperforming all other LLM variants and traditional analysts.
• Traditional analysts had a higher MAE (1.570) and were heavily biased towards 'Buy' ratings, whereas LLMs provided more balanced distributions.
• The Vanilla LLM baseline (1.447) already outperformed analysts, indicating that basic technical indicators are strong predictors.
• News summaries improved short-term (1-month) prediction accuracy but did not improve long-term accuracy compared to the Vanilla baseline.
• Substituting news summaries with sentiment scores reduced token usage without loss of performance, making it a more efficient approach.
• Raw news text introduced a positive bias in LLM ratings, correlating with sentiment scores and leading to less accurate long-term predictions.
• LLMs performed better in short-term horizons, while analysts showed relatively better performance over longer timeframes (18 months).

Data & experiments

• Dataset: S&P 500 constituents from January 2022 to June 2024, including analyst ratings, financial fundamentals (10-K/10-Q), news summaries, and historical price data.
• Model: GPT-4-32k (v0613) with a September 2021 training cutoff to ensure no information leakage from the test period.
• Methods: Five experimental setups: Vanilla (technical indicators), News (summaries), Sentiment (scores), Fundamentals (financial metrics), and Fundamentals + Sentiment.
• Evaluation: Mean Absolute Error (MAE) of predicted ratings against forward returns (1, 3, 6, 12, 18 months) categorized into quintiles.
• Comparison: LLM predictions are benchmarked against real-world analyst ratings from major investment banks.

Limitations & caveats

• The evaluation relies on forward returns, which can be sensitive to short-term market volatility and abnormal shifts.
• The model was not provided with qualitative factors like earnings call transcripts, management guidance, or investor sentiment.
• The study did not test the model's ability to process extremely large volumes of information, which human analysts often handle.
• Analyst ratings were assessed without exact target dates, introducing some uncertainty in the comparison.
• The LLM's tendency to generate positive ratings when exposed to news text suggests a potential bias that needs mitigation.
• The framework uses a static model (GPT-4-32k) and does not explore fine-tuning or newer model architectures.

Curator notes

Attempts to reduce leakage by using post-cutoff data.