Financial Statement Analysis with Large Language Models

In one sentence

Summary

The paper investigates whether large language models can perform financial statement analysis comparable to professional human analysts, addressing the gap in understanding LLM capabilities in quantitative, judgment-heavy tasks. The authors position this research to challenge the notion that LLMs are merely text-processing tools, exploring their potential to emulate human deductive reasoning in fundamental analysis without relying on narrative context or industry-specific soft information. This setup tests the lower bound of LLM performance, as the model is disadvantaged by the absence of management discussion and macroeconomic context typically used by humans. The methodology involves feeding anonymized and standardized balance sheets and income statements from the Compustat universe (1968-2021) to GPT-4. The model is instructed via a Chain-of-Thought prompt to mimic analyst workflows, computing ratios and synthesizing insights to predict the directional change in future earnings. The study benchmarks GPT-4 against consensus analyst forecasts, stepwise logistic regression, and an artificial neural network trained on the same financial variables. To ensure results are not driven by memory leakage, the authors anonymize data, test for look-ahead bias using out-of-sample 2023 predictions, and verify that the model cannot identify firms from numerical data alone. They also analyze the informational content of the generated narratives using BERT embeddings. Results show that GPT-4 achieves 60.31% accuracy and an F1-score of 63.45%, significantly outperforming human analysts (53-57%) and matching the state-of-the-art ANN (60.45% accuracy). The LLM exhibits a relative advantage in predicting earnings for loss-making and small firms where human analysts struggle. The generated narratives contain significant predictive power, with an ANN trained on GPT’s text embeddings achieving high accuracy. Trading strategies based on GPT’s predictions yield a Sharpe ratio of 3.36 and annual alphas exceeding 12% in the Fama-French three-factor model. However, the study notes limitations, including the model's reliance on specific prompt engineering and the potential for performance degradation in earlier LLM versions like GPT-3.5, suggesting that while LLMs are powerful, their deployment requires careful validation and may not yet fully replace specialized quantitative models in all contexts.

Research problem

The paper addresses whether large language models can perform complex financial statement analysis, specifically predicting the direction of future earnings, at a level comparable to professional human analysts. It seeks to determine if LLMs can derive economic insights from purely numerical data without narrative context and whether their performance stems from memorization or genuine reasoning capabilities. The research also explores the complementarity between LLMs and specialized machine learning models in fundamental analysis.

Core contributions

• First large-scale evidence that a general-purpose LLM (GPT-4) outperforms human analysts in predicting earnings direction using only standardized financial statements.
• Demonstration that LLMs can emulate human-like deductive reasoning through Chain-of-Thought prompting, generating useful narrative insights that drive predictive accuracy.
• Establishment of complementarities between LLMs and specialized quantitative models (ANNs), showing that GPT-4 performs on par with state-of-the-art ML while excelling in complex cases like loss-making firms.
• Empirical validation that LLM predictions are not driven by memory leakage or look-ahead bias, confirmed through anonymization tests and out-of-sample predictions.
• Evidence that trading strategies based on LLM-generated earnings forecasts yield significant alphas and higher Sharpe ratios than those based on human analysts or traditional ML models.

Method

• Anonymization and standardization of Compustat balance sheets and income statements to remove firm-specific identifiers.
• Chain-of-Thought prompting to instruct GPT-4 to mimic analyst workflows, including ratio computation and synthesis.
• Comparison of GPT-4 predictions against consensus analyst forecasts, logistic regression, and ANN benchmarks.
• Out-of-sample testing using 2022 data to predict 2023 earnings to validate generalization and rule out look-ahead bias.
• BERT-based text embedding analysis to extract and quantify predictive signals from GPT-4's generated narratives.
• Portfolio formation and asset pricing tests to evaluate the economic value of LLM predictions in trading strategies.
• Confidence score analysis to assess the relationship between model certainty and prediction accuracy.

Results & findings

• GPT-4 with Chain-of-Thought achieves 60.31% accuracy and 63.45% F1-score, significantly beating human analysts (53-57% accuracy).
• GPT-4 performance is on par with a specialized ANN (60.45% accuracy) but has a higher F1-score (63.45% vs 61.62%).
• GPT-4 shows relative advantage in predicting earnings for loss-making and small firms where human analysts are less accurate.
• ANNs trained on GPT-4's generated text embeddings achieve 59% accuracy, confirming the informational value of the narratives.
• A dual-input model combining financial statements and GPT-4 text achieves the highest accuracy (63.16%) and F1-score (66.33%).
• Long-short trading strategies based on GPT-4 predictions yield a monthly Sharpe ratio of 3.36 and annual alphas exceeding 12%.
• GPT-4 cannot identify firm names or fiscal years from anonymized financial data, ruling out memory-based predictions.
• Earlier models like GPT-3.5 perform significantly worse (52.29% accuracy), highlighting the importance of model version.

Data & experiments

• Compustat universe of 150,678 firm-year observations from 1968-2021, anonymized and standardized to prevent memory leakage.
• I/B/E/S analyst consensus forecasts used as a benchmark for human prediction accuracy.
• Stepwise logistic regression (Ou and Penman, 1989) and an Artificial Neural Network (ANN) trained on the same financial variables as benchmarks.
• Out-of-sample test using 2022 data to predict 2023 earnings to rule out look-ahead bias.
• Asset pricing tests forming long-short portfolios based on GPT-4 predictions to evaluate economic value.
• Text embedding analysis using BERT to quantify the predictive power of GPT-4's generated narratives.

Limitations & caveats

• The study relies on a specific prompt engineering approach (Chain-of-Thought), which may not generalize to all LLM tasks.
• Performance may vary across different LLM versions, as GPT-3.5 showed significantly lower accuracy.
• The model is disadvantaged by the absence of narrative context, which human analysts typically use, potentially underestimating LLM potential.
• The sample is limited to US-listed firms in Compustat, which may not generalize to other markets or asset classes.
• The study focuses on directional earnings prediction, which is a binary task, and may not capture the full complexity of financial analysis.
• Potential for overfitting to the specific financial statement structure used in the study.

Curator notes

Major paper on GPT-4 financial-statement reasoning and trading signal construction.