Navigating Complexity: GPT-4's Performance in Predicting Earnings and Stock Returns in China's A-Share Market

In one sentence

Summary

The paper addresses the challenge of applying large language models to financial analysis in emerging markets, specifically China's A-share market, which differs significantly from Western markets due to high retail investor participation, frequent policy interventions, and the prevalence of state-owned enterprises. The authors investigate whether GPT-4, trained primarily on English and Western data, can effectively analyze financial statements and predict earnings changes in this unique context, comparing its performance against human analysts and traditional benchmarks to assess generalizability and reliability. The methodology involves extracting annual financial data from 2000 to 2023 for A-share listed companies from Wind and CSMAR databases, applying strict filtering criteria to ensure data quality. The financial statements are standardized, anonymized, and normalized by industry medians to remove identifiers and scale effects. GPT-4 is prompted to act as a professional analyst, performing trend analysis, ratio calculation, industry comparison, and policy sensitivity analysis to predict the direction of next-year earnings per share (EPS) and provide a confidence score. The study evaluates prediction accuracy, F1 score, confidence calibration, and the economic value of the predictions by constructing portfolios based on GPT-4's signals and calculating stock returns, Sharpe ratios, and alpha. Results show that GPT-4's prediction accuracy fluctuates significantly between 10.62% and 48.67%, with an average F1 score of only 0.30, indicating poor classification performance. Despite low accuracy, the model maintains high confidence levels (75-90%), suggesting a significant overconfidence bias. Stock returns derived from GPT-4's predictions vary widely (-4.86% to 13.59%) and show no consistent correlation with prediction accuracy, implying that the model's insights do not translate into reliable alpha generation. The study concludes that while GPT-4 can perform structured financial analysis, it is not yet reliable for direct investment decisions in complex emerging markets, highlighting the need for hybrid models, improved calibration, and careful regulatory oversight.

Research problem

The paper investigates the efficacy of GPT-4 in predicting earnings changes and stock returns within China's A-share market, a context distinct from Western markets due to high retail investor presence, policy interventions, and state-owned enterprises. It addresses the gap in understanding how LLMs trained on Western data generalize to emerging markets with unique structural characteristics. The study aims to determine if GPT-4 can serve as a reliable tool for financial analysis and stock prediction in this environment compared to human analysts and traditional models.

Core contributions

• Evaluates GPT-4's performance in predicting earnings direction and stock returns in China's A-share market from 2000 to 2023.
• Introduces a structured prompt engineering approach for LLMs to simulate professional financial analyst reasoning on anonymized financial statements.
• Provides a comprehensive comparison of LLM performance against human analysts and traditional benchmarks in an emerging market context.
• Analyzes the relationship between LLM prediction confidence and actual accuracy, highlighting significant overconfidence bias.
• Assesses the economic value of LLM-based predictions by constructing portfolios and calculating risk-adjusted returns (Sharpe ratio, alpha).
• Discusses the implications of market-specific factors like policy interventions and retail investor behavior on AI model performance.

Method

• Data extraction from Wind and CSMAR databases with strict quality filters.
• Financial statement standardization and anonymization to remove identifiers and scale effects.
• Feature engineering: 50 financial forecasting variables including profitability, growth, efficiency, solvency, and cash flow.
• Structured prompt design: six-step process including trend analysis, ratio calculation, industry comparison, policy sensitivity, forecasting, and reasoning.
• GPT-4 inference: predicting EPS direction and confidence score based on anonymized financial data.
• Performance evaluation: accuracy, F1 score, confidence calibration, and portfolio-based returns (Sharpe ratio, alpha).
• Comparison with human analyst forecasts and traditional machine learning benchmarks.

Results & findings

• GPT-4 prediction accuracy fluctuates between 10.62% and 48.67%, with an average F1 score of 0.30.
• Model maintains high confidence levels (75-90%) despite low accuracy, indicating overconfidence bias.
• Stock returns from GPT-4 predictions vary widely (-4.86% to 13.59%) with no consistent correlation to accuracy.
• No reliable alpha generation observed from GPT-4-based trading strategies.
• Performance highlights challenges of applying Western-trained LLMs to emerging markets with unique structural features.
• Results suggest GPT-4 is not yet reliable for direct investment decisions in this context.
• High volatility in performance underscores sensitivity to market microstructure and policy changes.

Data & experiments

• Annual financial data from 2000 to 2023 for A-share listed companies from Wind and CSMAR databases.
• Strict data filtering: total assets > 100 million RMB, stock price > 1 RMB, fiscal year ending Dec 31, excluding ST companies.
• Standardized and anonymized balance sheets and income statements normalized by industry medians.
• Binary classification task: predicting increase or decrease in next year's EPS.
• Evaluation metrics: prediction accuracy, F1 score, confidence calibration, stock returns, Sharpe ratio, and alpha.
• Portfolio construction based on GPT-4 predictions to assess economic value and trading performance.

Limitations & caveats

• Focus on a single market (China A-shares) limits generalizability to other contexts.
• Reliance on historical data may not capture unprecedented market events or structural changes.
• Black-box nature of GPT-4 limits interpretability of its decision-making process.
• Anonymization and standardization may remove context crucial for accurate financial analysis.
• GPT-4's training data cutoff restricts awareness of recent policies and events.
• High overconfidence bias suggests need for improved calibration techniques.

Curator notes

Discovery status: trading_agent_focus_promoted. Source links: ECC Analyzer: Extract Trading Signal from Earnings Conference Calls using Large Language Model for Stock Performance Prediction; ECC Analyzer: Extracting Trading Signal from Earnings Conference Calls using Large Language Model for Stock Volatility Prediction