Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models

In one sentence

Summary

This paper investigates whether off-the-shelf large language models (LLMs) like ChatGPT can predict stock market reactions to news headlines without explicit financial training. The authors position LLMs as novel instruments for studying market information processing, comparing LLM assessments of news economic implications with actual market responses to quantify underreaction and information processing frictions. They develop a theoretical model incorporating LLM technology, information-processing constraints, and limits to arbitrage to explain why sophisticated models can exploit these inefficiencies. The empirical analysis uses 159,137 firm-headline-date observations for 4,123 U.S. stocks from October 2021 to May 2024, ensuring an out-of-sample evaluation post-Knowledge cutoff. The authors prompt GPT-4 to categorize headlines as positive, negative, or neutral, then measure initial market reactions and subsequent price drift. They compare GPT-4 against smaller models (GPT-3.5, Llama2) and embedding-based supervised learning methods. Experiments include long-short daily-rebalanced strategies, transaction cost sensitivity analyses with partial rebalancing, and topic modeling to decompose performance across news themes like earnings, insider transactions, and clinical trials. GPT-4 achieves ~90% hit rates for initial reactions and generates a long-short strategy with an annualized Sharpe ratio of 2.97 (zero costs), significantly outperforming smaller models. Predictability is concentrated in small-cap stocks and negative news, consistent with limits to arbitrage. The study finds that as LLM adoption rises, strategy returns decline, suggesting improved market efficiency. However, high turnover and transaction costs limit practical tradability for large investors. The paper concludes that financial reasoning is an emerging capacity of complex LLMs, offering a new lens on market inefficiencies and the impact of AI on price discovery.

Research problem

The paper addresses whether general-purpose LLMs, trained without explicit financial supervision, can accurately assess the economic implications of news headlines and predict subsequent stock market reactions. It seeks to understand if LLMs can serve as instruments to study market information processing, specifically identifying where and why markets underreact to information. The research also explores how LLM capabilities scale with model size and how widespread LLM adoption impacts market efficiency and return predictability.

Core contributions

• Demonstrates that GPT-4 can predict initial market reactions to news with ~90% hit rates without financial fine-tuning.
• Identifies significant post-announcement drift predictability using LLM signals, particularly for small caps and negative news.
• Develops a theoretical model linking LLM information-processing capacity to market underreaction and limits to arbitrage.
• Shows that return predictability declines as LLM adoption increases, providing evidence of AI-driven market efficiency improvements.
• Compares LLM zero-shot reasoning against embedding-based supervised learning, highlighting LLMs' robustness with limited data.
• Decomposes market inefficiencies by news topic, revealing that complex synthesis tasks (e.g., insider transactions) suffer more from underreaction than formulaic announcements.

Method

• Prompting GPT-4 to categorize news headlines as positive, negative, or neutral.
• Measuring initial market reaction (open-to-close or close-to-close returns) and subsequent drift.
• Constructing long-short portfolios based on LLM sentiment signals.
• Comparing performance across different LLM sizes (GPT-1 to GPT-4, Llama2 variants).
• Topic modeling using c-TF-IDF and K-Means clustering to group headlines into 15 themes.
• Partial rebalancing strategies to reduce turnover and test transaction cost sensitivity.
• Regression analysis with double-clustered standard errors by firm and date.
• Theoretical modeling of LLM information processing capacity and market equilibrium.

Results & findings

• GPT-4 achieves ~90% daily portfolio hit rates for predicting initial market reaction direction.
• Long-short strategy based on GPT-4 drift predictions yields 34 bps/day before costs.
• Annualized Sharpe ratio of 2.97 for GPT-4 drift strategy (zero costs), vs 1.66 for GPT-3.5.
• Predictability is concentrated in small-cap stocks and negative news, consistent with limits to arbitrage.
• Strategy returns decline over time (Sharpe from 6.54 in 2021Q4 to 1.22 in 2024), suggesting improved efficiency.
• GPT-4 outperforms embedding-based models, especially with limited training data.
• Complex news topics (insider transactions) show larger performance gaps between GPT-4 and GPT-3.5 than simple topics.
• High turnover (190%/day) makes full rebalancing sensitive to transaction costs; 25% rebalancing maintains Sharpe ratio.

Data & experiments

• 159,137 firm-headline-date observations for 4,123 U.S. stocks from CRSP and RavenPack (Oct 2021-May 2024).
• Out-of-sample evaluation using post-knowledge-cutoff headlines to avoid lookahead bias.
• Comparison of GPT-4 against GPT-3.5, Llama2, and embedding-based supervised models.
• Long-short daily-rebalanced trading strategies based on LLM sentiment scores.
• Transaction cost analysis with partial rebalancing strategies (25%, 50%, 100%).
• Topic modeling decomposition of performance across 15 news categories.

Limitations & caveats

• High daily turnover (190%) makes strategies sensitive to transaction costs, limiting practical tradability.
• Predictability is concentrated in small-cap stocks, which may have higher implementation costs.
• Strategy returns decline over time, suggesting alpha erosion as LLMs become more widespread.
• Short-side strategies are less profitable for large-cap stocks due to mean reversion after negative news.
• LLM assessments may not capture all nuances of complex financial news.
• Out-of-sample evaluation is limited to the period post-knowledge cutoff, potentially missing long-term trends.

Curator notes

Important for event-driven and news sentiment alpha studies.