The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT over Multimodal Stock Movement Prediction Challenges

In one sentence

Summary

This study addresses the gap in understanding how large language models perform in financial forecasting tasks without fine-tuning. Specifically, it investigates whether ChatGPT can effectively predict binary stock price movements (rise or fall) by integrating historical price features with social media sentiment from tweets. The research positions ChatGPT as a 'Wall Street Neophyte,' aiming to determine if its general language understanding translates to accurate market prediction or if it lacks the specialized reasoning required for complex financial time-series analysis. The work challenges the assumption that zero-shot LLMs can immediately replace specialized financial models in high-stakes prediction environments. The authors conduct a comprehensive zero-shot analysis using three benchmark datasets: BIGDATA22, ACL18, and CIKM18, which contain high-trade-volume US stocks, historical price features, and associated tweets. They compare ChatGPT against strong baselines including Logistic Regression, Random Forest, LSTM, and state-of-the-art multimodal models like SLOT and DTML. The experimental design tests vanilla zero-shot prompting and Chain-of-Thought (CoT) strategies to see if structured reasoning improves accuracy. Metrics include Accuracy and Matthews Correlation Coefficient (MCC). The study also performs ablation studies to isolate the impact of tweet inclusion and analyzes the quality of the model's generated explanations through case studies. Results show that ChatGPT generally underperforms both traditional statistical methods and advanced deep learning models, often failing to beat simple linear regression baselines. While incorporating tweets provides a positive, albeit modest, boost in performance, the model struggles to effectively fuse multimodal information. CoT prompting offers limited improvement in accuracy but provides some interpretability, though the explanations are often superficial and prone to hallucination or incorrect sentiment alignment. The paper concludes that ChatGPT lacks the stability and precision for direct trading applications without specialized training, highlighting significant limitations in handling long-tail stock distributions and complex market dynamics.

Research problem

The paper investigates the effectiveness of zero-shot large language models, specifically ChatGPT, in multimodal stock movement prediction. It seeks to determine if ChatGPT can accurately predict binary stock price movements (rise or fall) by combining historical price data with social media sentiment from tweets. The core problem is assessing whether general-purpose LLMs possess the inherent reasoning capabilities required for financial forecasting or if they require specialized fine-tuning to be competitive with domain-specific models.

Core contributions

• Conducts the first extensive zero-shot evaluation of ChatGPT on multimodal stock movement prediction tasks.
• Compares ChatGPT against a wide range of baselines, including traditional ML (LR, RF) and state-of-the-art deep learning models (SLOT, DTML).
• Analyzes the impact of Chain-of-Thought prompting on prediction accuracy and explainability.
• Provides ablation studies quantifying the value of tweet data versus historical price features alone.
• Offers detailed case studies and error analysis revealing limitations in multimodal fusion and reasoning stability.

Method

• Zero-shot inference using OpenAI's ChatGPT API without fine-tuning.
• Prompt engineering with vanilla instructions and Chain-of-Thought (CoT) steps.
• Multimodal input formatting combining tabular price data and concatenated tweet text.
• Comparison against 10+ baseline models including deep learning and traditional ML.
• Evaluation using Accuracy and Matthews Correlation Coefficient (MCC).
• Ablation studies removing tweet data to isolate feature importance.
• Case study analysis of model predictions versus ground truth.
• Error analysis focusing on sentiment misalignment and trend misinterpretation.

Results & findings

• ChatGPT underperforms state-of-the-art methods like SLOT and DTML across most datasets.
• ChatGPT often fails to beat traditional baselines like Logistic Regression and Random Forest.
• Inclusion of tweets generally improves ChatGPT's performance, confirming the value of social sentiment.
• Chain-of-Thought prompting yields only marginal accuracy improvements over vanilla zero-shot.
• Performance varies significantly across datasets, indicating sensitivity to data distribution.
• ChatGPT's explanations are often superficial, focusing on simple trends rather than complex market dynamics.
• The model struggles to align and fuse information across price features and tweet sentiment effectively.

Data & experiments

• Uses three benchmark datasets: BIGDATA22, ACL18, and CIKM18, covering high-volume US stocks.
• Datasets include historical price features (open, high, low, close, adjusted close, increments) and associated tweets.
• Evaluates binary classification of price movements (rise/fall) using Accuracy and MCC metrics.
• Compares zero-shot ChatGPT against Logistic Regression, Random Forest, LSTM, ALSTM, Adv-ALSTM, DTML, ALSTM-W, ALSTM-D, StockNet, and SLOT.
• Tests vanilla zero-shot and Chain-of-Thought prompting strategies.
• Performs ablation studies to measure the marginal gain of including tweet data.

Limitations & caveats

• Evaluation is limited to zero-shot performance; fine-tuned results are not reported.
• Only one model (ChatGPT/gpt-3.5-turbo) is tested, limiting generalizability to other LLMs.
• Datasets are restricted to US stock markets, potentially limiting applicability to other regions.
• Input length constraints require truncation of tweets and limited historical windows.
• Model may produce unanticipated responses or violate content policies.
• Explanations provided by the model are often superficial and may not reflect true reasoning.

Curator notes

Useful counterweight to overly optimistic trading papers.