Transforming Sentiment Analysis in the Financial Domain with ChatGPT

In one sentence

Summary

The paper addresses the challenge of financial sentiment analysis, specifically the difficulty of capturing nuanced, context-dependent sentiments in forex news where standard models often fail to distinguish between positive signals for one currency and negative signals for its pair. It positions ChatGPT 3.5 as a superior alternative to domain-specific models like FinBERT by leveraging its broad pre-training and zero-shot capabilities, thereby eliminating the need for costly fine-tuning on limited financial datasets. The research aims to validate whether large language models can effectively interpret complex financial narratives and align with market dynamics through strategic prompt design. The methodology involves a curated dataset of 2,291 manually annotated forex news headlines covering five major currency pairs, collected from Forex Live and FXstreet. The authors employ a zero-shot prompting approach, testing six distinct prompt templates that frame the task from various perspectives, such as a financial analyst or a sentiment analysis model. They evaluate performance using precision, recall, F1-score, and Sentiment Mean Absolute Error (S-MAE), while also measuring the correlation between predicted sentiment scores and actual market returns. The baseline comparison is established using FinBERT, implemented via the Hugging Face Transformers library, to ensure a rigorous benchmark against the current state-of-the-art in financial NLP. Results indicate that ChatGPT 3.5 achieves approximately 35% higher performance in sentiment classification and a 36% stronger correlation with market returns compared to FinBERT. The study finds that prompt engineering is critical, with prompts framing the model as a forex trader or analyst yielding the best results. However, the authors note limitations, including that sentiment alone does not fully predict price movements due to other market factors, and that API latency and costs vary with load. The paper concludes that while LLMs offer substantial improvements in accuracy and contextual understanding, they should be integrated into holistic trading systems rather than used as standalone predictive tools, and it releases the dataset to facilitate further research.

Research problem

Financial sentiment analysis faces challenges in interpreting nuanced, context-dependent sentiments in forex news, where standard models often misclassify sentiments for specific currency pairs. The paper investigates whether large language models, specifically ChatGPT 3.5, can outperform domain-specific models like FinBERT in this task using zero-shot prompting, thereby reducing the need for fine-tuning.

Core contributions

• Evaluates ChatGPT 3.5 for financial sentiment analysis using zero-shot prompting, demonstrating superior performance over FinBERT.
• Curates and publicly releases a novel dataset of 2,291 manually annotated forex news headlines linked to major currency pairs.
• Introduces a comprehensive evaluation framework combining traditional classification metrics with market return correlation analysis.
• Demonstrates the critical role of prompt engineering in optimizing LLM performance for specific financial use cases.
• Provides empirical evidence that LLMs can capture complex financial nuances without domain-specific fine-tuning.

Method

• Data Collection: Web scraping from Forex Live and FXstreet for five major forex pairs over 86 days.
• Annotation: Manual labeling of sentiment based on short-term market impact for each headline.
• Prompt Design: Creation of six zero-shot prompts with varying personas and output constraints.
• Model Evaluation: Comparison of ChatGPT 3.5 against FinBERT using precision, recall, F1, and S-MAE.
• Market Correlation: Analysis of the relationship between predicted sentiment scores and actual market returns.
• API Management: Monitoring of token usage, latency, and cost for operational efficiency assessment.

Results & findings

• ChatGPT 3.5 showed approximately 35% enhanced performance in sentiment classification compared to FinBERT.
• ChatGPT exhibited a 36% higher correlation with market returns than FinBERT.
• Prompt engineering significantly impacted performance, with trader-analyst personas yielding best results.
• Sentiment scores from ChatGPT correlated with market movements, though not perfectly predicting prices.
• The study confirms that zero-shot prompting can effectively leverage LLMs for financial NLP tasks.
• API latency and token costs were monitored, showing variability under different load conditions.

Data & experiments

• Dataset: 2,291 unique news headlines from Forex Live and FXstreet, covering AUDUSD, EURCHF, EURUSD, GBPUSD, and USDJPY.
• Annotation: Manual labeling of sentiment (positive, negative, neutral) based on short-term impact on the specific forex pair.
• Baseline: FinBERT model implemented via Hugging Face Transformers for sentiment classification and scoring.
• Experiments: Six distinct zero-shot prompts tested, varying in persona (analyst, trader, model) and output format (class, score, JSON).
• Metrics: Precision, recall, F1-score, Sentiment Mean Absolute Error (S-MAE), and correlation with market returns.

Limitations & caveats

• Sentiment analysis alone does not fully predict market movements due to other influencing factors.
• API latency and costs can vary significantly under different load conditions.
• The study focuses on short-term sentiment impact, which may not capture long-term trends.
• Results are based on GPT-3.5; newer models like GPT-4 may offer further improvements.
• Manual annotation, while accurate, is labor-intensive and may not scale easily.

Curator notes

Discovery status: round3_promoted. Source links: Can Large Language Models beat wall street? Evaluating GPT-4’s impact on financial decision-making with MarketSenseAI