The List / Foundation and Domain Language Models / paper
Comparative Investigation of GPT and FinBERT’s Sentiment Analysis Performance in News Across Different Sectors
Abstract
GPT (Generative Pre-trained Transformer) is a groundbreaking generative model that has facilitated substantial progress in natural language processing (NLP). As the GPT-n series has continued to evolve, its applications have garnered considerable attention across various industries, particularly in finance. In contrast, traditional financial research has primarily focused on analyzing structured data such as stock prices. However, recent trends highlight the growing importance of natural language techniques that address unstructured factors like investor sentiment and the impact of news. Positive or negative information about specific companies, industries, or the overall economy found in news or social media can influence investor behavior and market volatility, highlighting the critical need for robust sentiment analysis. In this context, we utilize the state-of-the-art language model GPT and the finance-specific sentiment analysis model FinBERT to perform sentiment and time-series analyses on financial news data, comparing the performance of the two models to demonstrate the potential of GPT. Furthermore, by examining the relationship between sentiment shifts in financial markets and news events, we aim to provide actionable insights for investment decision-making, emphasizing both the performance and interpretability of the models. To enhance the performance of GPT-4o, we employed a systematic approach to prompt design and optimization. This process involved iterative refinement, guided by insights derived from a labeled dataset. This approach emphasized the pivotal importance of prompt design in improving model accuracy, resulting in GPT-4o achieving higher performance than FinBERT. During the experiment phase, sentiment scores were generated from New York Times news data and visualized through time-series graphs for both models. Although both models exhibited similar trends, significant differences arose depending on news content characteristics across categories. According to the results, the performance of GPT-4o, optimized through prompt engineering, outperformed that of FinBERT by up to 10% depending on the sector. These findings emphasize the importance of prompt engineering and demonstrate GPT-4o’s potential to improve sentiment analysis. Furthermore, the categorized news data approach suggests potential applications in predicting the outlook of categorized financial products.
In one sentence
Summary
The paper addresses the comparative efficacy of general-purpose large language models versus domain-specific models in financial sentiment analysis. While FinBERT has been the standard for financial text processing, the rapid advancement of GPT models raises questions about their utility in specialized domains. The authors position this research to evaluate whether a generalist model like GPT-4o can surpass a specialist like FinBERT when properly guided, thereby challenging the necessity of domain-specific fine-tuning for certain NLP tasks in finance. The study focuses on news-based sentiment classification, a critical input for market sentiment indicators and investment decision support systems. The methodology involves a rigorous experimental design comparing GPT-4o and FinBERT on news data from The New York Times across three sectors: business, health, and technology. For GPT-4o, the authors employ a systematic prompt engineering process, testing eight initial prompts and refining them based on performance on a labeled Kaggle dataset. FinBERT is fine-tuned on a separate financial news dataset. Both models are evaluated using accuracy, precision, recall, and F1-score. The study generates time-series sentiment scores to visualize trends and compares the models' outputs to identify sector-specific performance variations and interpretability differences. Results indicate that GPT-4o, with optimized prompts, achieves higher accuracy than FinBERT, with performance gains of up to 10% depending on the sector. The study highlights that prompt design is a critical factor in unlocking GPT-4o's potential, often outweighing the benefits of domain-specific pre-training. However, the authors note that performance varies by sector, suggesting that news content characteristics influence model behavior. The findings suggest that GPT-4o can serve as a robust alternative to FinBERT for sentiment analysis, offering actionable insights for predicting financial product outlooks, though the study is limited by its reliance on news headlines and descriptions rather than full articles or real-time market data integration.
Research problem
The paper investigates whether general-purpose large language models (GPT-4o) can outperform domain-specific models (FinBERT) in financial sentiment analysis when subjected to systematic prompt engineering. It seeks to determine if the flexibility and advanced reasoning of GPT-4o can compensate for its lack of financial pre-training, particularly across different industry sectors.
Core contributions
- Systematic evaluation of GPT-4o versus FinBERT in financial sentiment analysis across multiple sectors.
- Development of a refined prompt engineering framework for GPT-4o that significantly boosts its accuracy.
- Demonstration that GPT-4o can outperform FinBERT by up to 10% in specific sectors.
- Analysis of sector-specific performance variations, highlighting the impact of news content characteristics.
- Generation of time-series sentiment data for comparative visualization and trend analysis.
Method
- Systematic prompt engineering with iterative refinement of GPT-4o prompts.
- Fine-tuning of FinBERT on a labeled financial news dataset.
- Data preprocessing including removal of HTML entities, URLs, and duplicates.
- Evaluation using classification metrics: accuracy, precision, recall, and F1-score.
- Time-series visualization of sentiment scores for trend analysis.
- Sector-specific analysis to identify performance variations.
Results & findings
- GPT-4o outperformed FinBERT by up to 10% depending on the sector.
- Prompt engineering was identified as a pivotal factor in improving GPT-4o's accuracy.
- FinBERT required fine-tuning on financial data, while GPT-4o relied on prompt optimization.
- Performance differences were significant across business, health, and technology sectors.
- Both models showed similar overall trends but diverged in specific sentiment classifications.
- GPT-4o demonstrated superior interpretability and flexibility in handling diverse news content.
Data & experiments
- Labeled Kaggle news dataset used for prompt design and initial model evaluation.
- New York Times news articles from April 2023 to July 2024 across business, health, and technology sectors.
- FinBERT fine-tuned on a Kaggle financial news dataset of 32,417 articles.
- Evaluation metrics include accuracy, precision, recall, and macro F1-score.
- Time-series analysis of sentiment scores generated by both models.
Limitations & caveats
- Relies on news headlines and descriptions, which may not capture the full context of financial news.
- Limited to three sectors, potentially missing nuances in other industries.
- Does not integrate sentiment scores with real-time market data for trading simulation.
- Prompt engineering process may be sensitive to specific dataset characteristics.
- FinBERT's performance may vary with different fine-tuning strategies or datasets.
Curator notes
Discovery status: round3_promoted. Source links: Can Large Language Models beat wall street? Evaluating GPT-4’s impact on financial decision-making with MarketSenseAI; FinBERT : A Large Language Model for Extracting Information from Financial Text†; GPT-InvestAR: Enhancing Stock Investment Strategies through Annual Report Analysis with Large Language Models; Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models