Innovative Sentiment Analysis and Prediction of Stock Price Using FinBERT, GPT-4 and Logistic Regression: A Data-Driven Approach

In one sentence

Summary

This paper addresses the challenge of leveraging financial news sentiment to predict stock market movements, specifically focusing on the Nigerian NGX All-Share Index. The authors position their work within the broader context of AI-driven financial forecasting, aiming to determine whether state-of-the-art large language models like FinBERT and GPT-4 offer superior predictive power compared to traditional, computationally efficient machine learning baselines. The research problem centers on the trade-off between model complexity, computational cost, and prediction accuracy in real-world trading scenarios, questioning the practical utility of resource-intensive transformers for simple binary classification tasks in emerging markets. The methodology involves scraping 24,923 news headlines from Nigerian financial news sources (Nairametric and Proshare) spanning 2010 to 2024, aggregated into 3,573 temporal observations. The target variable is binary: Class 1 for daily share price gains and Class 0 for unchanged or fallen prices. The authors employ time-series cross-validation to prevent data leakage. FinBERT-base is fine-tuned using Optuna for hyperparameter optimization and mixed-precision training. GPT-4 is used via API with a predefined sentiment classification prompt. Logistic Regression utilizes TF-IDF vectorization, also optimized with Optuna. All models are evaluated on accuracy, precision, recall, F1 score, and ROC AUC. Results indicate that Logistic Regression significantly outperforms both LLMs, achieving 81.83% accuracy and 89.76% ROC AUC. FinBERT achieved moderate performance (63.33% accuracy, 65.59% ROC AUC) despite its domain-specific training, while GPT-4 performed poorly (54.19% accuracy) using its predefined approach. The study concludes that for this specific task and dataset, the simplicity and interpretability of Logistic Regression, combined with effective feature engineering, provide a more robust and efficient solution than complex deep learning models. The authors suggest future work should explore hybrid models that combine the nuanced sentiment extraction of FinBERT with the predictive strength of traditional classifiers, noting that GPT-4 may require fine-tuning rather than zero-shot prompting for financial tasks.

Research problem

The paper investigates whether advanced large language models (FinBERT, GPT-4) outperform traditional machine learning (Logistic Regression) in predicting stock market direction using financial news sentiment. It specifically addresses the trade-off between model complexity, computational efficiency, and predictive accuracy in the context of the Nigerian stock market.

Core contributions

• Empirical comparison of FinBERT, GPT-4, and Logistic Regression for binary stock index prediction.
• Demonstration that Logistic Regression with TF-IDF outperforms domain-specific and general LLMs in accuracy.
• Application of time-series cross-validation and Optuna hyperparameter optimization for robust evaluation.
• Analysis of computational costs, highlighting the inefficiency of FinBERT for real-time prediction tasks.
• Provision of a Nigerian financial news dataset and preprocessing pipeline for sentiment-based forecasting.

Method

• Data Scraping: News headlines from Nairametric and Proshare using Data Miner.
• Preprocessing: NLTK for stopword removal, tokenization, lemmatization, and noise abatement.
• Feature Extraction: TF-IDF for Logistic Regression; BERT embeddings for FinBERT; raw text for GPT-4.
• Model Training: FinBERT fine-tuned with Optuna and mixed precision; LR optimized with Optuna.
• Evaluation: Time-series cross-validation (5 folds) to maintain temporal order.
• Metrics: Classification metrics including Accuracy, Precision, Recall, F1, and ROC AUC.

Results & findings

• Logistic Regression achieved 81.83% accuracy and 89.76% ROC AUC, the highest among all models.
• FinBERT achieved 63.33% accuracy and 65.59% ROC AUC, showing moderate performance.
• GPT-4 (predefined approach) achieved 54.19% accuracy, underperforming both other models.
• FinBERT required significant computational resources (A100 GPU, ~110 mins) compared to Logistic Regression.
• Logistic Regression's simplicity and interpretability made it the most efficient model for this task.
• Time-series cross-validation was critical for reliable evaluation in temporal financial data.

Data & experiments

• Dataset: 24,923 news headlines from Nairametric and Proshare (2010-2024), aggregated into 3,573 daily observations.
• Target: Binary classification of NGX All-Share Index daily returns (Gain vs. Unchanged/Fall).
• Models: FinBERT-base, GPT-4 (API), Logistic Regression.
• Validation: Time-series cross-validation (5 folds) to prevent data leakage.
• Metrics: Accuracy, Precision, Recall, F1 Score, ROC AUC.
• Optimization: Optuna for hyperparameter tuning of FinBERT and Logistic Regression.

Limitations & caveats

• GPT-4 was used in a zero-shot/predefined manner; fine-tuning might yield better results.
• Study focuses on a single emerging market (Nigeria), limiting generalizability to developed markets.
• News sentiment alone may not capture all factors influencing stock prices (e.g., macroeconomic data).
• FinBERT's moderate performance may be due to dataset size or specific financial context of the news.
• Logistic Regression's success relies heavily on TF-IDF feature engineering, which may not scale to more complex semantic tasks.

Curator notes

Discovery status: round4_promoted. Source links: Interpretable LLMs for Credit Risk: A Systematic Review and Taxonomy