Forecasting the S&P 500 Index Using Mathematical-Based Sentiment Analysis and Deep Learning Models: A FinBERT Transformer Model and LSTM

In one sentence

Summary

The research addresses the challenge of enhancing stock market forecasting by integrating unstructured textual data with quantitative time-series models. Specifically, it investigates whether the sentiment expressed in financial news provides predictive signal for broad market indices, positioning the work within the intersection of natural language processing and financial mathematics. The authors aim to validate the efficacy of domain-specific transformer models in extracting actionable sentiment metrics that can be mathematically mapped to market movements, thereby bridging the gap between qualitative news analysis and quantitative prediction frameworks. The methodology employs a two-stage pipeline using data from January 2018 to December 2022. First, the FinBERT transformer model is applied to daily news summaries extracted from The New York Times to classify sentiment as positive, neutral, or negative, generating a daily aggregate sentiment score. Second, a Long Short-Term Memory (LSTM) neural network is trained to predict the S&P 500 index closing price. The experimental design compares two LSTM configurations: a baseline model using only historical market variables (Open, High, Low, Close, Adj Close, Volume) and an enhanced model that includes the FinBERT sentiment score as an additional feature. Model hyperparameters are optimized via random search, and performance is evaluated using Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and Mean Absolute Error (MAE). Empirical results indicate that the inclusion of FinBERT sentiment scores consistently reduces prediction errors across all three metrics, confirming that news sentiment adds significant value to S&P 500 forecasting. The study highlights that using concise news summaries rather than full articles is a computationally efficient strategy that yields comparable predictive power. However, limitations include the reliance on a single news source (NYT), which may introduce source-specific bias, and the lack of backtesting for actual trading profitability. The findings suggest that while sentiment enhances accuracy, the model's generalizability to other markets or asset classes remains unverified, and the static nature of the pre-trained FinBERT model may not capture real-time shifts in financial language usage.

Research problem

The paper addresses the problem of improving the accuracy of stock market index forecasting by integrating unstructured textual sentiment data with traditional quantitative time-series models. It specifically investigates whether domain-specific transformer models like FinBERT can extract meaningful sentiment signals from news summaries that enhance the predictive power of LSTM networks for the S&P 500 index.

Core contributions

• Demonstrates that FinBERT-derived sentiment scores significantly improve S&P 500 prediction accuracy over baseline LSTM models using only price data.
• Validates the use of concise news summaries instead of full articles as an efficient and effective data source for sentiment analysis in financial forecasting.
• Provides empirical evidence supporting the mathematical framework for mapping transformer-based sentiment probabilities to quantitative market variables.
• Compares the efficacy of FinBERT against other sentiment analysis approaches, highlighting its superiority in the financial domain.

Method

• Data extraction of NYT news summaries using Python BeautifulSoup.
• Sentiment analysis using FinBERT, mapping labels to +1, 0, -1 weighted by classification probability.
• Calculation of daily aggregate sentiment scores by summing individual article scores.
• Feature engineering combining historical market variables with the new sentiment score.
• LSTM model training with random search hyperparameter optimization.
• Comparative evaluation using error metrics (MSE, RMSE, MAE) between baseline and enhanced models.

Results & findings

• The LSTM model with FinBERT sentiment scores outperformed the baseline model without sentiment on MSE, RMSE, and MAE.
• News summaries from NYT contained 30,797 entries, with 64.05% classified as negative, 27.9% neutral, and 8.05% positive.
• The average daily sentiment score was -0.18, indicating a general negative bias in the news corpus.
• Using summaries instead of full articles maintained predictive accuracy while reducing data volume and processing complexity.
• The study confirms that psychological factors captured in news sentiment have a measurable impact on short-term market movements.

Data & experiments

• Daily news summaries from The New York Times (2018-2022, 1826 days).
• S&P 500 index historical data (Open, High, Low, Close, Adj Close, Volume) from Yahoo Finance.
• FinBERT transformer model for sentiment classification (positive, neutral, negative) and probability extraction.
• LSTM neural network for time-series forecasting, optimized via random search.
• Evaluation metrics: MSE, RMSE, and MAE comparing models with and without sentiment features.

Limitations & caveats

• Relies on a single news source (The New York Times), which may not represent the full spectrum of market sentiment.
• Does not test the model on out-of-sample data or perform backtesting for actual trading profitability.
• The static nature of the pre-trained FinBERT model may not adapt to evolving financial language or new events.
• The study focuses on the S&P 500 index, limiting generalizability to individual stocks or other asset classes.
• Sentiment scores are aggregated daily, potentially losing intraday timing information that could be crucial for high-frequency strategies.

Curator notes

Discovery status: round3_promoted. Source links: FinBERT : A Large Language Model for Extracting Information from Financial Text†