FinBERT: Financial Sentiment Analysis with Pre-trained Language Models

In one sentence

Summary

Financial sentiment analysis faces challenges due to specialized domain language and scarce labeled data, rendering general-purpose models ineffective. This paper introduces FinBERT, a BERT-based model designed to address these issues by leveraging transfer learning. The core hypothesis is that pre-training on large corpora reduces the need for extensive labeled data and allows for effective fine-tuning on domain-specific texts. The research positions FinBERT as a solution to the limitations of lexicon-based methods and standard deep learning models that struggle with semantic nuance in financial contexts. The methodology involves further pre-training BERT on a financial corpus (TRC2-financial) and fine-tuning it for classification and regression tasks. Experiments utilize the Financial PhraseBank for three-class sentiment classification and the FiQA dataset for continuous sentiment scoring. The study compares FinBERT against LSTM classifiers with GloVe/ELMo embeddings and ULMFit. Key experimental designs include investigating the impact of further pre-training, training strategies to prevent catastrophic forgetting (slanted triangular learning rates, discriminative fine-tuning, gradual unfreezing), and analyzing which encoder layers contribute most to performance. Metrics include accuracy, F1 score, MSE, and R2. FinBERT achieves state-of-the-art results, improving accuracy by 15% on Financial PhraseBank and lowering MSE on FiQA compared to prior works. The model demonstrates robustness even with small training sets (250 examples). However, further pre-training on the domain corpus showed marginal gains over vanilla BERT, likely due to high baseline performance. The model struggles with distinguishing neutral statements from positive/negative ones, particularly when implicit sentiment or numerical comparisons are involved. These findings suggest FinBERT is highly effective for explicit sentiment extraction but may require additional context for implicit financial signals.

Research problem

Financial sentiment analysis is hindered by specialized vocabulary and a lack of labeled data, making general-purpose NLP models ineffective. Traditional lexicon-based methods fail to capture deep semantic meaning, while standard deep learning models require vast amounts of labeled data. The problem is to develop a model that can effectively learn financial semantics with limited labeled data by leveraging pre-trained language models.

Core contributions

• Introduction of FinBERT, a BERT-based language model specifically adapted for financial NLP tasks.
• Achievement of state-of-the-art results on both the Financial PhraseBank classification dataset and the FiQA sentiment scoring dataset.
• Comprehensive comparison of FinBERT against ULMFit, ELMo, and LSTM baselines, demonstrating superior performance.
• Investigation of training strategies, including slanted triangular learning rates and gradual unfreezing, to prevent catastrophic forgetting during fine-tuning.
• Analysis of the impact of further pre-training on a financial corpus versus the target task dataset.

Method

• BERT architecture with 12 encoder layers, 768 hidden size, and 12 attention heads.
• Further pre-training on TRC2-financial corpus using Masked Language Modeling and Next Sentence Prediction.
• Fine-tuning for classification by adding a dense layer after the [CLS] token.
• Fine-tuning for regression using Mean Squared Error loss.
• Slanted triangular learning rate schedule to optimize convergence.
• Discriminative fine-tuning with lower learning rates for lower layers.
• Gradual unfreezing of layers starting from the top to preserve pre-trained knowledge.
• 10-fold cross-validation for robust evaluation on FiQA.

Results & findings

• FinBERT outperforms all baselines (LSTM, ULMFit, ELMo) and previous state-of-the-art methods on Financial PhraseBank.
• Accuracy on Financial PhraseBank reached 0.86 (all data) and 0.97 (100% agreement subset).
• On FiQA, FinBERT achieved an MSE of 0.07 and R2 of 0.55, beating prior models.
• FinBERT outperforms baselines even with only 250 training examples, highlighting data efficiency.
• Further pre-training on the financial corpus provided only marginal improvements over vanilla BERT.
• Fine-tuning only the last 2 layers of BERT achieved comparable performance with significantly reduced training time.
• Most errors occurred between positive and neutral labels, indicating difficulty in distinguishing objective statements from sentiment.
• Discriminative fine-tuning and gradual unfreezing were effective in preventing catastrophic forgetting.

Data & experiments

• Financial PhraseBank: 4,845 sentences annotated for sentiment (positive, negative, neutral) with varying annotator agreement levels.
• FiQA Sentiment Dataset: 1,174 financial news headlines and tweets with continuous sentiment scores ranging from -1 to 1.
• TRC2-financial: A subset of Reuters TRC2 corpus used for further pre-training, containing 46,143 documents.
• Baselines: LSTM with GloVe, LSTM with ELMo, and ULMFit classifiers.
• Evaluation metrics: Accuracy, Macro F1, Cross Entropy Loss for classification; MSE and R2 for regression.
• Experiments on varying training set sizes to assess data efficiency.

Limitations & caveats

• Further pre-training on domain corpus showed minimal improvement, possibly due to dataset size or task simplicity.
• Model struggles with implicit sentiment and numerical comparisons in financial texts.
• Evaluation on FiQA used 10-fold cross-validation rather than the official test set, limiting direct comparability.
• Results may not generalize to other financial NLP tasks like named entity recognition or question answering.
• Computational cost of fine-tuning the entire model is high, though partial fine-tuning mitigates this.
• Dependence on high-quality labeled data for fine-tuning, though less than traditional deep learning.

Curator notes

Discovery status: round1_promoted_seed_for_round2. Source links: A Preliminary Look at the State of the Art of Large Language Models on Chartered Financial Analyst Exams; A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges; A Survey of Large Language Models in Finance: FinLLMs; Adaptive Financial Sentiment Analysis for NIFTY 50 via Instruction-Tuned LLMs ,RAG and Reinforcement Learning Approaches; Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges; AlphaFin: Benchmarking