FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining

In one sentence

Summary

The paper addresses the challenge of applying deep learning to financial text mining, where labeled data is scarce and general-domain language models fail to capture financial semantics. The authors propose FinBERT, a pre-trained language model that combines general and financial corpora. Unlike standard BERT, FinBERT employs six multi-task self-supervised pre-training objectives: Span Replace Prediction, Capitalization Prediction, Token-Passage Prediction, Sentence Deshuffling, Sentence Distance, and Dialogue Relation. This design aims to better capture financial vocabulary, named entities, and semantic structures. The model is trained using mixed precision on a distributed Horovod framework to handle the large corpus size efficiently. The pre-training data includes over 61 GB of text from five sources: English Wikipedia, BooksCorpus, FinancialWeb (news), YahooFinance, and RedditFinanceQA. The model is evaluated on three downstream tasks: Financial Sentence Boundary Detection (FinSBD), Financial Sentiment Analysis (Financial PhraseBank and FiQA), and Financial Question Answering (FiQA Task 2). Experiments compare FinBERT against rule-based, BiLSTM-CRF, and standard BERT baselines. The study also includes ablation studies to assess the impact of pre-training and performance on small training data scenarios, simulating low-resource financial environments. FinBERT significantly outperforms all state-of-the-art models across all three tasks. On Sentence Boundary Detection, FinBERT Large achieved a mean F1 score of 0.970, surpassing BERT-S by 0.085. For Sentiment Analysis, it reached 0.94 accuracy on PhraseBank and superior R-squared scores on FiQA. In Question Answering, it achieved an nDCG of 0.76 and MRR of 0.68, vastly improving upon previous methods. The model demonstrates robustness even with small training corpora. However, the paper notes that collecting labeled data remains expensive, and the model's effectiveness relies heavily on the quality and diversity of the pre-training financial corpora. The source code and models are publicly available.

Research problem

Financial text mining suffers from a lack of labeled training data and the inability of general-domain NLP models to capture financial-specific semantics and vocabulary. The paper aims to bridge this gap by creating a domain-specific language model that leverages unsupervised learning on large financial corpora to improve performance on downstream financial NLP tasks.

Core contributions

• Proposes FinBERT, the first domain-specific BERT pre-trained via multi-task learning on financial corpora.
• Introduces six novel self-supervised pre-training tasks (e.g., Capitalization Prediction, Sentence Distance) to capture financial semantics.
• Demonstrates state-of-the-art results on Financial Sentence Boundary Detection, Sentiment Analysis, and Question Answering.
• Provides a scalable distributed training architecture using Horovod and mixed precision for efficient pre-training.
• Releases source code, pre-trained models, and a curated financial corpus repository.

Method

• Transformer Encoder architecture with token, segment, position, and task embeddings.
• Six multi-task self-supervised pre-training objectives: Span Replace, Capitalization, Token-Passage, Sentence Deshuffling, Sentence Distance, Dialogue Relation.
• Simultaneous training on general (Wikipedia/Books) and financial (Web/Yahoo/Reddit) corpora.
• Distributed training using Horovod framework with mixed precision (FP16) optimization.
• Fine-tuning on task-specific supervised data for SBD, SA, and QA.
• Evaluation using standard NLP metrics (F1, Accuracy, MSE, R2, nDCG, MRR).

Results & findings

• FinBERT Large achieved a mean F1 of 0.970 on FinSBD, outperforming BERT-S by 0.085.
• On Financial PhraseBank, FinBERT Large reached 0.94 Accuracy and 0.93 F1.
• On FiQA Sentiment, FinBERT Large achieved R2 of 0.77 (headlines) and 0.36 (posts).
• On FiQA QA, FinBERT Large achieved nDCG 0.76 and MRR 0.68, significantly beating baselines.
• FinBERT outperformed vanilla BERT and task-specific pre-training variants in ablation studies.
• Model showed robust performance even when pre-trained on only 1/5th of the financial data.

Data & experiments

• Pre-training data: 61GB corpus from Wikipedia, BooksCorpus, FinancialWeb, YahooFinance, and RedditFinanceQA.
• Downstream tasks: Financial Sentence Boundary Detection (FinSBD), Sentiment Analysis (PhraseBank, FiQA), and Question Answering (FiQA).
• Baselines: Rule-based, BiLSTM-CRF, Deep-Att, BERT-S, and other SOTA financial NLP models.
• Metrics: F1 scores for SBD, Accuracy/F1 for Sentiment, MSE/R2 for regression sentiment, nDCG/MRR for QA.
• Ablation studies on pre-training impact and performance with small training corpora.

Limitations & caveats

• Pre-training requires significant computational resources (hundreds of GPUs).
• Effectiveness depends on the quality and diversity of the financial corpora.
• Limited evaluation to English-language financial texts.
• Does not address real-time trading or market prediction directly.
• Performance on extremely low-resource tasks may still be constrained by label scarcity.

Curator notes

Discovery status: round2_promoted. Source links: Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges; FinBERT: Financial Sentiment Analysis with Pre-trained Language Models; FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning; Financial Report Chunking for Effective Retrieval Augmented Generation