When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain

In one sentence

Summary

The paper addresses the limitation of existing financial language models that fail to fully leverage the richness of financial data through generic pre-training schemes. The authors argue that financial terminology, often consisting of multi-token phrases, requires specialized handling to capture domain-specific semantics effectively. They propose a novel pre-training methodology that incorporates preferential masking of financial keywords and phrases, alongside a span boundary objective to learn robust multi-word representations. This approach aims to bridge the gap between general English language models and the specialized linguistic patterns found in financial texts, such as earnings calls and regulatory filings, which often contain nuanced sentiment and complex entity relationships not present in standard corpora. To evaluate these models, the authors introduce FLUE (Financial Language Understanding Evaluation), a comprehensive benchmark suite comprising five distinct NLP tasks: financial sentiment classification and regression, news headline classification, named entity recognition, structure boundary detection, and financial question answering. The models, FLANG-BERT and FLANG-ELECTRA, are pre-trained on a diverse mix of general English data (Wikipedia, BooksCorpus) and financial datasets including SEC 10-K/10-Q filings, earnings call transcripts, analyst reports, and financial news from Reuters and Bloomberg. Experiments show that FLANG-ELECTRA achieves state-of-the-art results across most benchmarks, significantly outperforming baselines like FinBERT and standard BERT, particularly in tasks requiring deep financial understanding such as sentiment analysis and question answering. The findings highlight that domain-specific pre-training yields substantial performance gains, with FLANG-ELECTRA showing marked improvements in sentiment regression (lower MSE) and classification accuracy compared to prior works. However, the paper notes that improvements are muted on domain-agnostic tasks like Named Entity Recognition, suggesting that domain-specific knowledge is most critical for tasks involving semantic interpretation of financial language. The authors release all models, code, and the FLUE benchmark suite to the community, emphasizing the generalizability of their masking strategy to other domains. A key caveat is that while the method improves performance, it relies on publicly available data, and the benefits may vary depending on the specific downstream task's reliance on financial jargon versus general linguistic structure.

Research problem

Existing financial language models often use generic pre-training schemes that do not fully exploit the richness of financial data, particularly multi-token financial phrases. Additionally, the field lacks comprehensive evaluation benchmarks, limiting the ability to assess model performance across diverse financial NLP tasks. The paper seeks to develop a domain-specific pre-training methodology and a robust benchmark suite to address these gaps.

Core contributions

• Proposes a pre-training methodology with preferential masking of financial words and phrases, and a span boundary objective for robust multi-word representation.
• Introduces FLUE, a comprehensive benchmark suite for financial NLP covering sentiment analysis, news classification, NER, structure boundary detection, and question answering.
• Develops and releases FLANG-BERT and FLANG-ELECTRA, two new state-of-the-art financial language models.
• Demonstrates that domain-specific masking and pre-training significantly outperform prior financial models like FinBERT on most benchmarks.
• Provides ablation studies validating the effectiveness of each component of the proposed training strategy.

Method

• Pre-training uses a mix of general and financial corpora.
• Preferential masking: Financial terms are masked with higher probability (30%) than non-financial terms.
• Phrase masking: Multi-token financial phrases are masked as single units to learn phrase representations.
• Span boundary objective: Added to ELECTRA generator to predict tokens within masked spans based on boundary tokens.
• Multi-stage training: First stage masks single financial words; second stage masks both words and phrases.
• Fine-tuning uses supervised contrastive loss in addition to cross-entropy for classification tasks.

Results & findings

• FLANG-ELECTRA achieves state-of-the-art results on financial sentiment classification (91.9% accuracy on FPB) and regression (0.034 MSE on FiQA).
• FLANG-BERT outperforms FinBERT on nearly all benchmarks, with significant gains in sentiment analysis and question answering.
• Preferential masking of 30% of financial terms yields the lowest perplexity during pre-training.
• Multi-stage masking (single words then phrases) further reduces perplexity compared to single-stage masking.
• Improvements are muted on domain-agnostic tasks like NER, suggesting domain knowledge is most valuable for semantic tasks.
• FLANG models show lower perplexity than baselines, indicating better language modeling capabilities on financial text.

Data & experiments

• Pre-training data includes general English (Wikipedia, BooksCorpus) and financial sources (SEC filings, earnings calls, analyst reports, Reuters/Bloomberg news, Investopedia).
• FLUE benchmark includes 5 tasks: Financial PhraseBank sentiment classification, FiQA sentiment regression, Gold news headline classification, Financial NER, FinSBD3 structure boundary detection, and FiQA question answering.
• Experiments compare FLANG-BERT and FLANG-ELECTRA against baselines like BERT, FinBERT, and ELECTRA.
• Evaluation metrics include Accuracy, MSE, R2, F1, nDCG, and MRR across various tasks.
• Ablation studies analyze the impact of preferential masking, phrase masking, span boundary objective, and contrastive loss.

Limitations & caveats

• Improvements are limited on domain-agnostic tasks like NER, suggesting the method's benefits are task-dependent.
• Relies on publicly available financial data, which may not capture all nuances of private or real-time financial information.
• The FLUE benchmark, while comprehensive, may not cover all emerging financial NLP tasks.
• The study focuses on BERT and ELECTRA architectures; generalization to larger models (e.g., LLaMA) is not explicitly tested.
• Ethical risks are noted as low, but the use of financial data always carries potential for bias or misuse.

Curator notes

Discovery status: round2_promoted. Source links: Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges; Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks; CFGPT: Chinese Financial Assistant with Large Language Model; Empowering Many, Biasing a Few: Generalist Credit Scoring through Large Language Models; FinBERT: A Pretrained Language Model for Financial Communications; FinBERT: Financial Sentiment Analysis with Pre-trained Language Models; FinC