FinBERT2: A Specialized Bidirectional Encoder for Bridging the Gap in Finance-Specific Deployment of Large Language Models

In one sentence

Summary

The paper addresses the practical limitations of deploying large language models (LLMs) in finance, specifically their high computational cost, slower inference speeds, and suboptimal performance on discriminative tasks like sentiment analysis and named entity recognition compared to specialized encoders. It argues that while LLMs excel at generation, they are inefficient for real-time labeling and lack mature use cases for feature-based scenarios like topic modeling. The authors propose FinBERT2, a hybrid architecture strategy where FinBERT2 serves as a domain specialist to complement LLMs, bridging the gap in finance-specific deployment by offering a lightweight, customizable, and cost-effective alternative for specialized NLP tasks. FinBERT2 is pretrained on a curated 32B-token Chinese financial corpus, the largest of its kind for this parameter size, comprising analyst reports, company announcements, and news. The model employs a finance-customized tokenizer and is fine-tuned into three variants: Fin-Labelers for five classification tasks (industry, sentiment, NER), Fin-Retrievers for dense retrieval using contrastive learning on financial QA pairs, and Fin-TopicModel for unsupervised clustering. Experiments benchmark these against general BERTs, other FinBERTs, and leading LLMs (GPT-4, Claude, Qwen) on classification accuracy and retrieval metrics (Recall@k, nDCG) across custom financial benchmarks and general domains. Results show Fin-Labelers outperform leading LLMs by 9.7%-12.3% on average across five financial classification tasks, while Fin-Retrievers surpass open-source and proprietary embedders by +6.8% and +4.2% respectively in financial retrieval. The Fin-TopicModel demonstrates superior clustering and topic representation for financial titles. The paper concludes that fine-tuned encoders are more efficient and accurate for discriminative and feature-extraction tasks, suggesting a hybrid deployment where FinBERT2 handles labeling, retrieval, and topic modeling, while LLMs focus on generative tasks, optimizing cost and performance in financial enterprises.

Research problem

The paper identifies three key limitations of LLMs in financial deployment: (1) LLMs underperform fine-tuned BERTs on discriminative tasks like sentiment analysis despite higher costs; (2) General retrievers used in RAG systems show suboptimal performance on domain-specific financial retrieval; and (3) LLMs lack mature use cases for feature-based tasks like topic modeling and factor extraction. The problem is to find a more efficient, cost-effective, and accurate alternative for these specific financial NLP tasks.

Core contributions

• Pretraining FinBERT2 on a 32B-token Chinese financial corpus, the largest known for this model size.
• Developing Fin-Labelers that outperform leading LLMs by 9.7%-12.3% on five financial classification tasks.
• Creating Fin-Retrievers that improve financial retrieval accuracy by +6.8% over open-source and +4.2% over proprietary embedders.
• Constructing Fin-TopicModel for superior unsupervised topic clustering and representation in financial texts.
• Proposing a hybrid architecture where FinBERT2 complements LLMs for specialized, cost-effective financial NLP.

Method

• Pretraining: RoBERTa-style masked language modeling on 32B financial tokens with dynamic masking.
• Tokenizer: Expanded vocabulary by 14,000 financial terms using WordPiece on financial corpus and C4.
• Fin-Labelers: Task-specific fine-tuning with AdamW optimizer on five classification datasets.
• Fin-Retrievers: Contrastive learning with InfoNCE loss, mining hard negatives, and balancing positive/negative ratios.
• Fin-TopicModel: Embedding generation via Fin-Retriever, UMAP dimensionality reduction, HDBSCAN clustering, and c-TF-IDF keyword extraction.
• Evaluation: Comprehensive testing on custom financial benchmarks and general retrieval datasets (C-MTEB).

Results & findings

• Fin-Labelers outperform LLMs by 9.7%-12.3% on average across five financial classification tasks.
• Fin-Retrievers improve retrieval by +6.8% over BGE-base-zh and +4.2% over OpenAI's text-embedding-3-large.
• FinBERT2-base achieves an average F1 of 0.9295 on classification tasks, surpassing Qwen2-72b-Instruct (0.8329).
• Fin-Retriever-large achieves Recall@10 of 0.987 on Multi-Docs-FinQA, outperforming BGE-base-zh (0.960).
• Fin-TopicModel shows superior clustering quality and topic coherence compared to baseline methods.
• Vocabulary expansion and domain pretraining significantly boost performance over general backbones.

Data & experiments

• Pretraining corpus: 32B tokens from analyst reports, company announcements, and financial news.
• Downstream datasets: Five custom financial classification datasets (industry, sentiment, NER) and five financial retrieval datasets (FIR-Bench).
• Benchmarks: Comparison against general BERTs, other FinBERTs, and LLMs (GPT-4, Claude, Qwen) on classification and retrieval tasks.
• Metrics: Weighted F1 for classification, Recall@k and nDCG@10 for retrieval, and clustering metrics for topic modeling.
• Ablation studies on pretraining data volume, tokenizer expansion, and contrastive learning hyperparameters.

Limitations & caveats

• FinBERT2 is specialized for Chinese financial text, limiting direct applicability to other languages.
• The model is not designed for generative tasks, requiring LLMs for those use cases.
• Performance on general-domain retrieval tasks is slightly lower than some general embedders.
• The study focuses on classification and retrieval, not on downstream trading or portfolio optimization performance.
• The corpus, while large, is curated and may not capture all real-time or niche financial information.

Curator notes

Discovery status: specific_domain_focus_promoted. Source links: large language models sector analysis finance; large language models ETF finance