BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark

In one sentence

Summary

This paper addresses the lack of high-quality resources for Chinese financial natural language processing by introducing BBT-FinT5, a domain-specific pre-trained language model based on the T5 architecture. The authors argue that existing Chinese financial models are limited by smaller parameter sizes and lack of knowledge enhancement, particularly for entity understanding. To solve this, they propose a knowledge-enhanced pre-training method called Triple Masking (KETM), which integrates knowledge graph triples with text spans to improve entity memorization. The work positions itself as a comprehensive infrastructure contribution, aiming to standardize evaluation and improve baseline performance for downstream financial NLP applications in the Chinese market. The methodology involves constructing BBT-FinCorpus, a 300GB dataset comprising corporate announcements, research reports, financial news, and social media posts from major Chinese platforms. The BBT-FinT5 model, available in base (220M) and large (1B) parameter versions, is pre-trained using DeepSpeed optimization and BFLOAT16 precision. Evaluation is conducted on BBT-CFLEB, a new benchmark with six tasks including news classification, relation extraction, sentiment analysis, and question answering. Experiments compare BBT-FinT5 against general models (T5, GPT-2) and existing financial models (FinBERT, Mengzi-BERT), using metrics like F1-score and ROUGE across understanding and generation leaderboards. Results show that BBT-FinT5 significantly outperforms baselines, with the large version achieving the highest average scores on the CFLEB benchmark. The KETM method specifically boosts performance on relation extraction and summarization tasks, confirming the value of explicit knowledge integration. The released corpus and benchmark provide essential tools for the Chinese financial NLP community. However, the study is limited to Chinese text and does not address multimodal or multilingual aspects, nor does it evaluate real-time trading performance or market impact, focusing instead on static text understanding and generation capabilities.

Research problem

The paper identifies a critical gap in Chinese financial NLP: existing models are small, lack knowledge-enhanced pre-training strategies suitable for T5 architectures, and there is no comprehensive, open benchmark for evaluating financial language understanding and generation in Chinese. The authors aim to provide a large-scale corpus, a state-of-the-art pre-trained model, and a standardized benchmark to advance the field.

Core contributions

• Introduction of BBT-FinT5, the largest Chinese financial pre-trained language model with 1 billion parameters.
• Development of BBT-FinCorpus, a 300GB diverse corpus covering announcements, reports, news, and social media.
• Proposal of BBT-CFLEB, a benchmark with six tasks for evaluating financial language understanding and generation.
• Design of KETM, a knowledge-enhanced pre-training method using triple masking for T5 models.
• Release of open-source code, models, and data to facilitate community research in Chinese financial NLP.

Method

• Data collection from Sina Finance, Tencent Finance, Eastmoney, Guba, and Xueqiu.
• Text cleaning and PDF parsing to convert announcements and reports into raw text.
• Pre-training on T5 architecture using span masking and text-to-text paradigm.
• Knowledge Enhancement via Triple Masking (KETM): concatenating knowledge graph triples with text and masking elements.
• Training acceleration using DeepSpeed with BFLOAT16 precision to prevent gradient overflow.
• Evaluation on six specific financial NLP tasks with expert-validated practicality scores.

Results & findings

• BBT-FinT5-large outperforms all baselines on the CFLEB benchmark, achieving an average score of 77.07.
• The knowledge-enhanced version (FinT5-base-KE) shows significant improvements in relation extraction and summarization compared to the standard FinT5-base.
• Domain pre-training on BBT-FinCorpus yields substantial gains over general-domain T5-base, confirming corpus quality.
• BBT-FinT5 surpasses existing financial models like FinBERT and Mengzi-BERT in understanding tasks.
• Scaling parameters from 220M to 1B results in consistent performance improvements across all tasks.
• The benchmark effectively distinguishes between understanding and generation capabilities, revealing model strengths and weaknesses.

Data & experiments

• BBT-FinCorpus: 300GB of raw text from corporate announcements, research reports, financial news, and social media.
• BBT-CFLEB Benchmark: Six datasets including FinNL (classification), FinNA (summarization), FinRE (relation extraction), FinFE (sentiment), FinQA (question answering), and FinNSP (negative news detection).
• Comparison models: T5-base, GPT2-base, FinBERT, and Mengzi-BERT-base-fin.
• Evaluation metrics: F1-score for classification/extraction, ROUGE for summarization, and accuracy for sentiment.
• Experiments on understanding and generation leaderboards to assess model capabilities across different task types.

Limitations & caveats

• The corpus and model are limited to the Chinese language, restricting generalizability to other markets.
• The benchmark tasks are static and do not reflect real-time market dynamics or high-frequency trading scenarios.
• Knowledge graph integration relies on distant supervision, which may introduce noise.
• The study does not evaluate the model's performance on downstream trading or investment decision-making tasks.
• Social media data may contain noise and bias that are not fully addressed in the cleaning process.

Curator notes

Includes large Chinese financial corpus and benchmark.