XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters

In one sentence

Summary

The paper addresses the scarcity of open-source, large-scale language models specifically designed for the Chinese financial domain. While general-purpose models like GPT-4 and BLOOM exist, and smaller domain-specific models like FinBERT are available, there is a gap in high-parameter chat models capable of handling complex Chinese financial queries. The authors position XuanYuan 2.0 as the largest Chinese chat model to date, aiming to provide accurate, contextually appropriate responses in financial scenarios by leveraging both general linguistic capabilities and specialized financial knowledge. The core contribution is the model architecture and the proposed hybrid-tuning training framework. Built on the BLOOM-176B decoder-only architecture, the model is trained using a novel hybrid-tuning method that interleaves general pre-training, financial pre-training, and instruction tuning in a single stage. This approach uses crawled internet data for general pre-training, and a mix of unstructured (news, reports) and structured financial data for domain-specific instruction generation via Self-Instruct and Self-QA. The training leverages DeepSpeed and pipeline parallelism on NVIDIA A100 GPUs to manage the 176B parameters. Experiments compare XuanYuan 2.0 against other open-source Chinese conversational models, evaluating performance on manually assessed datasets covering general and financial dimensions. The findings indicate that XuanYuan 2.0 possesses a robust knowledge base and strong conversational capabilities in the Chinese financial domain, outperforming smaller predecessors. The hybrid-tuning method effectively mitigates catastrophic forgetting, allowing the model to retain general language skills while acquiring financial expertise. However, the paper notes that detailed quantitative evaluation rankings and further insights will be presented in a future version. Limitations include the reliance on manual assessment for current results and the need for continuous data gathering to optimize the model further. The model is primarily a foundational resource for Chinese financial NLP rather than a direct trading tool.

Research problem

The paper addresses the lack of open-source, large-scale (hundreds of billions of parameters) chat models specifically tailored for the Chinese language and the Chinese financial domain. Existing models are either too small (under 1B parameters) or lack specific financial domain adaptation, creating a gap for sophisticated conversational AI in Chinese finance.

Core contributions

• Introduction of XuanYuan 2.0, a 176-billion parameter Chinese financial chat model based on BLOOM-176B.
• Proposal of hybrid-tuning, a novel training method that integrates pre-training and instruction tuning to mitigate catastrophic forgetting.
• Development of a comprehensive dataset pipeline using Self-Instruct and Self-QA for generating financial instruction data from unstructured and structured sources.
• Demonstration of superior performance in Chinese financial conversational tasks compared to smaller domain-specific models.

Method

• Base architecture: BLOOM-176B decoder-only Transformer with ALiBi positional embeddings.
• Hybrid-tuning: Single-stage training that shuffles general pre-training, financial pre-training, and instruction data.
• Data generation: Self-Instruct for general instructions; Self-QA for financial instructions from unstructured/structured data.
• Training infrastructure: DeepSpeed distributed training with pipeline parallelism and ZeRO stage 1 optimization.
• Evaluation: Manual assessment of general and financial conversational capabilities.

Results & findings

• XuanYuan 2.0 is the largest Chinese chat model to date with 176B parameters.
• Hybrid-tuning successfully combines general and domain-specific knowledge without catastrophic forgetting.
• The model demonstrates robust knowledge base and conversational capabilities in the Chinese financial domain.
• Outperforms previous versions (XuanYuan 1.0) and smaller financial models in qualitative assessments.
• Detailed quantitative benchmarks are deferred to a future version of the paper.

Data & experiments

• Training data includes crawled internet text for general pre-training and financial news, reports, and social media for domain-specific data.
• Instruction tuning data generated using Self-Instruct (general) and Self-QA (financial) methods.
• Evaluation involves manual assessment on datasets covering general and financial dimensions.
• Comparison against other open-source Chinese conversational models and smaller financial PLMs like FinBERT and Mengzi.

Limitations & caveats

• Quantitative evaluation results are not fully detailed in this version; manual assessment is primary.
• The model is specific to the Chinese language, limiting direct applicability to English finance.
• Reliance on crawled data may introduce noise or bias not explicitly addressed.
• Computational requirements are extremely high, limiting accessibility.
• Future optimization requires continuous data gathering, indicating potential data staleness.

Curator notes

Discovery status: round1_promoted_seed_for_round2. Source links: A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges; Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges; BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark; BizBench: A Quantitative Reasoning Benchmark for Business and Finance; BloombergGPT: A Large Language Model for Finance; CFGPT: Chinese Financial Assistant with Large Language Model; DISC-FinL