DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning

In one sentence

Summary

This paper addresses the lack of open-source, domain-specific large language models for the Chinese financial market, where general LLMs often lack specialized knowledge, numerical reasoning capabilities, and real-time information processing skills. The authors propose DISC-FinLLM, a system designed to support financial professionals, developers, and students by integrating multi-turn dialogue, complex text processing, mathematical computation, and retrieval-augmented generation. The core innovation is the Multiple Experts Fine-tuning Framework (MEFF), which avoids catastrophic forgetting by training four distinct Low-rank adaptation (LoRA) modules on specialized instruction subsets, allowing for modular activation of capabilities without retraining the entire base model. The methodology centers on constructing DISC-FIN-SFT, a comprehensive instruction-tuning dataset of approximately 250,000 samples. This dataset is divided into four categories: financial consulting (translated and augmented FiQA, forum data), financial NLP tasks (derived from ten public datasets like FPB and CCKS, plus reading comprehension from East Money news), financial computing (seed tasks augmented via self-instruction and Chain-of-Thought prompting to train tool-use for calculators and solvers), and retrieval-enhanced generation (using Chain-of-Retrieval prompting on a knowledge base of 87k news and report abstracts). The model is built on the Baichuan-13B backbone, with each LoRA module fine-tuned on its respective data subset to handle specific tasks such as sentiment analysis, entity extraction, formula construction, and document-based Q&A. Experimental evaluations across multiple benchmarks demonstrate that DISC-FinLLM significantly outperforms baseline models like ChatGLM2 and FinGPT-v3. On the FinCUGE benchmark, the task-specific LoRA improved average F1/ROUGE scores by 2-9 points over the untrained base. In human-generated financial tests (FinEval), the consulting and task modules achieved higher accuracy than GPT-3.5 and matched GPT-4 in specific sub-domains. The computing module surpassed ChatGPT-3.5 in formula construction and result accuracy, while the retrieval module showed superior accuracy, usefulness, and reflectiveness in current affairs analysis. However, the study notes limitations in evaluating zero-shot generalization due to the scarcity of high-quality Chinese financial NLP datasets, and the reliance on ChatGPT for data generation introduces potential biases or quality inconsistencies in the instruction tuning corpus.

Research problem

General-purpose large language models lack the specialized financial knowledge, numerical reasoning, and real-time retrieval capabilities required for the Chinese financial market. Traditional financial NLP models are limited by data scarcity, single-task focus, and poor cross-task generalization. There is a need for an open-source, multi-capable financial LLM that can handle consulting, computation, and retrieval simultaneously without catastrophic forgetting.

Core contributions

• Proposes the Multiple Experts Fine-tuning Framework (MEFF) using modular LoRA adapters to enable distinct financial capabilities without retraining the base model.
• Constructs DISC-FIN-SFT, a 250k-sample instruction dataset covering consulting, NLP tasks, computing, and retrieval-augmented generation.
• Integrates tool-use plugins for financial calculations (calculator, equation solver, counter, probability table) via Chain-of-Thought prompting.
• Implements a retrieval-enhanced generation pipeline using Chain-of-Retrieval (CoR) prompting on a proprietary financial knowledge base.
• Demonstrates that modular fine-tuning outperforms full-parameter fine-tuning and existing baselines like FinGPT-v3 on financial benchmarks.

Method

• Base model: Baichuan-13B, a general domain Chinese LLM.
• Multiple Experts Fine-tuning Framework (MEFF): Four independent LoRA modules trained on specialized data subsets.
• Data construction: Translation of English datasets, self-chat prompting for forum data, Chain-of-Thought for computing, Chain-of-Retrieval for QA.
• Tool integration: Model learns to generate tool call commands for calculator, equation solver, counter, and probability table.
• Retrieval pipeline: Question generation from documents, retrieval of top-k references, answer generation with irrelevant document filtering.
• Evaluation: Few-shot prompting for NLP tasks, manual creation of calculation tests, GPT-3.5 automated evaluation for retrieval quality.

Results & findings

• On FinCUGE, the task-specific LoRA improved average performance by 2-9 points over the untrained Baichuan-13B-Chat base.
• In FinEval human tests, DISC-FinLLM (Consulting) achieved 51.6% average accuracy, outperforming FinGPT-v3 (49.6%) and ChatGLM2 (47.4%).
• The computing module achieved 0.35 accuracy in formula and result calculation, surpassing ChatGPT-3.5 (0.26) and Baichuan-13B-Chat (0.12).
• The retrieval module scored higher than the base model in accuracy (4.13 vs 4.08), usefulness (4.29 vs 4.15), and reflectiveness (3.95 vs 3.88) on current affairs analysis.
• Ablation studies showed that full-parameter fine-tuning on all data resulted in lower performance than the modular LoRA approach, validating the MEFF framework.
• The model effectively handles multi-turn dialogues and complex financial queries, demonstrating robustness in the Chinese financial context.

Data & experiments

• DISC-FIN-SFT dataset: 250k samples including 63k consulting, 110k NLP tasks, 57k computing, and 20k retrieval instructions.
• Data sources: Translated FiQA, Jing-Guan forum posts, 10 public Chinese financial NLP datasets (FPB, CCKS, etc.), East Money news, and research reports.
• Benchmarks: FinCUGE (sentiment, extraction, summarization), FinEval (human-generated finance/economy/accounting questions), custom calculation and current affairs datasets.
• Evaluation metrics: Accuracy, F1, ROUGE for NLP; accuracy for computation; GPT-3.5 judged metrics (accuracy, usefulness, linguistic quality, reflectiveness) for retrieval.
• Baselines: Baichuan-13B, ChatGLM, ChatGLM2, GPT-3.5, GPT-4, FinGPT-v3.

Limitations & caveats

• Reliance on ChatGPT for data generation (translation, answer generation) may introduce biases or factual errors into the training data.
• Limited ability to evaluate zero-shot generalization due to the scarcity of high-quality, unseen Chinese financial NLP datasets.
• The model is based on Baichuan-13B, which may have inherent limitations compared to larger or more recent base models.
• Retrieval performance depends on the quality and coverage of the proprietary knowledge base (87k documents from 2023).
• Computational tools are limited to four specific plugins, potentially restricting complex financial modeling capabilities.

Curator notes

Discovery status: round1_promoted_seed_for_round2. Source links: A Comprehensive Review of Generative AI in Finance; A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges; Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges; BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark; Benchmarking Large Language Models on CFLUE - A Chinese Financial Language Understanding Evaluation Dataset; BizFinBench: A Business-Driven