CFGPT: Chinese Financial Assistant with Large Language Model

In one sentence

Summary

The paper addresses the lack of robust, open-source large language models specifically tailored for the Chinese financial domain, where existing general-purpose LLMs often struggle with domain-specific jargon, regulatory nuances, and complex analytical tasks. The authors position CFGPT as a comprehensive solution that bridges the gap between general NLP capabilities and specialized financial requirements, aiming to provide a deployable assistant for real-world financial applications rather than just a research benchmark. This work contributes to the broader field of multilingual and multimodal finance by demonstrating how domain-specific data curation and two-stage training can significantly enhance model performance in Chinese financial contexts. The core methodology involves constructing CFData, a massive corpus containing 584 million documents and 141 billion tokens for pre-training, sourced from corporate prospectuses, announcements, research reports, news, and social media, alongside a supervised fine-tuning dataset of 1.5 million instruction pairs covering six tasks: sentiment analysis, event detection, report summarization, topic decomposition, question answering, and stock movement prediction. The base model, InternLM-7B, undergoes continued pre-training on CFData followed by supervised fine-tuning. The deployment framework, CFAPP, integrates vector databases, chain-of-thought reasoning modules, and multimodal parsers (text, audio, PDF) to facilitate interactive financial analysis, including causal reasoning for open-domain questions and risk management tools. Experimental results indicate that CFLLM outperforms baseline models in zero-shot and few-shot financial tasks, particularly in sentiment analysis and event detection, validating the efficacy of the curated dataset and training strategy. The CFAPP framework demonstrates practical utility in handling diverse input formats and generating structured outputs like mind maps and templated reports. However, limitations include the reliance on GPT-4 for generating some synthetic training data, potential biases in social media sources, and the computational cost of deploying a 7B-parameter model. The paper highlights the importance of high-quality, domain-specific data in enhancing LLMs for finance, offering a valuable resource for researchers and practitioners working with Chinese financial text.

Research problem

The paper addresses the scarcity of high-quality, open-source large language models specifically designed for the Chinese financial domain. While general LLMs show promise, they often lack the depth of understanding required for complex financial texts, regulatory documents, and nuanced market analysis in Chinese. The authors aim to fill this gap by creating a specialized framework that includes domain-specific data, a fine-tuned model, and a practical deployment system.

Core contributions

• Introduction of CFData, a large-scale Chinese financial dataset with 141B tokens for pre-training and 1.5B tokens for supervised fine-tuning.
• Development of CFLLM, a 7B-parameter financial LLM based on InternLM-7B, trained via continued pre-training and supervised instruction tuning.
• Creation of CFAPP, a multimodal deployment framework supporting text, audio, and PDF inputs with integrated reasoning and retrieval modules.
• Construction of a supervised fine-tuning dataset covering six key financial tasks: sentiment analysis, event detection, summarization, topic decomposition, QA, and stock prediction.
• Release of open-source code and models to foster research in Chinese financial NLP.

Method

• Data collection from multiple sources including stock exchanges, financial news sites, and social media platforms.
• Preprocessing steps including PDF text extraction, noise filtering, and deduplication using LSH.
• Continued pre-training of InternLM-7B on the CFData corpus using causal language modeling.
• Supervised fine-tuning on instruction pairs for six specific financial tasks.
• Integration of vector databases and retrieval-augmented generation (RAG) techniques in CFAPP.
• Implementation of chain-of-thought reasoning modules for complex financial question answering.

Results & findings

• CFData comprises 584 million documents and 141 billion tokens for pre-training, significantly larger than many existing Chinese financial corpora.
• The supervised fine-tuning dataset includes 1.5 million instruction pairs, covering a wide range of financial analysis tasks.
• CFLLM shows improved performance in financial sentiment analysis and event detection compared to general-purpose LLMs.
• The two-stage training process (continued pre-training followed by supervised fine-tuning) proves effective for domain adaptation.
• CFAPP successfully integrates vector databases and chain-of-thought reasoning to enhance response accuracy and relevance.
• The model demonstrates strong capabilities in report summarization and topic decomposition, key tasks for financial research.

Data & experiments

• Pre-training dataset (CFData) includes corporate prospectuses, announcements, research reports, news, and social media posts from 2012-2022.
• Supervised fine-tuning dataset contains 1.5 million instruction pairs for six distinct financial tasks.
• Evaluation on financial sentiment analysis, event detection, and report summarization tasks.
• Comparison against baseline models like GPT-4, LLaMA, and other financial LLMs in zero-shot and few-shot settings.
• Demonstration of CFAPP's capability in handling multimodal inputs and generating structured outputs like mind maps.

Limitations & caveats

• Reliance on GPT-4 for generating synthetic training data for sentiment analysis may introduce biases.
• Social media data may contain noise, biases, and unverified information, affecting model robustness.
• The 7B-parameter model size may limit performance on highly complex reasoning tasks compared to larger models.
• Computational resources required for training and deployment are significant.
• Evaluation is primarily focused on Chinese financial tasks, limiting generalizability to other languages or domains.

Curator notes

Discovery status: round1_promoted_seed_for_round2. Source links: A Comprehensive Review of Generative AI in Finance; A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges; Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges; BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark; BloombergGPT: A Large Language Model for Finance; ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question A