RA-CFGPT: Chinese financial assistant with retrieval-augmented large language model

In one sentence

Summary

The paper addresses the challenge of applying large language models to the complex and knowledge-intensive domain of Chinese finance. It argues that existing financial LLMs often neglect the critical role of retrieval-augmented generation (RAG) during training, leading to insufficient handling of intricate financial terminology and compliance requirements. The authors position their work as a solution that integrates retrieval mechanisms directly into the fine-tuning process to ensure outputs are both accurate and compliant with regulatory standards. The core method involves constructing a specialized hybrid knowledge base tailored for various financial aspects. The authors fine-tune a Chinese LLM using this retrieved document context as background knowledge for multiple financial tasks. The system workflow is designed to not only generate accurate responses but also to explicitly flag associated risks, ensuring that the generated content meets strict compliance mandates. This approach contrasts with standard pre-training or supervised fine-tuning that lacks dynamic knowledge retrieval. The primary findings highlight the effectiveness of integrating RAG into the fine-tuning pipeline for Chinese financial tasks. The system demonstrates improved performance in question answering, document analysis, and risk assessment by leveraging the hybrid knowledge base. The key contribution is the establishment of a workflow that balances accuracy with regulatory compliance, addressing a significant gap in current financial LLM applications. The paper emphasizes the necessity of domain-specific knowledge bases and retrieval mechanisms for reliable financial AI systems. Additional survey-useful detail: Construction of a hybrid knowledge base tailored for diverse Chinese financial domain aspects. Construction of a proprietary hybrid knowledge base for Chinese financial data. The paper is truncated, limiting the availability of detailed quantitative results.

Research problem

The paper identifies that current financial large language models (LLMs) often overlook the knowledge-intensive nature of the financial domain. Specifically, they underutilize retrieval-augmented generation (RAG) during training, which is critical for handling complex financial terminology and ensuring compliance. The problem is how to build a Chinese financial assistant that is both accurate and compliant with regulatory mandates.

Core contributions

• Construction of a hybrid knowledge base tailored for diverse Chinese financial domain aspects.
• Fine-tuning of a Chinese LLM using retrieved documents from the hybrid knowledge base as background context.
• Development of a system workflow that ensures generated outputs meet accuracy and compliance mandates.
• Implementation of risk flagging mechanisms within the generation process to enhance safety and reliability.
• Demonstration of improved performance in financial question answering, document analysis, and risk assessment.

Method

• Construction of a hybrid knowledge base for Chinese financial data.
• Retrieval of relevant documents from the knowledge base based on user queries.
• Fine-tuning of a Chinese LLM using retrieved documents as background context.
• Integration of RAG into the training pipeline to enhance knowledge retention.
• Implementation of a workflow for compliance checking and risk flagging.
• Evaluation of the model on financial question answering and document analysis tasks.

Results & findings

• The hybrid knowledge base improves the model's understanding of complex financial terminology.
• Fine-tuning with retrieved context enhances the accuracy of financial question answering.
• The system successfully flags associated risks in generated financial documents.
• Compliance mandates are better met through the integrated RAG workflow.
• The approach addresses the gap in knowledge-intensive financial LLM applications.
• The system demonstrates robustness in handling diverse Chinese financial tasks.

Data & experiments

• Construction of a proprietary hybrid knowledge base for Chinese financial data.
• Fine-tuning experiments on a Chinese LLM using the hybrid knowledge base.
• Evaluation on financial question answering, document analysis, and risk assessment tasks.
• Assessment of compliance and risk flagging capabilities in generated outputs.
• Comparison with standard LLM approaches that do not integrate RAG during fine-tuning.

Limitations & caveats

• The paper is truncated, limiting the availability of detailed quantitative results.
• The effectiveness of the hybrid knowledge base construction is not fully detailed.
• The specific metrics for compliance and risk flagging accuracy are not provided in the text.
• The generalizability of the Chinese LLM to other languages or domains is not discussed.
• The computational cost of the RAG-integrated fine-tuning process is not analyzed.
• The paper does not compare with other state-of-the-art financial LLMs in depth.

Curator notes

Discovery status: round2_promoted. Source links: A Comprehensive Review of Generative AI in Finance; Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges; CFGPT: Chinese Financial Assistant with Large Language Model; DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning; FinBloom: Knowledge Grounding Large Language Model with Real-time Financial Data; XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters