FLAME: Financial Large Language Model Evaluation System in Chinese

In one sentence

Summary

The paper addresses the lack of professional, Chinese-language benchmarks for evaluating financial Large Language Models (LLMs), arguing that existing NLP-centric benchmarks fail to capture domain-specific expertise and practical applicability. The authors propose FLAME, a holistic evaluation system designed to assess both theoretical knowledge and real-world business capabilities within the Chinese financial context, ensuring alignment with regulatory standards and industry needs. This system aims to provide a reliable standard for developing and selecting financial LLMs that are both academically rigorous and commercially viable. FLAME consists of two core benchmarks: FLAME-Cer and FLAME-Sce. FLAME-Cer contains approximately 16,000 manually reviewed multiple-choice questions covering 14 authoritative financial certifications, including CPA, CFA, and FRM, graded by professional depth. FLAME-Sce features nearly 100 tertiary tasks across 10 primary business scenarios, such as risk control, document generation, and customer service, evaluated via a weighted multi-dimensional manual scoring system covering accuracy, compliance, and practicality. The authors evaluate six representative LLMs, including GPT-4o, Qwen2.5, and Baichuan4-Finance, using zero-shot settings to compare their performance across these diverse and complex financial tasks. Results indicate that Baichuan4-Finance leads with an average accuracy of 93.62% on certifications and 84.15% usability on scenarios, significantly outperforming generalist models like GPT-4o and other open-source models. While models excel in data processing and knowledge retrieval, they struggle with complex financial analysis and document generation, highlighting current limitations in deep reasoning. The study underscores the importance of domain-specific fine-tuning for Chinese financial applications and provides a valuable benchmark for future model development, though it notes that manual evaluation remains resource-intensive and potentially subjective.

Research problem

Current financial LLM benchmarks primarily evaluate models from an NLP perspective, lacking comprehensive assessment of professional financial knowledge and practical business application capabilities, particularly in the Chinese language context. There is a need for a rigorous, industry-aligned evaluation system that tests both theoretical expertise and real-world compliance and risk management skills.

Core contributions

• Introduction of FLAME, a comprehensive Chinese financial LLM evaluation system.
• Development of FLAME-Cer, a benchmark with 16,000 questions across 14 professional certifications.
• Creation of FLAME-Sce, a scenario-based benchmark with nearly 100 tasks across 10 business domains.
• Establishment of a multi-dimensional manual evaluation framework with weighted scoring for complex tasks.
• Comprehensive empirical evaluation of six leading LLMs, identifying Baichuan4-Finance as the top performer.

Method

• Construction of FLAME-Cer using questions from 14 authoritative financial certifications.
• Manual review of all certification questions to ensure accuracy and representativeness.
• Design of FLAME-Sce with hierarchical scenarios (primary, secondary, tertiary).
• Development of a weighted multi-dimensional scoring system for FLAME-Sce.
• Dimensions include accuracy, completeness, compliance, relevance, professionalism, clarity, and practicality.
• Zero-shot evaluation of six LLMs on both benchmarks.
• Comparison of model performance across different certification types and business scenarios.

Results & findings

• Baichuan4-Finance achieves the highest average accuracy on FLAME-Cer (93.62%).
• Qwen2.5-72B-Instruct ranks second on FLAME-Cer (88.24%).
• Baichuan4-Finance leads in FLAME-Sce usability (84.15%).
• GPT-4o and Qwen2.5-72B-Instruct perform similarly on FLAME-Sce (~79%).
• All models struggle with the CAA (Actuarial) exam, with best accuracy only 66.51%.
• Models perform well in data processing but poorly in financial analysis and document generation.
• Baichuan4-Finance excels in customer service and risk control scenarios.
• ERNIE-4.0-Turbo-128K stands out in financial marketing scenarios (93.94%).

Data & experiments

• FLAME-Cer: 16,000 manually reviewed questions from 14 certifications (CPA, CFA, FRM, etc.).
• FLAME-Sce: Nearly 100 tasks across 10 primary scenarios (e.g., risk control, compliance, marketing).
• Evaluation of 6 LLMs: GPT-4o, GLM-4, ERNIE-4.0, Qwen2.5, XuanYuan3, Baichuan4-Finance.
• Zero-shot evaluation setting for all models.
• Metrics: Accuracy for FLAME-Cer; weighted multi-dimensional manual scoring for FLAME-Sce.
• Case studies provided for each certification and primary scenario to illustrate evaluation criteria.

Limitations & caveats

• Manual evaluation for FLAME-Sce is resource-intensive and potentially subjective.
• The benchmark focuses on Chinese language, limiting generalizability to other markets.
• Some complex tasks like actuarial science remain challenging for all LLMs.
• Performance may vary with different prompt engineering or fine-tuning strategies.
• The study does not evaluate real-time market data integration or execution capabilities.

Curator notes

Includes certification-style and scenario-style tests.