Large Language Model Evaluation on Financial Benchmarks

In one sentence

Summary

The paper addresses the lack of comprehensive, professional evaluation standards for Large Language Models in the financial domain, particularly within the Chinese context. Existing benchmarks often rely on NLP metrics like text similarity rather than assessing practical financial capabilities. To fill this gap, the authors propose FLAME, a dual-benchmark system designed to rigorously test both theoretical knowledge and real-world application skills of financial LLMs. The system is developed by the School of Finance at Renmin University of China to ensure academic and professional authority, covering everything from basic terminology to complex regulatory compliance and risk control scenarios. FLAME consists of two core components: FLAME-Cer and FLAME-Sce. FLAME-Cer contains approximately 16,000 manually reviewed multiple-choice questions covering 14 authoritative financial certifications, including CPA, CFA, and FRM, with accuracy as the primary metric. FLAME-Sce features nearly 100 tertiary tasks across 10 primary business scenarios, such as intelligent customer service, document generation, and risk control, evaluated using a multi-dimensional manual scoring system that weighs accuracy, compliance, and practicality. The authors evaluate six representative LLMs, including GPT-4o, Qwen2.5, and Baichuan4-Finance, using zero-shot settings to compare their performance across these diverse and rigorous benchmarks. The results indicate that Baichuan4-Finance leads in most tasks, achieving an average accuracy of 93.62% on FLAME-Cer and a usability rate of 84.15% on FLAME-Sce. While models generally perform well in financial knowledge theory and data computation, they struggle with complex tasks like financial analysis and document generation. The study highlights that specialized financial models often outperform general-purpose LLMs in domain-specific contexts, though challenges remain in handling nuanced regulatory compliance and deep analytical reasoning. The paper concludes that FLAME provides a vital tool for advancing the development and deployment of reliable financial LLMs in China.

Research problem

Current financial LLM benchmarks primarily evaluate models from an NLP perspective, such as text similarity or n-gram overlap, failing to assess practical financial capabilities or professional knowledge. There is a lack of comprehensive, authoritative evaluation systems that cover both theoretical certification knowledge and real-world business scenarios, particularly for the Chinese financial market. This gap makes it difficult for financial institutions to reliably select and deploy LLMs for specific professional tasks.

Core contributions

• Introduction of FLAME, a comprehensive Chinese financial LLM evaluation system comprising two distinct benchmarks.
• Development of FLAME-Cer, a benchmark with 16,000 manually reviewed questions covering 14 major financial certifications.
• Creation of FLAME-Sce, a scenario-based benchmark with nearly 100 tasks across 10 primary business areas like risk control and customer service.
• Establishment of a multi-dimensional manual evaluation framework for FLAME-Sce that weights accuracy, compliance, and practicality.
• Empirical evaluation of six representative LLMs, providing a performance baseline for the Chinese financial LLM landscape.
• Release of the FLAME dataset and evaluation instructions on GitHub to facilitate community research and model improvement.

Method

• Construction of FLAME-Cer by curating questions from 14 authoritative financial certification exams.
• Manual review of all FLAME-Cer questions to ensure accuracy and representativeness.
• Design of FLAME-Sce with a hierarchical structure: 10 primary, 21 secondary, and ~100 tertiary scenarios.
• Development of a weighted scoring system for FLAME-Sce incorporating dimensions like accuracy, compliance, and practicality.
• Zero-shot evaluation of six LLMs on both benchmarks without fine-tuning or few-shot prompting.
• Comparison of model performance across different certification types and business scenarios.
• Case studies provided to illustrate specific task types and model outputs for both benchmarks.

Results & findings

• Baichuan4-Finance achieves the highest average accuracy on FLAME-Cer at 93.62%, significantly outperforming GPT-4o (78.23%).
• Baichuan4-Finance leads in FLAME-Sce with a usability rate of 84.15%, compared to GPT-4o's 79.88%.
• All models struggle with the CAA (Actuarial) exam, with the best score being only 66.51%, indicating difficulty in complex quantitative reasoning.
• Models perform best in Fund and Securities Qualification exams, with Baichuan4-Finance exceeding 97% accuracy.
• In FLAME-Sce, models excel in 'Financial Knowledge and Theory' (>80% usability) but lag in 'Financial Analysis and Research' (~45%).
• Qwen2.5-72B-Instruct shows strong performance in data calculation (91.61%), while ERNIE-4.0 leads in marketing (93.94%).
• XuanYuan3-70B-Chat underperforms significantly in FLAME-Sce (61.34%), suggesting gaps in practical application despite financial tuning.
• The study confirms that specialized financial models (Baichuan4-Finance) are currently superior to general models in Chinese financial contexts.

Data & experiments

• FLAME-Cer dataset: ~16,000 questions from 14 certifications (CPA, CFA, FRM, etc.), manually reviewed for accuracy.
• FLAME-Sce dataset: ~5,000 questions across 10 primary, 21 secondary, and nearly 100 tertiary financial business scenarios.
• Models evaluated: GPT-4o, GLM-4-PLUS, ERNIE-4.0-Turbo-128K, Qwen2.5-72B-Instruct, XuanYuan3-70B-Chat, Baichuan4-Finance.
• Evaluation setting: Zero-shot inference for all models.
• Metrics: Accuracy for FLAME-Cer; multi-dimensional weighted manual scoring for FLAME-Sce.
• Scenarios include: financial knowledge, compliance, document generation, customer service, risk control, and data processing.

Limitations & caveats

• The evaluation is limited to the Chinese language and Chinese financial regulatory context.
• FLAME-Cer relies on multiple-choice questions, which may not fully capture generative capabilities.
• FLAME-Sce evaluation involves manual scoring, which can be subjective and costly to scale.
• The study evaluates only six models, limiting the generalizability of the findings to the broader LLM landscape.
• Zero-shot setting may not reflect real-world performance where fine-tuning or RAG is used.
• The 'Analysis and Research' scenario shows low performance, suggesting current LLMs are not yet ready for deep financial analysis.

Curator notes

Includes English and Japanese financial benchmark discussion.