FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models

In one sentence

Summary

The paper addresses the lack of comprehensive evaluation for Large Language Models in the Chinese financial domain, specifically highlighting gaps in security and complex agent tasks. It introduces FinEval, a benchmark comprising 8,351 questions across four categories: Financial Academic Knowledge (4,661 questions across 34 subjects), Financial Industry Knowledge (1,434 questions on practical scenarios like investment research), Financial Security Knowledge (1,640 questions on application and network security), and Financial Agent (616 questions on tool usage and reasoning). The dataset is curated by finance and security experts to ensure high quality and relevance to real-world Chinese financial contexts. The authors evaluate 19 LLMs, including closed-source models like Claude 3.5-Sonnet and GPT-4o, open-source models like Qwen2.5-72B, and financial-specific models like XuanYuan3-70B. Experiments utilize zero-shot, five-shot, and Chain-of-Thought prompting. Evaluation metrics include accuracy for multiple-choice questions, Rouge-L for industry knowledge, and GPT-4o as a judge for open-ended agent tasks. The study also includes a human baseline comparison with ordinary individuals and financial experts to contextualize model performance. Results show Claude 3.5-Sonnet achieves the highest weighted average score of 72.9, followed by GPT-4o at 71.9. Notably, general open-source models like Qwen2.5-72B often outperform specialized financial models, suggesting superior generalization. While LLMs significantly outperform ordinary individuals (average score 30.1), they still trail financial experts (average score 85.9). Error analysis reveals that open-source models struggle more with logical reasoning and context, while closed-source models are prone to ambiguity handling issues. The benchmark highlights that despite progress, LLMs require further improvement in complex financial reasoning and security robustness.

Research problem

Existing benchmarks lack comprehensive coverage of financial security and complex agent tasks in the Chinese context. There is a need to evaluate LLMs not just on academic knowledge but on practical industry applications, security vulnerabilities, and multi-step reasoning capabilities required in real-world financial scenarios.

Core contributions

• Introduction of FinEval, a comprehensive benchmark with 8,351 questions covering academic, industry, security, and agent domains.
• First inclusion of financial security and financial agent tasks in a large-scale Chinese financial evaluation.
• Establishment of human baselines comparing LLMs against ordinary individuals and financial experts.
• Extensive evaluation of 19 diverse LLMs, providing detailed performance breakdowns across multiple prompting strategies.
• Error analysis categorizing model failures into logical reasoning, contextual misunderstanding, and ambiguity handling.

Method

• Data collection from mock exams, financial websites, and expert curation.
• Quality control via consensus review by three finance/security experts.
• Conversion of mathematical formulas to LaTeX for consistent processing.
• Evaluation using multiple prompting strategies: zero-shot, few-shot, and CoT.
• Use of GPT-4o as a judge for open-ended agent and industry tasks.
• Rouge-L metric for short-answer industry knowledge questions.
• Accuracy metric for multiple-choice academic and security questions.
• Human baseline testing with ordinary individuals and domain experts.

Results & findings

• Claude 3.5-Sonnet achieved the highest weighted average score of 72.9 in zero-shot setting.
• GPT-4o scored 71.9, with particularly strong performance in financial security (81.8).
• Qwen2.5-72B-Instruct ranked third overall (69.4) but outperformed many closed-source models in security.
• General open-source models often outperformed financial-specific fine-tuned models.
• LLMs surpassed ordinary individuals (30.1) but lagged behind experts (85.9).
• Open-source models showed higher rates of logical reasoning errors compared to closed-source models.
• Closed-source models were more prone to ambiguity handling weaknesses and hallucinations.
• XuanYuan3-70B-Chat improved significantly with five-shot CoT, ranking first in that setting.

Data & experiments

• Dataset contains 8,351 questions: 4,661 academic, 1,434 industry, 1,640 security, 616 agent.
• Data sourced from mock exams, financial websites, SecEval, and expert-created agent tasks.
• Evaluated 19 models including Claude 3.5-Sonnet, GPT-4o, Qwen2.5, and XuanYuan3.
• Experimental settings: zero-shot, five-shot, zero-shot CoT, five-shot CoT.
• Metrics: Accuracy, Rouge-L, and GPT-4o judge for subjective/open-ended tasks.
• Human baseline comparison with 3 ordinary students and 3 expert respondents.

Limitations & caveats

• Evaluation limited to 20% of the dataset for final results due to resource constraints.
• Human baseline sample size is small (3 ordinary, 3 expert participants).
• Agent tasks evaluated only in zero-shot setting due to complexity.
• GPT-4o judge may introduce bias in subjective evaluations.
• Dataset focuses on Chinese language, limiting generalizability to other languages.
• Security knowledge adapted from SecEval, which may not cover all financial security nuances.

Curator notes

Contains Chinese finance questions across academic, industry, security, and agent domains.