The List / Multimodal and Multilingual Finance / paper
Benchmarking Large Language Models on CFLUE - A Chinese Financial Language Understanding Evaluation Dataset
Abstract
In light of recent breakthroughs in large language models (LLMs) that have revolutionized natural language processing (NLP), there is an urgent need for new benchmarks to keep pace with the fast development of LLMs. In this paper, we propose CFLUE, the Chinese Financial Language Understanding Evaluation benchmark, designed to assess the capability of LLMs across various dimensions. Specifically, CFLUE provides datasets tailored for both knowledge assessment and application assessment. In knowledge assessment, it consists of 38K+ multiple-choice questions with associated solution explanations. These questions serve dual purposes: answer prediction and question reasoning. In application assessment, CFLUE features 16K+ test instances across distinct groups of NLP tasks such as text classification, machine translation, relation extraction, reading comprehension, and text generation. Upon CFLUE, we conduct a thorough evaluation of representative LLMs. The results reveal that only GPT-4 and GPT-4-turbo achieve an accuracy exceeding 60\% in answer prediction for knowledge assessment, suggesting that there is still substantial room for improvement in current LLMs. In application assessment, although GPT-4 and GPT-4-turbo are the top two performers, their considerable advantage over lightweight LLMs is noticeably diminished. The datasets and scripts associated with CFLUE are openly accessible at https://github.com/aliyun/cflue.
In one sentence
Summary
The paper addresses the lack of comprehensive evaluation benchmarks for Large Language Models (LLMs) in the Chinese financial domain. Existing datasets are limited in size, diversity, or task scope, focusing primarily on multiple-choice questions or event extraction. The authors propose CFLUE (Chinese Financial Language Understanding Evaluation) to provide a robust, multi-dimensional assessment tool that covers both financial knowledge retention and practical NLP application capabilities, filling a critical gap for evaluating Chinese-language financial AI systems. CFLUE comprises two main components: a knowledge assessment with 38,636 multiple-choice questions from 15 types of professional qualification exams, including answer prediction and reasoning explanations; and an application assessment with 16,522 instances across five NLP tasks: text classification, machine translation, relation extraction, reading comprehension, and text generation. The dataset was carefully curated to mitigate contamination by using recent data and rephrasing questions. The authors evaluated 15 LLMs, including OpenAI models, lightweight general-domain models (e.g., Qwen, LLaMA), and specialized financial LLMs (e.g., FinGPT, Tongyi-Finance), using metrics like accuracy, F1, BLEU, and ROUGE. Results show that GPT-4 and GPT-4-turbo are the top performers in knowledge assessment, achieving over 60% accuracy, indicating significant room for improvement in current LLMs. However, in application assessment, the advantage of GPT-4 over lightweight models diminishes, with models like Qwen-72B and ChatGLM3-6B performing competitively. Notably, specialized financial LLMs underperformed general-domain models, suggesting that domain-specific fine-tuning on limited corpora may not capture broad financial knowledge effectively. Fine-tuning general models on CFLUE data significantly boosted their performance, allowing smaller models to surpass larger proprietary models in certain tasks.
Research problem
The paper identifies a critical gap in evaluating Large Language Models (LLMs) within the Chinese financial context. Existing benchmarks are either too small, lack task diversity, or focus narrowly on specific sub-tasks like event extraction or multiple-choice QA. There is no comprehensive benchmark that assesses both deep financial knowledge retention and practical NLP application skills in Chinese, hindering the development and evaluation of financial AI systems.
Core contributions
- Proposes CFLUE, the first comprehensive Chinese financial LLM benchmark covering both knowledge and application assessments.
- Constructs a knowledge dataset of 38K+ multiple-choice questions from 15 types of professional qualification exams with solution explanations.
- Develops an application dataset of 16K+ instances across five NLP tasks: text classification, machine translation, relation extraction, reading comprehension, and text generation.
- Conducts a thorough evaluation of 15 representative LLMs, including general, lightweight, and specialized financial models.
- Provides detailed analysis of model performance across different tasks, revealing discrepancies between knowledge and application capabilities.
- Releases the dataset and evaluation scripts to the public to facilitate future research in Chinese financial NLP.
Method
- Data Collection: Knowledge questions sourced from online mock exams; application data from shared tasks, financial firm data, and public websites.
- Contamination Mitigation: Used PDF/Word extraction, GPT-4 rephrasing, and strict date filtering (2023) for web data.
- Model Evaluation: Zero-shot prompting for all models; specific prompts for each subtask in application assessment.
- Fine-tuning: LoRA fine-tuning on CFLUE training set for open-source models (rank 8, alpha 32, dropout 0.1).
- Metrics: Accuracy/F1 for classification/QA; BLEU/ROUGE for generation; COMET for translation.
- Analysis: Breakdown of performance by subject, question type, and input length.
Results & findings
- GPT-4 and GPT-4-turbo achieve >60% accuracy in knowledge assessment, the highest among all models, but still leave substantial room for improvement.
- Lightweight models like Qwen-72B and Qwen-14B outperform ChatGPT in knowledge assessment, showing that size and training data quality matter significantly.
- Specialized financial LLMs (FinGPT, DISC-FinLLM, Tongyi-Finance) underperform general-domain models of similar size, suggesting limited corpus diversity or overfitting.
- In application assessment, the performance gap between GPT-4 and lightweight models is much smaller than in knowledge assessment.
- Qwen-72B is the top-performing open-source model in application assessment, often matching or exceeding GPT-4-turbo in specific tasks like text generation.
- Fine-tuning general models on CFLUE data significantly boosts performance; e.g., Qwen-7B-CFLUE surpasses ChatGPT in answer prediction.
- Performance on quantitative reasoning questions is significantly lower than the overall average, highlighting a specific weakness in LLMs.
- ChatGPT sometimes outperforms GPT-4 in reasoning and reading comprehension tasks, indicating inconsistent performance across different LLM generations.
Data & experiments
- Knowledge Assessment: 38,636 multiple-choice questions from 15 types of Chinese financial qualification exams (e.g., CPA, banking, securities).
- Application Assessment: 16,522 instances across 5 tasks: text classification (6 subtasks), machine translation (2 directions), relation extraction (4 subtasks), reading comprehension, and text generation (5 subtasks).
- Models Evaluated: 3 OpenAI models (GPT-4, GPT-4-turbo, ChatGPT), 9 lightweight general-domain models (LLaMA2, Vicuna, ChatGLM3, Qwen, Baichuan2), and 3 specialized financial LLMs (FinGPT, DISC-FinLLM, Tongyi-Finance).
- Metrics: Accuracy and F1 for prediction; BLEU, ROUGE, COMET, and BERTScore for generation and translation tasks.
- Experimental Design: Zero-shot evaluation for all models; supervised fine-tuning (LoRA) conducted for open-source models on the training split of CFLUE.
Limitations & caveats
- The knowledge assessment relies on multiple-choice questions, which may not fully capture complex reasoning or open-ended financial analysis.
- The application assessment tasks are diverse but may not cover all real-world financial NLP scenarios.
- The study focuses on zero-shot evaluation, which may not reflect the performance of models fine-tuned for specific tasks.
- The financial LLMs evaluated may have limited training data, affecting the generalizability of the findings.
- The dataset is in Chinese, limiting its direct applicability to English-language financial contexts.
- The evaluation metrics (BLEU, ROUGE) may not adequately capture the quality of financial text generation.
Curator notes
Discovery status: round2_promoted. Source links: Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges; DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning; From Scores to Skills: A Cognitive Diagnosis Framework for Evaluating Financial Large Language Models