The List / Foundation and Domain Language Models / paper
From Scores to Skills: A Cognitive Diagnosis Framework for Evaluating Financial Large Language Models
Abstract
Large Language Models (LLMs) have shown promise for financial applications, yet their suitability for this high-stakes domain remains largely unproven due to inadequacies in existing benchmarks. Existing benchmarks solely rely on score-level evaluation, summarizing performance with a single score that obscures the nuanced understanding of what models truly know and their precise limitations. They also rely on datasets that cover only a narrow subset of financial concepts, while overlooking other essentials for real-world applications. To address these gaps, we introduce FinCDM, the first cognitive diagnosis evaluation framework tailored for financial LLMs, enabling the evaluation of LLMs at the knowledge-skill level, identifying what financial skills and knowledge they have or lack based on their response patterns across skill-tagged tasks, rather than a single aggregated number. We construct CPA-KQA, the first cognitively informed financial evaluation dataset derived from the Certified Public Accountant (CPA) examination, with comprehensive coverage of real-world accounting and financial skills. It is rigorously annotated by domain experts, who author, validate, and annotate questions with high inter-annotator agreement and fine-grained knowledge labels. Our extensive experiments on 30 proprietary, open-source, and domain-specific LLMs show that FinCDM reveals hidden knowledge gaps, identifies under-tested areas such as tax and regulatory reasoning overlooked by traditional benchmarks, and uncovers behavioral clusters among models. FinCDM introduces a new paradigm for financial LLM evaluation by enabling interpretable, skill-aware diagnosis that supports more trustworthy and targeted model development, and all datasets and evaluation scripts will be publicly released to support further research.
In one sentence
Summary
Existing financial LLM benchmarks suffer from score flattening and narrow concept coverage, obscuring nuanced model capabilities. The authors propose FinCDM, a cognitive diagnosis framework inspired by educational assessment, which evaluates LLMs based on mastery of latent financial skills rather than single aggregate scores. This approach aims to provide interpretable, skill-aware diagnostics that identify precise knowledge gaps and strengths across diverse financial domains, addressing the inadequacy of current evaluation paradigms in high-stakes financial applications. The core contribution is CPA-KQA, a dataset derived from the Certified Public Accountant examination, covering 70 key financial concepts with rigorous expert annotation and high inter-annotator agreement. The method employs non-negative matrix co-factorization to model the relationship between questions, latent skills, and model proficiency. The authors evaluate 30 proprietary, open-source, and domain-specific LLMs on this dataset, comparing their performance against traditional benchmarks like FinEval-KQA to demonstrate the superior diagnostic power of their framework in capturing fine-grained knowledge states. Results reveal that models with similar aggregate scores exhibit divergent knowledge specializations; for instance, Gemini excels in general accounting while Doubao dominates financial cost management. The study highlights that prior benchmarks overlook critical areas like tax and regulatory reasoning, which FinCDM exposes as significant weaknesses. Additionally, the analysis uncovers behavioral clusters among models and emphasizes the importance of linguistic resources, showing that models lacking Chinese pretraining perform poorly. These findings support more targeted model development and deployment strategies in finance. Additional survey-useful detail: Proposes FinCDM, the first cognitive diagnosis framework for financial LLMs that moves beyond aggregate metrics to assess knowledge-skill mastery. CPA-KQA dataset: 210 questions covering 70 financial concepts from CPA exams, annotated by domain experts with high inter-annotator agreement. Dataset is based on Chinese CPA exams, potentially limiting generalizability to other jurisdictions or languages.
Research problem
Current financial LLM benchmarks rely on aggregate scores that flatten performance metrics, hiding specific knowledge gaps and strengths. They also suffer from narrow concept coverage, failing to assess the full range of real-world financial skills required for high-stakes applications. This paper addresses the need for a diagnostic evaluation framework that can identify what models truly know and where they fail at a granular, skill-level.
Core contributions
- Proposes FinCDM, the first cognitive diagnosis framework for financial LLMs that moves beyond aggregate metrics to assess knowledge-skill mastery.
- Constructs CPA-KQA, a high-quality dataset derived from CPA exams with 70 annotated financial concepts and rigorous expert validation.
- Develops a non-negative matrix co-factorization method to infer latent skill proficiency and question-skill associations from model responses.
- Evaluates 30 diverse LLMs, revealing hidden knowledge gaps and specialization patterns not captured by traditional benchmarks.
- Demonstrates the limitations of existing benchmarks like FinEval through re-annotation, showing their skewed concept coverage.
Method
- Non-negative matrix co-factorization to decompose response matrix X into question-skill (E), model-skill (U), and skill-concept (V) matrices.
- Generative probabilistic model assuming Bernoulli distribution for observed responses based on latent skill alignment.
- Expert-driven annotation process with two-phase validation ensuring high inter-annotator agreement (Krippendorff’s alpha 0.937).
- Construction of CPA-KQA dataset based on CPA exam content and skill specification outlines.
- Re-annotation of FinEval dataset using CPA-KQA taxonomy to assess coverage imbalance.
- Evaluation of 30 LLMs using consistent prompting and decoding configurations (temperature 1.2, max 64 tokens).
Results & findings
- FinCDM reveals that models with similar aggregate scores have divergent knowledge specializations (e.g., Gemini vs. Doubao).
- Prior benchmarks like FinEval show significant structural imbalance, focusing on a narrow subset of financial concepts.
- FinCDM identifies under-tested areas such as tax and regulatory reasoning, which are critical for real-world applications.
- Models with limited Chinese language capabilities (e.g., Falcon-7B) significantly underperform, highlighting linguistic resource importance.
- Behavioral clusters are identified, such as GPT-3.5 and DeepSeek-VL sharing strengths in financial reporting and valuation.
- GLM-4 achieves the highest concept mastery (40/70 concepts with >0.9 probability) on CPA-KQA.
Data & experiments
- CPA-KQA dataset: 210 questions covering 70 financial concepts from CPA exams, annotated by domain experts with high inter-annotator agreement.
- FinEval-KQA: Re-annotated subset of FinEval using CPA-KQA taxonomy to benchmark existing dataset coverage.
- Evaluation of 30 LLMs including proprietary (GPT-4o, Claude, Gemini), open-source (Qwen, Llama, DeepSeek), and financial-specific models.
- Experiments use non-negative matrix co-factorization to estimate model proficiency across latent skills.
- Metrics include concept mastery probability, aggregate accuracy, and identification of behavioral clusters among models.
Limitations & caveats
- Dataset is based on Chinese CPA exams, potentially limiting generalizability to other jurisdictions or languages.
- Evaluation focuses on multiple-choice questions, which may not capture complex reasoning or generation capabilities.
- The number of questions per concept is limited (3 per concept), which may affect the robustness of skill inference.
- Results are based on API access for many models, which may introduce variability in response generation.
- The framework assumes binary mastery of skills, which may oversimplify the continuous nature of knowledge.
Curator notes
Discovery status: round1_promoted_seed_for_round2. Source links: AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework; Benchmarking Large Language Models on CFLUE - A Chinese Financial Language Understanding Evaluation Dataset; BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs; ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering; DocFinQA: A Long-Context Financial Reasoning Dataset; FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought F