BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs

In one sentence

Summary

The paper addresses the critical gap in evaluating Large Language Models (LLMs) for real-world financial applications, where existing benchmarks often lack the contextual complexity, adversarial noise, and business-grounded reasoning required in practice. BizFinBench is introduced as the first benchmark specifically designed to assess LLMs across five dimensions: numerical calculation, reasoning, information extraction, prediction recognition, and knowledge-based question answering. It comprises 6,781 well-annotated queries in Chinese, derived from real user interactions on the iwencai APP, covering nine fine-grained categories such as anomalous event attribution and financial tool usage. The dataset emphasizes business practicality by including noisy, time-sensitive, and adversarial contexts to test model robustness beyond simple fact retrieval. To ensure rigorous evaluation, the authors propose IteraJudge, an iterative calibration-based framework that reduces bias when LLMs serve as evaluators. This method employs dimension-decoupled assessment, sequential correction generation, and reference-aligned scoring to provide more reliable metrics than standard LLM-as-a-Judge approaches. The benchmark was used to evaluate 25 state-of-the-art models, including proprietary systems like GPT-4o, Claude-3.5-Sonnet, and Gemini-2.0-Flash, as well as open-source models like DeepSeek-R1 and Qwen series. Experiments were conducted with strict JSON output constraints and chain-of-thought reasoning, measuring performance across objective and subjective metrics to capture both accuracy and nuanced financial understanding. Results indicate distinct capability patterns: proprietary models dominate in reasoning and information extraction, with ChatGPT-o3 and Gemini-2.0-Flash leading in complex tasks, while open-source models like DeepSeek-R1 show strong competitiveness in numerical calculation and entity recognition. However, no model dominates across all tasks, and performance varies significantly by category; for instance, prediction recognition shows minimal variance among top models, whereas information extraction has the largest spread. The study reveals that current LLMs handle routine queries competently but struggle with complex scenarios requiring cross-concept reasoning and temporal analysis. Limitations include the focus on Chinese-language data, which may limit generalizability to other markets, and the potential for evaluation bias despite IteraJudge's improvements, suggesting a need for further human-in-the-loop validation in high-stakes financial deployments.

Research problem

The paper addresses the challenge of evaluating LLM reliability in logic-heavy, precision-critical financial domains where existing benchmarks fail to capture real-world complexity, business context, and adversarial robustness. It specifically targets the gap between academic benchmark performance and practical applicability in financial services, such as automated analysis, risk assessment, and investment strategy formulation. The problem is compounded by the lack of standardized, business-aligned evaluation metrics that can handle noisy, time-sensitive, and multi-step reasoning tasks inherent in financial decision-making.

Core contributions

• Introduction of BizFinBench, the first business-driven financial benchmark with 6,781 real-world Chinese queries spanning five dimensions and nine categories.
• Development of IteraJudge, a novel iterative calibration-based evaluation framework that reduces bias in LLM-as-a-Judge assessments through dimension-decoupled scoring.
• Comprehensive evaluation of 25 SOTA LLMs (proprietary and open-source), revealing distinct capability patterns and performance gaps across financial tasks.
• Creation of a high-quality dataset sourced from real user queries on the iwencai APP, ensuring business relevance and contextual complexity.
• Demonstration that no single model dominates all financial tasks, highlighting specific strengths and weaknesses in areas like numerical calculation, reasoning, and extraction.

Method

• Data Collection: Aggregated real user queries from iwencai APP, cleaned and classified by financial experts into 9 categories.
• Context Construction: Retrieved relevant financial data (prices, news, disclosures) and introduced adversarial distractors to test robustness.
• Annotation: Multi-layered human annotation by three senior financial experts, requiring unanimous consensus for data inclusion.
• IteraJudge Pipeline: Dimension-decoupled assessment, sequential correction generation, and reference-aligned scoring to refine LLM evaluations.
• Evaluation Metrics: Combined objective (accuracy) and subjective (judge scores) metrics, with Spearman correlation used to measure evaluator alignment.
• Model Deployment: Open-source models on 8x H100 cluster; proprietary models via API; strict JSON output with chain-of-thought reasoning.

Results & findings

• Proprietary models dominate reasoning tasks: ChatGPT-o3 (83.58) and Gemini-2.0-Flash (81.15) lead, while open-source models trail by up to 19.49 points.
• DeepSeek-R1 leads in numerical calculation (64.04) and information extraction (71.46), outperforming many proprietary models in specific subtasks.
• Performance spread is largest in information extraction, with DeepSeek-R1 at 71.46 vs. Qwen3-1.7B at 11.23, indicating significant capability gaps.
• Prediction recognition shows minimal variance, with top models scoring between 39.16 and 50.00, suggesting a ceiling in current predictive capabilities.
• IteraJudge improves evaluation reliability, with Spearman correlation increases of up to 17.24% over vanilla LLM-as-a-Judge on financial data description tasks.
• Model scale correlates with performance in numerical reasoning, with larger Qwen3 models (32B) outperforming smaller ones (1.7B) across nearly all tasks.
• Financial time reasoning (FTR) is particularly challenging, with a 32.19-point gap between GPT-4o (76.20) and Llama-3.1-8B (2.91).

Data & experiments

• Dataset: 6,781 annotated Chinese queries from iwencai APP, covering 9 categories including anomalous event attribution, financial numerical computation, and emotion recognition.
• Models Evaluated: 25 LLMs, including GPT-4o, o3, o4-mini, Gemini-2.0-Flash, Claude-3.5-Sonnet, DeepSeek-R1, Qwen series, and Xuanyuan3-70B.
• Metrics: Objective metrics (accuracy, Spearman correlation) and subjective metrics (judge scores) using IteraJudge for nuanced evaluation.
• Experimental Setup: Strict JSON output format with chain-of-thought reasoning; evaluation on 8x NVIDIA H100 cluster for open-source models; API access for proprietary models.
• Ablation Studies: Comparison of IteraJudge against vanilla LLM-as-a-Judge, showing significant improvements in Spearman correlation for evaluation reliability.

Limitations & caveats

• Dataset is exclusively in Chinese, limiting generalizability to non-Chinese financial markets and languages.
• Evaluation relies on LLM-as-a-Judge, which may still introduce biases despite IteraJudge's improvements.
• Focus on real-world queries may not cover all niche or highly specialized financial tasks.
• Adversarial distractors are designed by experts, potentially missing other forms of real-world noise or complexity.
• Performance metrics may not fully capture the practical utility of LLMs in high-stakes financial decision-making.
• The study does not address long-term deployment challenges, such as model drift or regulatory compliance.

Curator notes

Discovery status: round2_promoted. Source links: DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning; FinBERT: Financial Sentiment Analysis with Pre-trained Language Models; FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning; From Scores to Skills: A Cognitive Diagnosis Framework for Evaluating Financial Large Language Models; InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning; Revolutionizing Finance with LLMs: An Overview