The List / Trading, Investment, and Portfolio Management / paper
FinReasoning: A Hierarchical Benchmark for Reliable Financial Research Reporting
Abstract
Large language models (LLMs) are increasingly deployed in financial research workflows, where their role is evolving from single-model assistance for human analysts toward autonomous collaboration among multiple agents. Yet real-world deployments still expose factual errors, numerical inconsistencies, and shallow analysis, which can distort assessments of corporate fundamentals and trigger severe economic losses. While existing benchmarks have begun to evaluate such failures, they score all aspects of the generated analysis in one pass, failing to distinguish whether a model fails at foundational stages like auditing and correction, or underperforms at generating research-grade insights. Consequently, it obscures capability bottlenecks and the specialized strengths essential for multi-agent role assignment. To address these gaps, we introduce FinReasoning, a hierarchical benchmark that decomposes the core capabilities of financial research into semantic consistency, data alignment, and deep insight. We further propose a fine-grained evaluation framework that strengthens hallucination-correction assessment and incorporates a 12-indicator rubric for core analytical skills. FinReasoning reveals clear capability stratification across model types. Closed-source models (like Doubao-Seed-1.8) perform strongly overall and are better suited for core reasoning agents in multi-agent financial systems; open-source general models (like Qwen3-235B) show clear capability divergence and consistently underperform on Semantic Consistency, making them less suited for quality-sensitive generation tasks; financial-domain models (like Fin-R1) generate moderate insights but lack foundational auditing skills. Our work has already been deployed in pilot tests across several real-world scenarios. The resource is available at https://github.com/TongjiFinLab/FinReasoning.
In one sentence
Summary
The paper addresses the lack of granular evaluation for LLMs in financial research, where existing benchmarks collapse distinct reasoning stages into single scores, obscuring capability bottlenecks in multi-agent systems. FinReasoning decomposes financial reasoning into three tracks: Semantic Consistency (error detection/correction in long texts), Data Alignment (numerical verification and rule-based reasoning on structured data), and Deep Insight (research-grade analysis using a 12-indicator expert rubric). This hierarchical approach allows for precise diagnosis of model strengths and weaknesses across the analytical workflow, from proofreading to strategic synthesis. The benchmark utilizes 4,800 samples derived from real-world financial literature, news, and A-share market data (2023-2025), constructed via error injection and expert-validated QA generation. The authors evaluate 19 LLMs, including closed-source, open-source general, and financial-domain models, using a hybrid evaluation framework combining objective metrics with a fused LLM-as-a-Judge protocol. Experiments assess error localization, calculation accuracy, and causal depth, with robustness checks confirming high inter-expert agreement and judge stability. The setup isolates performance across difficulty levels, from simple verification to complex multi-step reasoning. Results show clear capability stratification: closed-source models like Doubao-Seed-1.8 and GPT-5 lead in overall performance and are best suited for core reasoning agents. Open-source general models (e.g., Qwen3-235B) underperform on Semantic Consistency, while financial-domain models (e.g., Fin-R1) lack foundational auditing skills despite moderate insight generation. A key finding is that reasoning-specialized models often fail on simple data retrieval tasks due to over-reasoning. The benchmark is deployed in pilot banking scenarios, highlighting its utility for system design, though limitations include potential judge bias and the specific focus on Chinese A-share data.
Research problem
Current LLM benchmarks for financial research fail to distinguish between foundational errors (like hallucination and numerical inconsistency) and higher-level analytical failures, obscuring the specific capabilities required for multi-agent financial systems. This lack of granularity prevents accurate diagnosis of model bottlenecks and hinders optimal role assignment in autonomous research workflows.
Core contributions
- Introduction of FinReasoning, a hierarchical benchmark decomposing financial research into semantic consistency, data alignment, and deep insight tracks.
- Development of a fine-grained evaluation framework with a 12-indicator rubric for deep insight and structured protocols for hallucination correction.
- Construction of a 4,800-sample dataset using real-world financial texts, A-share data, and controlled error injection mechanisms.
- Empirical evaluation of 19 LLMs revealing systematic capability stratification across model types and reasoning stages.
- Deployment of the benchmark in real-world banking pilot tests, demonstrating practical utility for multi-agent system design.
Method
- Semantic Consistency track: Error injection into financial texts followed by evaluation of error location (Sentence-F1) and correction quality (BERTScore, LLM-Judge).
- Data Alignment track: Rule-based reasoning on structured A-share data, evaluating success rate, answer accuracy, and retrieval F1.
- Deep Insight track: Open-ended analysis of financial events scored via LLM-Judge against a 4-dimension, 12-indicator expert rubric.
- Evaluation framework: Two-stage Z-Score and Min-Max normalization to enable cross-track comparison.
- Judge fusion: Combining outputs from DeepSeek-V3 and Qwen3-235B via grid search to maximize alignment with expert annotations.
- Error taxonomy: Derived from pilot studies of 200 financial texts, categorizing terminology, fact, and logic errors.
Results & findings
- Closed-source models (Doubao-Seed-1.8, GPT-5) achieve the highest normalized scores (96.8, 95.4) and are best for core reasoning agents.
- Open-source general models (Qwen3-235B) show significant divergence, underperforming on Semantic Consistency (57.5 normalized score).
- Financial-domain models (Fin-R1) rank lowest overall (46.9), lacking foundational auditing skills despite moderate insight generation.
- Reasoning-specialized models (DeepSeek-R1) fail on simple verification tasks (25.7 score) due to over-reasoning and format instability.
- Causal reasoning (Justification and Causal Depth) is the primary differentiator for top-tier models in the Deep Insight track.
- Model scale helps but is insufficient; Llama-3.1-70B underperforms significantly in Deep Insight compared to smaller closed-source models.
- Fact errors are easier to detect than Terminology or Logic errors, with a ~12-point performance gap in the Semantic Consistency track.
Data & experiments
- Dataset of 4,800 samples including 677 financial literature pieces, 10,000+ news reports, 901 research reports, and A-share market data (2023-2025).
- Evaluation of 19 LLMs: 16 general models (closed-source and open-source) and 3 financial-domain models.
- Three-track experimental design: Semantic Consistency (error detection/correction), Data Alignment (verification/calculation/reasoning), and Deep Insight (analytical quality).
- Hybrid evaluation using objective metrics (F1, BERTScore) and a fused LLM-as-a-Judge protocol validated by financial experts.
- Robustness checks including inter-expert agreement (κ=0.79), judge perturbation tests, and bootstrap confidence intervals.
Limitations & caveats
- The benchmark focuses primarily on Chinese A-share market data and Chinese-language financial texts, limiting generalizability to other markets.
- The LLM-as-a-Judge protocol, while validated, may still exhibit subtle biases or stringency differences compared to human experts.
- Financial-domain models' poor performance on foundational tasks suggests that current fine-tuning methods may not adequately preserve basic reliability.
- The study evaluates zero-shot performance, which may not reflect the capabilities of models with extensive in-context learning or RAG integration.
- The error injection mechanisms, while realistic, may not capture all forms of hallucination present in real-world model outputs.
Curator notes
Discovery status: report_analysis_focus_promoted. Source links: SEC-QA: A Systematic Benchmark for Evaluating Long-Context Question Answering on SEC Filings; SECQUE: A Benchmark for Evaluating Question-Answering on SEC Filings