FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning

In one sentence

Summary

The paper addresses the critical gap in financial NLP benchmarks, which typically prioritize final numerical answers over transparent, verifiable intermediate reasoning. Existing datasets like FinQA lack systematic step-level supervision, making it difficult to distinguish genuine multi-step inference from pattern matching. The authors introduce FinChain, the first benchmark designed specifically for verifiable Chain-of-Thought evaluation in finance, aiming to expose weaknesses in symbolic financial reasoning that previous metrics overlooked. This positioning highlights the need for benchmarks that enforce transparency and auditability in financial AI systems. FinChain is constructed using parameterized symbolic templates across 58 topics in 12 financial domains, generating contamination-free data with executable Python code for automatic verification. The authors propose CHAINEVAL, a dynamic alignment metric using Dynamic Time Warping to jointly assess step-level semantic and numerical consistency alongside final-answer correctness. They evaluate 26 leading LLMs, including frontier proprietary models and fine-tuned open-weight systems, under zero-shot conditions. The experimental design isolates reasoning ability from document parsing, focusing on structured symbolic operations ranging from basic compound interest to advanced investment analysis. Results show that while frontier models like GPT-5 lead in overall performance, they still exhibit significant limitations in advanced multi-step symbolic reasoning, with performance degrading sharply on complex tasks. Domain-adapted and math-enhanced models narrow the gap but do not close it, suggesting that scale alone is insufficient without structured supervision. Error analysis reveals heterogeneous failure modes, including computational, conceptual, and hallucination errors. A key limitation is the synthetic nature of the data, which lacks the linguistic diversity of real-world financial texts, though it provides a controlled testbed for developing trustworthy, interpretable financial AI.

Research problem

Current financial NLP benchmarks emphasize final numerical answers and neglect the intermediate reasoning steps required for transparency and verification. This makes it difficult to diagnose where reasoning fails or to distinguish genuine multi-step inference from surface-level pattern matching. The paper addresses the need for a benchmark that enforces verifiable, step-by-step symbolic reasoning in finance.

Core contributions

• Introduction of FinChain, the first from-scratch symbolic benchmark for financial reasoning spanning 12 domains and 58 topics.
• Proposal of CHAINEVAL, a verifiable reasoning measure that evaluates step-level consistency and final-answer correctness.
• Demonstration that frontier LLMs struggle with verifiable multi-step financial reasoning, particularly on advanced templates.
• Provision of a contamination-free, executable dataset enabling fully machine-verifiable evaluation of reasoning chains.

Method

• Symbolic template generation with parameterized variables and executable Python code.
• Dynamic Time Warping (DTW) for aligning predicted and gold reasoning step sequences.
• Gated step-level similarity combining semantic and numerical consistency.
• Final Answer Correctness (FAC) metric with 5% tolerance for numerical deviations.
• Composite CHAINEVAL score weighting step alignment and final answer correctness.
• Expert review and calibration for data validation and metric correlation.

Results & findings

• Frontier models like GPT-5 achieve the highest CHAINEVAL scores but still show degradation on advanced tasks.
• Fine-tuned models (e.g., Fin-R1, Mathstral) show limited gains compared to frontier systems.
• CHAINEVAL correlates strongly with human expert judgments, outperforming ROUGE and BERTScore.
• Open-weight models exhibit heterogeneous performance across different financial domains.
• Error analysis shows weaker models suffer from diverse failure modes including hallucinations.

Data & experiments

• FinChain dataset with 2,900 test cases generated from symbolic templates across 12 financial domains.
• Evaluation of 26 proprietary and open-weight LLMs under zero-shot settings.
• Use of CHAINEVAL metric validated against human expert judgments with high correlation.
• Error analysis categorizing failures into computational, conceptual, hallucination, and parsing errors.

Limitations & caveats

• Dataset is entirely synthetic, lacking linguistic diversity of real-world financial texts.
• Focuses on symbolic numerical reasoning, excluding qualitative or strategic decision-making.
• Does not cover real-world document parsing or long-context understanding challenges.
• Evaluation is zero-shot, potentially underestimating models with few-shot capabilities.
• Parsing of reasoning steps relies on regex and LLM-based extraction, which may introduce noise.

Curator notes

Discovery status: round1_promoted_seed_for_round2. Source links: A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges; A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law; BloombergGPT: A Large Language Model for Finance; ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering; Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learning; FinBERT: Financial Sentiment Analysis with Pre-trained Language Models