FinanceReasoning: A Financial Benchmark for Large Reasoning Models

In one sentence

Summary

The paper addresses the lack of credible, comprehensive, and challenging benchmarks for evaluating the numerical reasoning capabilities of Large Reasoning Models (LRMs) in the financial domain. Existing datasets suffer from annotation errors, ambiguous questions, and insufficient coverage of complex financial concepts, leading to inflated performance metrics that do not reflect real-world reasoning abilities. The authors introduce FinanceReasoning to provide a reliable evaluation suite that captures the nuances of financial calculations, such as fee structures, valuation formulas, and statistical tests, ensuring that model performance is assessed with high precision and domain-specific rigor. The benchmark comprises 2,238 problems derived from re-annotating four public datasets and generating 908 new high-quality problems via Human-AI collaboration. A key contribution is the open-sourced financial function library containing 3,133 Python-formatted functions extracted from Investopedia, which serve as structured knowledge for retrieval-augmented generation. The authors evaluate six LRMs and seven standard LLMs using Chain-of-Thought (CoT) and Program-of-Thought (PoT) prompting. Experiments include knowledge augmentation via function retrieval and a collaborative setup where a reasoner model generates the logic and a programmer model executes the code. Results show that OpenAI o1 with PoT achieves the highest accuracy (89.1% on Hard problems), but LRMs still struggle with formula application and numerical precision. Knowledge augmentation using the function library improves GPT-4o's accuracy to 91.6%. The combination of DeepSeek-R1 as a reasoner and Claude 3.5 Sonnet as a programmer yields 87.8% accuracy, correcting most calculation errors. The study highlights that PoT is more token-efficient and accurate than CoT for complex tasks, and identifies formula misapplication and rounding errors as primary failure modes for current models.

Research problem

The paper identifies a critical gap in evaluating Large Reasoning Models (LRMs) for financial numerical reasoning. Existing benchmarks are criticized for poor annotation quality, ambiguous questions, and insufficient coverage of complex financial concepts, leading to unreliable performance assessments. The specific problem is the need for a credible, comprehensive, and challenging benchmark that rigorously tests models' ability to apply domain-specific formulas and perform precise multi-step calculations in high-stakes financial contexts.

Core contributions

• Introduction of FinanceReasoning, a benchmark with 2,238 problems featuring rigorous re-annotation of public datasets and expert-verified new problems.
• Construction and release of a comprehensive financial function library with 3,133 Python-formatted functions to support structured knowledge retrieval.
• Demonstration that combining Reasoner and Programmer models significantly enhances numerical precision and overall reasoning performance.
• Analysis of LRM limitations, identifying formula application errors and numerical calculation inaccuracies as primary bottlenecks in financial reasoning.
• Evidence that Program-of-Thought (PoT) prompting is more token-efficient and accurate than Chain-of-Thought (CoT) for complex financial tasks.

Method

• Re-annotation of existing datasets (CodeFinQA, CodeTAT-QA, FinCode, FinanceMath) to fix ambiguities and errors.
• LLM-assisted generation of new problems using seed functions, followed by expert verification.
• Difficulty grading based on operator count, parentheses, and code lines in Python solutions.
• Retrieval-augmented generation using Contriever for function and passage retrieval.
• Model collaboration: Reasoner (DeepSeek-R1) generates logic, Programmer (Claude 3.5 Sonnet) generates/executes code.
• Strict evaluation with 0.2% error margin and precise formatting requirements.

Results & findings

• OpenAI o1 with PoT achieves 89.1% accuracy on the Hard subset, outperforming other models.
• DeepSeek-R1 with PoT achieves 85.3% on Hard, but improves to 87.8% when combined with Claude 3.5 Sonnet as a programmer.
• GPT-4o with knowledge augmentation (LLM-instructed retrieval + function retrieval) reaches 91.6% accuracy on Hard problems.
• PoT reduces token consumption significantly compared to CoT (e.g., GPT-4o uses 54k vs 173k tokens for Hard subset).
• Formula application errors increase with difficulty, while numerical extraction errors remain low.
• Re-annotation corrected 15.6% of questions from existing datasets, revealing lower true performance of LRMs.
• DeepSeek-R1 excels in text-based reasoning (CoT), while OpenAI o1 excels in programming (PoT).

Data & experiments

• Dataset: 2,238 problems (1,000 Easy, 1,000 Medium, 238 Hard) with hybrid text/table contexts and Python solutions.
• Function Library: 3,133 Python functions extracted from Investopedia, covering 1,864 financial concepts.
• Models Evaluated: 6 LRMs (e.g., OpenAI o1, DeepSeek-R1) and 7 LLMs (e.g., GPT-4o, Claude 3.5 Sonnet).
• Methods: Chain-of-Thought (CoT) and Program-of-Thought (PoT) prompting strategies.
• Augmentation: Retrieval of financial functions and passages to enhance model knowledge.
• Metrics: Strict accuracy with a 0.2% error margin, token consumption, and error type analysis.

Limitations & caveats

• The benchmark focuses on numerical reasoning and may not capture broader financial understanding or qualitative analysis.
• Knowledge augmentation relies on a fixed function library, which may not cover all niche financial scenarios.
• Model collaboration requires multiple API calls, increasing latency and cost compared to single-model inference.
• The difficulty grading heuristic is based on code complexity, which may not perfectly align with human-perceived difficulty.
• Evaluation is limited to proprietary and open-source models available at the time of writing.

Curator notes

Includes updated public tasks and new questions with Python solutions.