Document-Level Numerical Reasoning across Single and Multiple Tables in Financial Reports

In one sentence

Summary

The paper addresses the critical gap in evaluating Large Language Models (LLMs) on document-level numerical reasoning within long, structured financial documents. While existing benchmarks focus on single-table or short-context settings, financial analysis often requires integrating scattered evidence across multiple tables and narrative text in annual reports exceeding 100k tokens. The authors identify two primary bottlenecks: context rot, where models fail to locate relevant tables in long inputs, and errors in multi-step numerical reasoning even when evidence is retrieved. This work positions itself as a necessary step toward reliable automated financial statement analysis by providing a rigorous benchmark and a method to mitigate these specific failure modes. To address these challenges, the authors construct FinLongDocQA, a dataset comprising 7,527 question-answer pairs derived from 1,456 S&P 500 annual reports (2022-2024). The dataset is designed to require cross-table and cross-page reasoning, with evidence often spanning dozens of pages. The construction pipeline involves LLM-generated QA candidates, rule-based filtering to ensure multi-hop requirements and calculation correctness, and manual review for financial relevance. The authors also propose FinLongDocAgent, a multi-agent system built on AutoGen. It employs an Expansion Agent to formulate metric-aware queries, a Solving Agent to perform calculations with page-grounded citations, and an Evaluation Agent to verify completeness and trigger iterative retrieval rounds if operands are missing. Experiments evaluate closed-source and open-source LLMs against baselines like single-round RAG, GraphRAG, and agentic web search. Experimental results demonstrate that standard long-context prompting and single-round RAG are insufficient for this task, with top models achieving only ~33% Exact Match. FinLongDocAgent significantly outperforms these baselines, raising Exact Match to 41.34% with Gemini-3-Flash. The ablation study highlights the critical role of the Evaluation Agent in reducing missing operand errors. Error analysis reveals that retrieval failures account for 62% of errors, followed by evidence utilization (19%) and calculation errors (8%). The paper concludes that iterative retrieval and verification are essential for reliable financial numerical QA. Limitations include the dataset's focus on S&P 500 firms and the computational overhead of multi-agent interactions. The work is highly relevant for surveys on financial document understanding, RAG architectures, and LLM reasoning capabilities in structured domains.

Research problem

The paper addresses the inability of current LLM benchmarks to evaluate document-level numerical reasoning in long financial reports. Existing datasets largely focus on single-table or short-context settings, failing to capture the complexity of integrating evidence scattered across multiple tables and text spans in annual reports. The core problem is twofold: locating relevant evidence in long documents (context rot) and performing accurate multi-step arithmetic on retrieved data.

Core contributions

• Introduction of FinLongDocQA, the first benchmark targeting document-level financial numerical QA requiring reasoning across multiple tables in long-form annual reports.
• Creation of a high-quality dataset with 7,527 QA pairs from 1,456 S&P 500 filings, featuring page-grounded evidence annotations and executable Python code for verification.
• Proposal of FinLongDocAgent, a multi-agent, multi-round RAG method that iteratively retrieves evidence, performs intermediate calculations, and verifies results.
• Comprehensive evaluation of closed-source and open-source LLMs, revealing significant performance gaps in cross-table reasoning compared to single-table benchmarks.
• Detailed error analysis identifying retrieval failure (62%) and evidence utilization errors (19%) as primary bottlenecks in financial numerical QA.

Method

• Dataset Construction: LLM-generated QA candidates filtered by rule-based checks (multi-page, executable code) and manual review.
• Document Preprocessing: PDFs converted to Markdown using MinerU, segmented into page-level chunks for retrieval.
• FinLongDocAgent Architecture: Three agents (Expansion, Solving, Evaluation) operating in a multi-round loop.
• Expansion Agent: Converts questions into metric-aware queries, adding synonyms and refining based on missing operands.
• Solving Agent: Retrieves top-k pages, performs calculations, and cites page numbers for each step.
• Evaluation Agent: Verifies answer completeness and faithfulness, returning missing components to trigger re-retrieval.
• Retrieval: Uses BGE-M3 dense retriever with page-level indexing, retrieving top-15 chunks per round.
• Verification: Iterative process continues until the Evaluation Agent signals 'NONE' or max rounds (5) are reached.

Results & findings

• FinLongDocQA reveals that LLMs struggle with context rot in reports exceeding 129k tokens, often failing to locate relevant tables.
• Even with retrieved evidence, LLMs make errors in multi-step numerical reasoning, including extracting wrong values and misapplying formulas.
• Single-round RAG improves performance over long-context prompting but remains limited by retrieval recall and lack of verification.
• FinLongDocAgent achieves the best performance, with Gemini-3-Flash reaching 41.34% Exact Match, a significant improvement over the 33.03% baseline.
• The Evaluation Agent is the most critical component, reducing failures due to missing operands and unit mismatches by triggering additional retrieval rounds.
• Retrieval errors account for 62% of total failures, indicating that evidence localization is the primary bottleneck.
• Graph-based RAG methods underperform dense RAG, likely because financial table relations are implicit and not well-captured by entity graphs.
• Performance drops sharply as the number of required evidence tables increases, highlighting the difficulty of cross-table reasoning.

Data & experiments

• Dataset: FinLongDocQA, containing 7,527 QA pairs from 1,456 S&P 500 annual reports (2022-2024), with an average input length of ~129k tokens.
• Evaluation: Tested on 4 LLMs (GPT-4o-mini, Gemini-3-Flash, Qwen3-Thinking 30B, DeepSeek-v3.2) using Exact Match, Tolerance Accuracy, and F1.
• Baselines: Compared against LLM-only, long-context prompting, single-round dense RAG (BGE), GraphRAG, HippoRAG, and WebDancer.
• Metrics: Exact Match (EM), Tolerance Accuracy (Tol. Acc), and token-level F1, with normalization for currency and formatting.
• Ablation: Analyzed the impact of individual agents (Expansion, Solving, Evaluation) in FinLongDocAgent.
• Difficulty Analysis: Categorized questions by the number of evidence tables (Easy: ≤1, Medium: 2, Hard: ≥3) to assess cross-table reasoning capability.

Limitations & caveats

• Dataset is limited to S&P 500 companies, potentially lacking diversity in industry-specific reporting styles.
• Focus on annual reports may not generalize to quarterly filings or other financial documents with different structures.
• Multi-agent approach incurs higher computational cost and latency compared to single-round RAG.
• Reliance on LLM-generated QA pairs, even with filtering, may introduce subtle biases or errors.
• Evaluation focuses on numerical accuracy, not on the quality of natural language explanations or insights.

Curator notes

Discovery status: report_analysis_focus_promoted. Source links: DocFinQA: A Long-Context Financial Reasoning Dataset