The List / Financial QA, Reasoning, and Table Understanding / paper
FinQA: A Dataset of Numerical Reasoning over Financial Data
Abstract
The sheer volume of financial statements makes it difficult for humans to access and analyze a business’s financials. Robust numerical reasoning likewise faces unique challenges in this domain. In this work, we focus on answering deep questions over financial data, aiming to automate the analysis of a large corpus of financial documents. In contrast to existing tasks on general domain, the finance domain includes complex numerical reasoning and understanding of heterogeneous representations. To facilitate analytical progress, we propose a new large-scale dataset, FinQA, with Question-Answering pairs over Financial reports, written by financial experts. We also annotate the gold reasoning programs to ensure full explainability. We further introduce baselines and conduct comprehensive experiments in our dataset. The results demonstrate that popular, large, pre-trained models fall far short of expert humans in acquiring finance knowledge and in complex multi-step numerical reasoning on that knowledge. Our dataset – the first of its kind – should therefore enable significant, new community research into complex application domains. The dataset and code are publicly available at https://github.com/czyssrs/FinQA.
In one sentence
Summary
Financial analysis requires automating the interpretation of heterogeneous data sources, yet existing NLP benchmarks lack the complexity of real-world financial reasoning. The authors address this gap by introducing FinQA, a dataset designed to test deep numerical reasoning over financial documents. Unlike general-domain QA tasks that involve simple calculations, FinQA questions require integrating information from both unstructured text and structured tables within S&P 500 earnings reports. The dataset is constructed by financial experts to ensure domain relevance, focusing on questions that demand multi-step arithmetic operations and table aggregations to derive answers, thereby establishing a rigorous benchmark for explainable financial AI. The dataset comprises 8,281 examples derived from 2,789 pages of earnings reports, each annotated with gold reasoning programs defined by a Domain Specific Language (DSL) containing mathematical and table operations. To evaluate performance, the authors propose FinQANet, a retriever-generator framework that first retrieves supporting facts using BERT and then generates executable programs using pre-trained models like RoBERTa and FinBERT. Experiments compare these baselines against human experts and non-expert crowd workers, measuring both execution accuracy and program accuracy to assess the correctness of the reasoning path, not just the final answer. Results show that the best model, FinQANet with RoBERTa-large, achieves 61.24% execution accuracy, significantly outperforming non-expert crowds (50.68%) but falling far short of human experts (91.16%). The model struggles most with questions requiring both text and table integration, multi-step reasoning, and unit conversions involving constants. Error analysis reveals that failures often stem from a lack of financial domain knowledge and difficulties in aligning numerical values across heterogeneous formats. These findings highlight the substantial gap between current LLM capabilities and the robust, explainable reasoning required for professional financial analysis.
Research problem
The paper addresses the challenge of automating complex numerical reasoning over financial documents, a task that requires integrating heterogeneous data sources (text and tables) and performing multi-step calculations. It highlights the insufficiency of existing general-domain QA benchmarks in capturing the depth of financial analysis.
Core contributions
- Introduction of FinQA, the first large-scale dataset for numerical reasoning over financial reports.
- Creation of 8,281 expert-annotated QA pairs with fully executable reasoning programs.
- Definition of a Domain Specific Language (DSL) for financial numerical reasoning.
- Proposal of FinQANet, a retriever-generator baseline framework for financial QA.
- Comprehensive evaluation showing significant performance gaps between LLMs and human experts.
Method
- Retriever: BERT-based classifier to retrieve top 3 supporting facts.
- Generator: LSTM decoder with attention over retrieved facts and question.
- DSL: 10 operations (add, subtract, multiply, divide, exp, greater, table-sum, table-avg, table-max, table-min).
- Step Memory: Tokens (#0, #1) to track intermediate calculation results.
- Evaluation: Execution accuracy and symbolic program equivalence.
- Baselines: TF-IDF, Seq2Seq, NeRd, Longformer.
Results & findings
- FinQA contains 8,281 examples with gold reasoning programs.
- Human experts achieve 91.16% execution accuracy.
- Best model (FinQANet-RoBERTa-large) achieves 61.24% execution accuracy.
- Non-expert crowd accuracy is 50.68%.
- Table-only questions are easier (67.38%) than text-only (54.86%) or combined (43.80%).
- Multi-step programs (>2 steps) drop accuracy to 22.78%.
- FinBert did not outperform BERT-large, suggesting limited pre-training corpus size.
- Retriever recall for top 3 facts is 89.66%.
Data & experiments
- Dataset: 8,281 QA pairs from S&P 500 earnings reports (1999-2019).
- Annotation: Performed by 11 financial experts (CPAs, MBAs) on Upwork.
- Benchmarks: FinQANet baselines using BERT, RoBERTa, and FinBERT.
- Metrics: Execution accuracy and symbolic program accuracy.
- Comparison: Human experts vs. non-expert MTurk workers vs. models.
Limitations & caveats
- Dataset limited to S&P 500 companies, potentially lacking diversity.
- Focuses on simple table layouts, excluding complex nested structures.
- Retriever may miss relevant facts, impacting generator performance.
- Models struggle with implicit financial knowledge and unit conversions.
- FinBert's limited pre-training corpus hindered its performance.
- Program accuracy may have false negatives due to multiple valid programs.
Curator notes
Includes expert-labeled question-answer pairs and gold reasoning programs.