DocFinQA: A Long-Context Financial Reasoning Dataset

In one sentence

Summary

The paper addresses the critical gap in financial LLM evaluation where existing datasets rely on short, pre-selected document excerpts, failing to reflect the reality of analysts sifting through hundreds of pages of SEC filings. The authors introduce DocFinQA, a long-document financial question-answering dataset that augments the original FinQA questions with full 10-K filing contexts, increasing average context length from under 700 words to 123k words. This setup tests the model's ability to perform quantitative reasoning and locate specific numerical data within massive, noisy documents, a task essential for realistic financial workflows. The dataset includes 801 unique SEC filings, with questions annotated with Python programs for interpretable, executable answers, ensuring that models must ground their reasoning in the provided text rather than relying on parametric knowledge alone. The experimental evaluation compares retrieval-based QA pipelines against retrieval-free long-context LLMs. For retrieval, the study employs dense embedding models like ColBERT (including a finetuned variant), Sentence-BERT, and OpenAI Ada, alongside sparse BM25, to retrieve top-k chunks from chunked documents. These chunks are then fed into various LLMs, including Falcon, MPT, Llama 2, CodeLlama, Mistral, and GPT-3.5, using few-shot in-context learning. The results demonstrate that retrieval-based approaches generally outperform retrieval-free methods, with finetuned ColBERT showing the highest hit rates. However, even the best configurations yield low accuracy; GPT-3.5 achieved only 42.6% accuracy with retrieval, while non-expert humans scored 41%. The study also highlights that code-finetuned models and those with supervised fine-tuning perform better, and that larger context windows do not necessarily improve performance if the relevant information is lost in the middle of the document. A case study on documents exceeding 100k tokens reveals that over 40% of questions remain unanswerable even by models with 128k context windows, indicating severe limitations in long-range dependency handling. The paper notes that non-expert human performance is significantly lower than expert performance in short-context settings, underscoring the difficulty of the task. Limitations include the lack of full validation for training and development sets and potential biases in the automatically generated Python code. The dataset is publicly available and serves as a benchmark for improving long-context financial reasoning, with implications for applications requiring specificity and long-range context understanding in legal and financial domains.

Research problem

Existing financial QA datasets use short document excerpts, failing to simulate the realistic scenario where analysts must reason over hundreds of pages of SEC filings. The paper addresses the need for a benchmark that tests long-context numerical reasoning and information retrieval in financial documents, where the signal is buried in noise and context length is a primary constraint.

Core contributions

• Introduction of DocFinQA, a dataset extending FinQA with full SEC filing contexts (avg 123k words) for 7,437 questions.
• Creation of a code-enhanced version of FinQA with readable Python programs for interpretable numerical reasoning.
• Comprehensive evaluation of retrieval-based and retrieval-free LLMs on long-document financial QA.
• Analysis of chunking strategies, retrieval models (ColBERT, Sentence-BERT, Ada), and LLM performance across varying context lengths.
• Case study on 100k+ token documents revealing significant performance drops and limitations in current long-context models.

Method

• Data Collection: Mapped FinQA questions to SEC EDGAR filings, downloaded HTML/XML, and parsed to markdown.
• Chunking: Split documents into 2,750-character chunks with 20% overlap for retrieval-based evaluation.
• Retrieval: Encoded questions and chunks using dense models (ColBERT, SBERT, Ada) to compute cosine similarity.
• Alignment: Used four-gram similarity to identify the golden chunk for retrieval performance measurement.
• QA Evaluation: Few-shot in-context learning with top-k retrieved chunks fed into LLMs for answer generation.
• Code Generation: Converted FinQA derivation syntax to readable Python code using WizardCoder for interpretability.
• Human Evaluation: Recruited data professionals to answer a subsample of 200 long-document questions.

Results & findings

• DocFinQA increases context length by 175x compared to FinQA, averaging 123k words per document.
• Finetuned ColBERT achieved the highest retrieval hit rate (0.35 HR@1, 0.55 HR@3), outperforming Sentence-BERT and Ada.
• GPT-3.5 achieved 42.6% QA accuracy with retrieval, while non-expert humans scored 41% on long documents.
• Retrieval-free methods (System 2 Attention, Iterative) performed poorly, with GPT-4 scoring only 23% with S2A.
• Code-finetuned models (CodeLlama) and supervised fine-tuned models (Llama 2+SFT) outperformed base models.
• Over 40% of questions in 100k+ token documents were unanswerable even by models with 128k context windows.
• Non-expert human performance (41%) was much lower than expert performance in short-context FinQA (91.2%).

Data & experiments

• Dataset: 7,437 questions from 801 unique SEC 10-K filings, augmented with full document text and Python code.
• Retrieval Models: ColBERT (pretrained and finetuned), Sentence-BERT, OpenAI Ada, and BM25 for chunk retrieval.
• LLMs Evaluated: Falcon 7B, MPT 7B/30B, Llama 2 7B/13B, CodeLlama 7B/13B, Mistral 7B, GPT-3.5, and GPT-4 (case study).
• Metrics: Hit Rate (HR@k) for retrieval, accuracy for QA, and human evaluation on a subsample of long documents.
• Experimental Design: Few-shot in-context learning with top-k retrieved chunks; comparison of retrieval vs. retrieval-free methods.

Limitations & caveats

• Training and development sets were not fully validated by humans, only the test set.
• Python code annotations were generated by WizardCoder, potentially containing errors or false positives/negatives.
• Approximately 6.5% of FinQA questions were dropped due to missing SEC filings or parsing issues.
• Human evaluation was performed by non-experts, limiting the comparison to expert-level financial analysis.
• The dataset focuses on 10-K filings, which may not generalize to other financial document types like earnings call transcripts.

Curator notes

Expands local financial question answering to complete report context.