The List / Reports, Filings, Accounting, and Risk / paper
SEC-QA: A Systematic Benchmark for Evaluating Long-Context Question Answering on SEC Filings
Abstract
The financial domain frequently deals with large numbers of long documents that are essential for daily operations. Significant effort is put towards automating financial data analysis. However, a persistent challenge, not limited to the finance domain, is the scarcity of datasets that accurately reflect real-world tasks for model evaluation. Existing datasets are often constrained by size, context, or relevance to practical applications. Moreover, LLMs are currently trained on trillions of tokens of text, limiting access to novel data or documents that models have not encountered during training for unbiased evaluation. We propose SEC-QA, a continuous dataset generation framework with two key features: 1) the semi-automatic generation of Question-Answer (QA) pairs spanning multiple long context financial documents, which better represent real-world financial scenarios; 2) the ability to continually refresh the dataset using the most recent public document collections, not yet ingested by LLMs. Our experiments show that current retrieval augmented generation methods systematically fail to answer these challenging multi-document questions. In response, we introduce a QA system based on program-of-thought that improves the ability to perform complex information retrieval and quantitative reasoning pipelines, thereby increasing QA accuracy.
In one sentence
Summary
The paper addresses the critical gap in evaluating LLMs on complex, real-world financial tasks involving long-context, multi-document reasoning. Existing benchmarks suffer from data leakage and lack the structural complexity of professional financial analysis, such as navigating 10-Ks and 10-Qs. The authors propose SEC-QA, a framework that semi-automatically generates quantitative question-answer pairs from a database of S&P 500 metrics linked to their source SEC filings. This design ensures the benchmark is refreshable with new data, preventing training contamination, and covers parallel, multi-hop, and structural reference questions that mimic actual analyst workflows. Experiments evaluate four systems: Vanilla RAG, Multi-Query RAG, CodeGen+PageR, and CodeGen+DocS+PageR, using GPT-4 and Ada embeddings. The CodeGen systems utilize program-of-thought to decompose questions into code that calls helper functions for document selection, page retrieval, and value extraction. Results show that Vanilla RAG performs poorly (30% accuracy on multi-doc tasks) due to retrieval bottlenecks, often retrieving wrong fiscal years. In contrast, CodeGen+DocS+PageR achieves 80% accuracy by leveraging metadata to filter documents before retrieval. The study highlights that document-level recall is the primary driver of QA performance, and that compound metric extraction remains a significant challenge even for advanced models. The findings underscore the necessity of structured retrieval and code-based reasoning for financial QA, as neural retrieval alone is insufficient for navigating the repetitive and dense structure of SEC filings. The proposed system offers a viable path for automating complex financial data extraction, such as calculating revenue growth or comparing metrics across competitors. However, the approach incurs higher computational costs and latency due to multiple LLM calls. Limitations include the reliance on private databases for ground truth and the difficulty of applying this framework to public sector reports with inconsistent reporting standards, suggesting that the method is currently best suited for regulated, standardized financial disclosures.
Research problem
The paper tackles the lack of robust, non-leaked benchmarks for evaluating LLMs on complex, multi-document financial question answering. It specifically addresses the failure of standard RAG systems to handle quantitative reasoning, multi-hop references, and structural navigation within long SEC filings like 10-Ks and 10-Qs.
Core contributions
- SEC-QA: A dynamic, refreshable benchmark framework for multi-document financial QA using SEC filings.
- A semi-automatic question generation pipeline that creates complex quantitative questions from structured financial databases.
- A program-of-thought QA system that uses code to decompose questions and leverage document metadata for retrieval.
- Comprehensive evaluation showing standard RAG fails on multi-hop financial tasks while code-based methods succeed.
Method
- Semi-automatic generation of QA pairs using a database of financial metrics and linked SEC filings.
- Question templates covering parallel, multi-hop, structural, and multi-output references.
- Program-of-thought approach where LLMs generate Python code to execute retrieval and extraction steps.
- Helper functions: select_document (metadata filtering), retrieve_relevant_pages (vector search), extract_value (LLM extraction).
- Evaluation using Precision@K, Recall@K, and F1@K for retrieval, and exact match with 1% tolerance for QA.
- Stability testing across different years to ensure benchmark consistency and prevent data leakage.
Results & findings
- Vanilla RAG achieved only 30% accuracy on multi-document QA tasks, significantly lagging behind code-based methods.
- CodeGen+DocS+PageR reached 80% accuracy (108/135 questions) by using document selection to filter search spaces.
- Document-level recall had the highest correlation (R2=0.94) with QA accuracy, highlighting retrieval as the bottleneck.
- Compound metric extraction (e.g., Revenue Per Employee) was particularly difficult, with all models performing below 40%.
- CodeGen systems required ~4x more LLM calls than Vanilla RAG, indicating higher operational costs.
- Case studies showed standard RAG frequently retrieved pages from incorrect fiscal years for the same company.
Data & experiments
- Dataset: SEC-QA with questions generated from S&P 500 companies' 10-K, 10-Q, and 8-K filings (2010-2023).
- Metrics: 10 key financial metrics linked to specific documents and pages for precise retrieval evaluation.
- Systems: Vanilla RAG, Multi-Query RAG, CodeGen+PageR, and CodeGen+DocS+PageR.
- Evaluation: Document recall, page recall, and QA accuracy with a 1% margin of error for numerical answers.
- Stability: Tested across five year-based dataset versions to ensure consistent difficulty and prevent leakage.
Limitations & caveats
- Relies on private databases for ground truth, which may not be available for all public documents.
- Higher computational cost and latency due to multiple LLM calls in code generation.
- Difficulty in applying the framework to public sector reports with inconsistent standards.
- Compound metric extraction remains challenging, with models struggling to compute derived values.
- Current evaluation assumes access to structured metadata, which may not always be available.
Curator notes
Semi-automatic question-answer generation from recent filings.