Hierarchical Retrieval with Evidence Curation for Open-Domain Financial Question Answering on Standardized Documents

In one sentence

Summary

The paper addresses the challenge of applying Retrieval-Augmented Generation (RAG) to standardized financial documents like SEC filings, where repetitive boilerplate text and similar table structures cause traditional retrieval methods to misidentify near-duplicates, leading to irrelevant or redundant context. The authors propose the Hierarchical Retrieval with Evidence Curation (HiREC) framework to mitigate these issues. HiREC employs a two-stage retrieval process: first, it retrieves relevant documents using cover-page summaries to narrow the search space, and second, it selects specific passages within those documents. This hierarchical approach reduces confusion among similar texts. Additionally, an evidence curation module filters out irrelevant passages and generates complementary queries when initial evidence is insufficient, ensuring complete information for complex comparative questions. To evaluate the framework, the authors construct LOFin, a large-scale open-domain financial question-answering benchmark comprising 145,897 SEC filings (10-K, 10-Q, 8-K) from S&P 500 companies and 1,595 question-answer pairs. The experimental design compares HiREC against state-of-the-art RAG baselines (e.g., Dense, IRCoT, Self-RAG) and commercial systems (Perplexity, SearchGPT). The method utilizes a fine-tuned DeBERTa-v3 cross-encoder for passage retrieval, an E5 bi-encoder for document retrieval, and GPT-4o for answer generation using Program-of-Thought or Chain-of-Thought prompting. Metrics include page-level recall/precision and answer accuracy across numeric and textual categories. HiREC outperforms all baselines, achieving at least 13% higher answer accuracy and 10% higher page recall than the second-best model. It demonstrates superior cost-efficiency, using fewer tokens and lower API costs while maintaining high precision. The framework effectively handles multi-document and multi-hop reasoning, though it faces limitations in temporal aggregation across multiple years and complex numerical computations from tables. The approach also shows generalizability to legal document retrieval, indicating robustness beyond the financial domain.

Research problem

Standardized financial documents (e.g., SEC filings) share similar formats, repetitive boilerplate, and table structures, which cause traditional RAG systems to retrieve near-duplicate or irrelevant text. This undermines the accuracy and completeness of open-domain financial question answering. The problem is to design a retrieval and curation mechanism that can distinguish relevant evidence from noisy, standardized content and iteratively fill information gaps for complex queries.

Core contributions

• Proposes HiREC, a hierarchical retrieval framework that first retrieves relevant documents and then selects pertinent passages to reduce confusion from near-duplicate content.
• Introduces an evidence curation process that filters irrelevant passages and generates complementary queries to supplement missing information for accurate financial QA.
• Constructs and releases LOFin, a large-scale open-domain financial QA benchmark with 145,897 SEC documents and 1,595 QA pairs, addressing challenges in standardized document retrieval.

Method

• Hierarchical retrieval: Document-level retrieval using cover-page summaries and E5 bi-encoder, followed by passage-level retrieval using a fine-tuned DeBERTa-v3 cross-encoder.
• Query transformation: LLM rewrites user questions to remove noisy financial terms and focus on key metrics.
• Evidence curation: Jointly performs passage filtering, answerability checking, and complementary question generation using a single LLM prompt.
• Complementary retrieval: Iteratively retrieves additional documents based on generated complementary queries when initial evidence is insufficient.
• Answer generation: Uses Program-of-Thought (PoT) for numeric questions and Chain-of-Thought (CoT) for textual questions.
• Fine-tuning: Cross-encoder trained on FinQA table data to better capture financial table semantics.

Results & findings

• HiREC improves answer accuracy by at least 13% compared to existing RAG methods.
• HiREC achieves at least 10% higher page recall than the second-best baseline (Dense).
• HiREC outperforms commercial LLMs with web search (Perplexity, SearchGPT) on financial QA tasks.
• HiREC uses an average of only 3.7 passages per query, demonstrating high efficiency.
• Ablation shows hierarchical retrieval is critical for accuracy; removing it drops performance significantly.
• Evidence curation improves precision by filtering irrelevant passages, preventing conflicts in generation.
• HiREC + Deepseek-14B outperforms Dense + GPT-4o, showing robustness with smaller generators.
• Cost analysis shows HiREC uses significantly fewer tokens and lower costs than iterative baselines like IRCoT.

Data & experiments

• LOFin benchmark: 145,897 SEC filings (10-K, 10-Q, 8-K) from S&P 500 companies (2001-2025) and 1,595 open-domain QA pairs.
• Evaluation on numeric (table/text) and textual question categories.
• Comparison against baselines: Dense, HybridSearch, HHR, IRCoT, RQ-RAG, Self-RAG, and commercial systems Perplexity and SearchGPT.
• Metrics: Page-level recall/precision and answer accuracy (GPT-4o evaluation for textual, exact match for numeric).
• Ablation studies on hierarchical retrieval, evidence curation, and fine-tuning components.
• Cost-efficiency analysis comparing token usage and API costs.

Limitations & caveats

• Retrieval failures occur when questions require aggregating data across multiple fiscal years (temporal scope reasoning).
• Generation failures can happen due to faulty arithmetic reasoning even with correct retrieval.
• Performance on multi-document QA is lower than single-document, indicating complexity in synthesis.
• Relies on LLMs for evidence curation, which may introduce latency or cost.
• Benchmark is limited to S&P 500 companies, potentially limiting generalizability to smaller firms.
• Fine-tuning on FinQA may bias the retriever towards specific table structures.

Curator notes

Discovery status: report_analysis_focus_promoted. Source links: DocFinQA: A Long-Context Financial Reasoning Dataset; SEC-QA: A Systematic Benchmark for Evaluating Long-Context Question Answering on SEC Filings