Decomposing Retrieval Failures in RAG for Long-Document Financial Question Answering

In one sentence

Summary

This paper addresses the critical failure mode in financial Retrieval-Augmented Generation (RAG) where the correct regulatory filing is retrieved, but the specific page or chunk containing the answer is missed. The authors argue that standard document-level metrics obscure within-document retrieval errors, which lead to hallucinations when generators extrapolate from incomplete context. They propose an oracle-based analysis to decompose retrieval performance into document, page, and chunk levels, providing empirical upper bounds on performance. The study focuses on Financial Question Answering (QA) over long SEC filings, highlighting the need for granular retrieval to support verifiable, high-stakes financial analysis. The authors evaluate diverse retrieval strategies on a 150-question subset of FinanceBench, including dense, sparse, hybrid, hierarchical, and query reformulation methods. They introduce a novel domain-fine-tuned page scorer using a bi-encoder trained with contrastive learning on financial filings. This scorer ranks pages before chunk retrieval, filtering the search space to relevant sections. The experimental setup uses Qwen-2.5-7B-Instruct for generation and measures performance via document recall, page recall, and chunk-level BLEU/ROUGE-L scores. The page scorer is trained on document-level splits to prevent leakage and evaluated using 5-fold cross-validation. Results show that while baseline methods achieve high document recall, page recall lags significantly, with the best baseline at 0.46 compared to an oracle document bound of 0.60. The proposed page scorer achieves 0.55 page recall, outperforming all baselines and improving chunk retrieval metrics. It particularly excels on metrics-generated questions, exceeding the oracle document bound. However, performance drops on earnings call transcripts due to structural differences. The study concludes that explicit page-level modeling improves within-document retrieval, though generalization to other document types and the need for gold annotations during training remain limitations.

Research problem

The paper addresses the specific problem of within-document retrieval failures in financial RAG systems, where the correct document is found but the precise page or chunk containing the answer is missed. This leads to generators producing plausible but incorrect answers due to incomplete context. The authors aim to decompose retrieval failures into document discovery versus page/chunk retrieval to quantify the gap and propose methods to close it.

Core contributions

• Introduction of an oracle-based retrieval gap analysis that decomposes retrieval into document, page, and chunk levels.
• Comprehensive evaluation of diverse retrieval strategies (dense, sparse, hybrid, hierarchical, query reformulation) on FinanceBench.
• Development of a domain-fine-tuned page scorer that treats pages as intermediate retrieval units between documents and chunks.
• Demonstration that page-level filtering significantly improves chunk retrieval and downstream generation accuracy compared to direct chunk search.

Method

• Oracle-based analysis to quantify retrieval gaps at document, page, and chunk levels.
• Evaluation of dense, sparse, hybrid, and hierarchical retrieval strategies using ChromaDB.
• Implementation of query reformulation techniques including HyDE and Multi-HyDE.
• Cross-encoder reranking using BAAI/bge-reranker-v2-m3 to refine initial retrieval results.
• Training of a bi-encoder page scorer using contrastive learning with Multiple Negatives Ranking loss.
• Two-stage retrieval pipeline: page ranking followed by chunk retrieval within selected pages.
• Use of numeric match accuracy to evaluate the correctness of generated financial answers.

Results & findings

• Baseline methods show high document recall but low page recall, with the best baseline achieving 0.46 page recall.
• Oracle document setting reveals a page recall upper bound of 0.60, indicating significant headroom for improvement.
• The proposed page scorer achieves 0.55 page recall, outperforming all baselines and closing the gap to the oracle document bound.
• On metrics-generated questions, the page scorer exceeds the oracle document page recall of 0.68, achieving 0.81.
• Downstream generation accuracy (numeric match) improves from 0.12 (best baseline) to 0.30 with the page scorer.
• Performance on earnings call transcripts is weak (0.10 page recall) due to non-standard structure compared to 10-K filings.
• Chunk-level BLEU and ROUGE-L scores improve significantly with the page scorer, validating the filtering approach.

Data & experiments

• Evaluation on a 150-question subset of FinanceBench, covering 10-K, 10-Q, 8-K, and earnings call transcripts.
• Comparison of multiple retrieval baselines including BGE-M3, BM25, SPLADE, HyDE, Multi-HyDE, and cross-encoder reranking.
• Use of oracle document and oracle page settings to establish empirical upper bounds for retrieval performance.
• Evaluation metrics include document recall, page recall, chunk-level BLEU/ROUGE-L, and numeric match accuracy for generated answers.
• 5-fold cross-validation for training the page scorer to prevent data leakage across document types.

Limitations & caveats

• Evaluation is limited to 150 questions from FinanceBench, predominantly 10-K filings.
• The page scorer requires gold annotations during training, which may not be available in all settings.
• Performance on earnings call transcripts is poor due to structural differences from training data.
• Reliance on random in-batch negatives during training may limit the effectiveness of the page scorer.
• Generative results are based on a single LLM (Qwen-2.5-7B-Instruct), limiting generalizability of generation metrics.
• Evaluation metrics like ROUGE-L and numeric match may not fully capture correctness in complex financial contexts.

Curator notes

Discovery status: report_analysis_focus_promoted. Source links: Hierarchical Retrieval with Evidence Curation for Open-Domain Financial Question Answering on Standardized Documents; Optimizing Retrieval Strategies for Financial Question Answering Documents in Retrieval-Augmented Generation Systems