Metadata-Driven Retrieval-Augmented Generation for Financial Question Answering

In one sentence

Summary

The paper addresses the challenge of Retrieval-Augmented Generation (RAG) in long, structured financial filings where relevant evidence is sparse and cross-referenced. Standard dense retrieval often fails to capture the hierarchical structure and implicit relationships within annual reports, leading to poor precision. The authors position their work as a systematic investigation into metadata-driven RAG, aiming to treat documents as hierarchical knowledge structures rather than flat collections of chunks. This approach seeks to improve the reliability and accuracy of financial question-answering systems by integrating multi-level metadata throughout the retrieval pipeline. The methodology involves an offline indexing pipeline where LLMs generate document-level summaries, key entities, and thematic clusters, along with chunk-level enrichments such as parent references and potential question-answer pairs. The authors benchmark a spectrum of enhancements including pre-retrieval filtering, post-retrieval reranking, and enriched embeddings. They evaluate these techniques on the FinanceBench dataset, which contains 150 manually annotated question-answer-evidence triples from 10-K filings, using the RAGChecker framework for fine-grained evaluation of precision, recall, faithfulness, and hallucination. The experimental design compares naive RAG, hybrid retrieval, and various metadata-integrated architectures. Results indicate that while powerful rerankers are essential for precision, the most significant performance gains come from embedding chunk metadata directly with text, creating "contextual chunks." The optimal architecture combines LLM-driven pre-retrieval optimizations (file filtering and query rewriting) with these contextual embeddings. The study also presents a custom metadata reranker that offers a cost-effective alternative to commercial solutions. Limitations include the reliance on a relatively small public subset of FinanceBench and the computational overhead of LLM-based metadata generation. The paper provides a blueprint for building robust, metadata-aware RAG systems for financial document analysis, highlighting the trade-off between peak performance and operational efficiency.

Research problem

Retrieval-Augmented Generation (RAG) struggles with long, structured financial filings where relevant evidence is sparse and cross-referenced. Standard dense retrieval methods often fail to identify the most relevant passages because they treat document chunks in isolation, ignoring the hierarchical structure and implicit relationships within financial reports. This leads to poor precision and recall in financial question-answering tasks, which require high accuracy and reliability.

Core contributions

• Proposes a novel, multi-stage RAG architecture that leverages LLM-generated metadata to treat documents as hierarchical knowledge structures.
• Introduces a sophisticated indexing pipeline that generates document-level summaries, key entities, thematic clusters, and chunk-level enrichments.
• Systematically benchmarks a spectrum of RAG enhancements, including pre-retrieval filtering, post-retrieval reranking, and enriched embeddings.
• Demonstrates that embedding chunk metadata directly with text ("contextual chunks") yields the most significant performance gains.
• Presents a custom metadata reranker that offers a cost-effective alternative to commercial solutions, highlighting practical trade-offs.

Method

• Offline indexing pipeline using LLMs to generate document-level summaries, key entities, and thematic clusters.
• Chunk-level enrichment with parent cluster references, associated entities, and potential question-answer pairs.
• Pre-retrieval optimization via intelligent file filtering and query rewriting guided by document-level metadata.
• Post-retrieval refinement using metadata-driven entity and cluster exploration and a custom metadata reranker.
• Semantic embedding enrichment by embedding chunks together with their metadata to create contextually richer vectors.
• Evaluation using RAGChecker to measure precision, recall, faithfulness, and hallucination on FinanceBench.

Results & findings

• Contextual chunks, created by embedding chunk metadata directly with text, provided the most significant performance gains over baseline RAG.
• A powerful reranker is essential for precision, but the combination of pre-retrieval optimizations and contextual embeddings yields superior overall performance.
• The proposed optimal architecture combines LLM-driven pre-retrieval filtering and query rewriting with contextual embeddings.
• The custom metadata reranker offers a compelling, cost-effective alternative to commercial solutions, balancing accuracy and operational efficiency.
• RAGChecker metrics revealed that metadata-driven approaches improve both context precision and claim recall, reducing noise and hallucination.

Data & experiments

• Evaluated on the FinanceBench dataset, specifically the open-source subset containing 150 manually annotated question-answer-evidence triples from 10-K filings.
• Used the RAGChecker evaluation framework for fine-grained assessment of retrieval and generation quality, including precision, recall, faithfulness, and hallucination.
• Compared naive RAG, hybrid retrieval, and various metadata-integrated architectures against the proposed optimal architecture.
• Assessed the impact of pre-retrieval filtering, query rewriting, contextual chunk embedding, and custom reranking on retrieval precision and answer quality.

Limitations & caveats

• Evaluation is based on a relatively small public subset of FinanceBench (150 cases), which may limit generalizability.
• The computational overhead of LLM-based metadata generation could be a barrier for real-time applications.
• The study focuses on 10-K filings, and performance on other financial document types (e.g., earnings calls, 8-Ks) is not extensively tested.
• The custom metadata reranker's performance may vary across different financial domains and document structures.
• The paper does not address the potential biases introduced by LLM-generated metadata.

Curator notes

Discovery status: report_analysis_focus_promoted. Source links: DocFinQA: A Long-Context Financial Reasoning Dataset; Financial Report Chunking for Effective Retrieval Augmented Generation