Optimizing Retrieval Strategies for Financial Question Answering Documents in Retrieval-Augmented Generation Systems

In one sentence

Summary

This paper addresses the challenge of retrieving accurate information from complex financial documents, such as 10-K reports, within Retrieval-Augmented Generation (RAG) systems. The authors argue that standard RAG pipelines fail to handle domain-specific vocabulary and multi-hierarchical tabular data effectively. They propose an end-to-end pipeline divided into pre-retrieval, retrieval, and post-retrieval phases to enhance the quality of retrieved context for Large Language Models (LLMs). The core problem is mitigating hallucinations and improving factual grounding in high-stakes financial applications where precision is critical. The methodology involves three key stages. In pre-retrieval, the system employs LLM-based query expansion and corpus markdown restructuring to preserve document structure. For retrieval, the authors fine-tune embedding models (specifically Stella en 1.5B v5) using contrastive learning on financial data and implement a hybrid strategy combining dense embeddings and sparse BM25 retrieval, optimizing a weighting parameter alpha per dataset. Post-retrieval, a reranker refines the top-20 results, and a selection agent filters documents to reduce context noise. Finally, a DPO-trained GPT-4o mini model generates answers. Experiments are conducted on seven datasets (FinDER, FinQABench, FinanceBench, TATQA, FinQA, ConvFinQA, MultiHiertt) using NDCG@10 for retrieval and RAGAS metrics for generation. Results show that fine-tuned hybrid retrieval significantly outperforms baseline models, with NDCG@10 scores improving from ~0.32 to ~0.51. Query expansion and markdown restructuring were identified as the most effective preprocessing techniques. The DPO-trained generation agent achieved higher answer relevance and context precision than standard GPT-4o. Limitations include the difficulty of handling streaming real-time data and potential security vulnerabilities. The work is highly relevant for finance LLM surveys focusing on RAG optimization, information retrieval, and financial question answering systems.

Research problem

The paper addresses the difficulty of retrieving accurate and relevant information from complex financial documents, such as 10-K reports, within Retrieval-Augmented Generation (RAG) systems. Standard RAG pipelines often fail to handle domain-specific vocabulary, multi-hierarchical tabular data, and ambiguous queries, leading to hallucinations and poor factual grounding. The authors aim to improve the retrieval quality to ensure LLMs generate accurate, contextually appropriate responses for financial question answering tasks.

Core contributions

• Proposes a comprehensive three-phase RAG pipeline (pre-retrieval, retrieval, post-retrieval) tailored for financial documents.
• Introduces specific preprocessing techniques, including LLM-based query expansion and corpus markdown restructuring, which outperform default methods.
• Demonstrates the effectiveness of fine-tuning embedding models (Stella en 1.5B v5) with contrastive learning on financial data.
• Implements and optimizes a hybrid retrieval strategy that dynamically balances dense and sparse retrieval scores.
• Develops a selection agent and DPO-trained generation model to refine context usage and improve answer quality.

Method

• Pre-retrieval: LLM-based query expansion and corpus markdown restructuring to enhance semantic context and structure.
• Retrieval: Fine-tuning Stella embedding models using Multiple Negatives Ranking Loss (MNRLoss) on financial pairs.
• Hybrid Retrieval: Linear weighted fusion of dense (embedding) and sparse (BM25) scores, optimizing alpha per dataset.
• Reranking: Using Cross-Encoder models (BGE, Voyage) to re-rank the top-20 retrieved documents.
• Selection Agent: An LLM-based agent filters top-10 documents for factual relevance and clarity to reduce noise.
• Generation: DPO training on GPT-4o mini using preferred/non-preferred response pairs to align with financial expertise.
• Evaluation: NDCG@10 for retrieval ranking; RAGAS metrics for generation quality.

Results & findings

• Fine-tuned Stella en 1.5B v5 achieved an NDCG@10 of 0.50864, significantly outperforming the baseline 0.32178.
• Query Expansion with LLM yielded the highest retrieval performance (NDCG@10 0.48601) among preprocessing methods.
• Corpus Markdown Restructuring was the most effective corpus preprocessing method, outperforming table augmentation/extraction.
• Hybrid retrieval optimal alpha varied by dataset, indicating task-specific needs for dense vs. sparse weighting.
• Reranking with Voyage Rerank-2 improved overall NDCG@10 to 0.59898 compared to 0.51508 with BGE reranker.
• DPO-trained GPT-4o mini outperformed standard GPT-4o in Answer Relevance (0.8924 vs 0.8663) and Context Precision.
• FinBERT performed poorly compared to general-purpose models fine-tuned on financial data, highlighting the need for domain adaptation.

Data & experiments

• Evaluated on seven financial question answering datasets: FinDER, FinQABench, FinanceBench, TATQA, FinQA, ConvFinQA, and MultiHiertt.
• Used NDCG@10 as the primary metric for retrieval ranking quality across all datasets.
• Assessed generation quality using RAGAS metrics: Answer Relevance and Context Precision.
• Compared multiple embedding models (e5-large-v2, GritLM, FinBERT, TAPAS, Stella variants) and rerankers.
• Conducted ablation studies on preprocessing methods, hybrid retrieval weights (alpha), and reranking strategies.

Limitations & caveats

• The pipeline struggles with rapidly changing, time-sensitive financial data (streaming data integration is a future work item).
• Security risks, such as malicious prompt attacks, are acknowledged but not fully addressed in the current pipeline.
• The method assumes static documents; real-world deployment requires robust monitoring for regulatory compliance.
• Performance of table-specific preprocessing (augmentation/extraction) was inferior to markdown restructuring, suggesting limitations in handling complex tables without structural preservation.
• The study focuses on English-language financial documents; multilingual extension is noted as future work.

Curator notes

Discovery status: report_analysis_focus_promoted. Source links: Financial Report Chunking for Effective Retrieval Augmented Generation