FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generation

In one sentence

Summary

The paper addresses the critical need for accurate, up-to-date information retrieval in financial Question-Answering (QA), where standard LLMs struggle with factual accuracy and hallucination. Existing benchmarks often rely on clear, predefined contexts, failing to capture the ambiguity, brevity, and domain-specific jargon characteristic of real-world professional searches. FinDER is introduced as a rigorous testbed that challenges models to retrieve relevant evidence from large corpora of 10-K filings, simulating the complex information needs of investment professionals. The research positions FinDER as a necessary evolution in financial AI benchmarks, emphasizing the retrieval step as central to reliable financial analysis. The dataset comprises 5,703 query-evidence-answer triplets derived from real-world financial inquiries regarding S&P 500 companies, with evidence manually annotated by investment bank analysts and CPAs. The experimental setup evaluates a RAG pipeline using the RAGAS framework, testing four retrieval models (BM25, GTE, mE5, E5-Mistral) and four generation models (GPT-o1, Claude-3.7-Sonnet, Qwen-QWQ-32B, Deepseek-R1-Distill). The methodology includes assessing retrieval recall, LLM-based reranking of top-10 passages to top-5, and generation correctness and faithfulness across qualitative and quantitative reasoning tasks. The evaluation highlights the impact of query ambiguity by comparing performance on real-world queries versus well-formed, expert-refined versions. Results indicate that dense neural retrievers, particularly E5-Mistral, significantly outperform sparse methods like BM25 in context recall, though performance drops sharply on ambiguous queries. LLM-based reranking effectively improves precision by filtering noisy retrieval results, with models like Claude-3.7-Sonnet and GPT-o1 showing strong reasoning capabilities. However, generation correctness remains low (around 30%) even with retrieved context, highlighting that while retrieval is improved, factual accuracy in complex financial reasoning is still a major hurdle. The paper concludes that robust financial QA requires not just better generation models but superior retrieval strategies and query disambiguation, providing a valuable benchmark for future RAG development in finance.

Research problem

Financial Question-Answering (QA) requires high precision and factual accuracy, but existing benchmarks fail to capture the ambiguity, brevity, and domain-specific jargon of real-world professional searches. Standard LLMs struggle with hallucinations and outdated information, making effective retrieval crucial. The problem is to create a realistic benchmark that evaluates the entire RAG pipeline, from interpreting ambiguous queries to retrieving evidence from 10-K filings and generating accurate answers.

Core contributions

• Creation of FinDER, a dataset of 5,703 expert-annotated query-evidence-answer triplets based on S&P 500 10-K filings.
• Introduction of a benchmark that captures real-world query characteristics, including ambiguity, acronyms, and concise expressions.
• Comprehensive evaluation of state-of-the-art retrieval models (sparse and dense) and LLMs on financial QA tasks.
• Analysis of the impact of query ambiguity on retrieval performance by comparing real-world queries with expert-refined versions.
• Demonstration of the effectiveness of LLM-based reranking in improving retrieval precision for financial documents.

Method

• Data Collection: Scraped 10-K filings from EDGAR for S&P 500 companies; collected queries from financial Q&A service.
• Annotation: Investment analysts and CPAs annotated evidence snippets and answers with cross-validation.
• Retrieval: Parsed HTML to paragraphs; evaluated BM25, GTE, mE5, E5-Mistral using Context Recall.
• Reranking: LLMs reranked top-10 retrievals to top-5; evaluated using F1-score (Precision/Recall).
• Generation: Evaluated GPT-o1, Claude-3.7, Qwen-QWQ, Deepseek-R1 on Correctness and Faithfulness.
• Evaluation Framework: Used RAGAS for automated metrics; LLM-as-a-judge for scoring.
• Context Settings: Tested without context, partial context (top-10), and perfect context (gold standard).

Results & findings

• E5-Mistral (decoder-based) outperforms encoder-based models and BM25 in Context Recall across all financial categories.
• Neural retrievers uniformly outperform BM25, highlighting the value of semantic embeddings in finance.
• Retrieval precision drops significantly on ambiguous real-world queries compared to well-formed ones (e.g., E5-Mistral: 33.9% vs 25.7%).
• LLM-based reranking improves precision, with Claude-3.7-Sonnet and GPT-o1 achieving high F1-scores.
• Generation correctness is low (~30%) even with retrieved context, indicating significant challenges in factual accuracy.
• Claude-3.7-Sonnet achieves the highest Faithfulness scores, while GPT-o1 and Qwen-QWQ excel in arithmetic tasks.
• Providing perfect context dramatically improves correctness (up to ~68%), underscoring the importance of retrieval quality.
• No single model dominates all tasks, suggesting architectural nuances affect performance on specific reasoning types.

Data & experiments

• Dataset: 5,703 triplets from 490 S&P 500 companies' 10-K filings, annotated by investment analysts and CPAs.
• Retrieval Models: BM25, GTE, mE5, E5-Mistral evaluated on Context Recall.
• Generation Models: GPT-o1, Claude-3.7-Sonnet, Qwen-QWQ-32B, Deepseek-R1-Distill evaluated on Correctness and Faithfulness.
• Reranking: LLMs rerank top-10 retrieved passages to top-5, evaluated on F1-score.
• Metrics: RAGAS framework metrics including Context Recall, Context Precision, Correctness, and Faithfulness.
• Comparison: Performance gap between ambiguous real-world queries and well-formed expert-refined queries.

Limitations & caveats

• Dataset is limited to 10-K filings, potentially missing other relevant financial sources like earnings calls or news.
• Evaluation is based on a 10% subset for some experiments, though the full dataset is described.
• LLM-as-a-judge evaluation may introduce biases or inconsistencies compared to human evaluation.
• The dataset covers only S&P 500 companies, limiting generalizability to smaller firms or other markets.
• Quantitative reasoning tasks are a small portion (15.48%) of the dataset, potentially underrepresenting numerical challenges.
• Real-world deployment may face additional challenges like dynamic data updates not captured in static 10-Ks.

Curator notes

Discovery status: round1_promoted_seed_for_round2. Source links: ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering; DocFinQA: A Long-Context Financial Reasoning Dataset; Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models; FinQA: A Dataset of Numerical Reasoning over Financial Data; FinTradeBench: A Comprehensive Benchmark for Fundamental and Technical Analysis in Financial Trading; FinanceBench: A New Benchmark for Financial Question Answering; Financial Report Chunking for Eff