The List / RAG, Search, and Knowledge Systems / paper
Financial Report Chunking for Effective Retrieval Augmented Generation
Abstract
Chunking information is a key step in Retrieval Augmented Generation (RAG). Current research primarily centers on paragraph-level chunking. This approach treats all texts as equal and neglects the information contained in the structure of documents. We propose an expanded approach to chunk documents by moving beyond mere paragraph-level chunking to chunk primary by structural element components of documents. Dissecting documents into these constituent elements creates a new way to chunk documents that yields the best chunk size without tuning. We introduce a novel framework that evaluates how chunking based on element types annotated by document understanding models contributes to the overall context and accuracy of the information retrieved. We also demonstrate how this approach impacts RAG assisted Question&Answer task performance. Our research includes a comprehensive analysis of various element types, their role in effective information retrieval, and the impact they have on the quality of RAG outputs. Findings support that element type based chunking largely improve RAG results on financial reporting. Through this research, we are also able to answer how to uncover highly accurate RAG.
In one sentence
Summary
This research addresses the critical preprocessing bottleneck in Retrieval Augmented Generation (RAG) systems applied to complex financial documents. While standard RAG pipelines typically rely on naive paragraph or fixed-token chunking, this approach often ignores the semantic structure of documents, leading to fragmented context and reduced factual accuracy. The authors argue that financial reports, such as 10-Ks and 10-Qs, contain distinct structural elements like tables, titles, and narrative sections that should guide how text is segmented for indexing. The study positions element-based chunking as a superior alternative to size-based strategies, aiming to preserve contextual integrity and improve the precision of information retrieval for downstream LLM generation tasks. The methodology involves a systematic comparison of chunking strategies using the FinanceBench dataset, which contains 141 questions derived from 80 SEC filings. The authors employ the Unstructured library's Chipper model, a vision-encoder-decoder, to identify and extract structural elements (titles, narrative text, tables) from the documents. These elements are then merged into chunks respecting structural boundaries, enriched with metadata like keywords and summaries, and indexed in a Weaviate vector database using sentence transformers. The retrieval performance is evaluated using page-level accuracy and paragraph-level ROUGE/BLEU scores, while the final Q&A accuracy is assessed via manual evaluation and automated GPT-4 comparison against ground-truth answers. Baselines include fixed token sizes (128, 256, 512) and aggregated retrieval results. Results indicate that element-based chunking yields more consistent retrieval performance and higher Q&A accuracy than fixed-size methods. Specifically, the element-based approach achieved a manual Q&A accuracy of 53.19%, outperforming the best fixed-size baseline (48.23% for 512 tokens). The study also found that aggregating results from multiple chunking strategies further improved retrieval metrics, though it risked exceeding LLM context limits. A key finding is that structural chunking eliminates the need for hyperparameter tuning of chunk sizes, offering a more generalizable solution. However, the authors note limitations in automated evaluation consistency and the computational overhead of document understanding models, suggesting future work on broader domain applicability and optimized element relations.
Research problem
The paper addresses the inefficiency of standard chunking methods in RAG systems when processing complex, structured financial documents like SEC filings. It investigates how ignoring document structure leads to fragmented context and reduced retrieval accuracy, proposing that structural element-based chunking can preserve semantic integrity and improve Q&A performance.
Core contributions
- Proposes a novel element-based chunking framework for financial documents that respects structural boundaries (tables, titles, narratives) rather than using fixed token sizes.
- Introduces a comprehensive evaluation of chunking strategies on the FinanceBench dataset, comparing structural chunking against baseline fixed-size methods.
- Demonstrates that structural chunking improves both retrieval consistency (page and paragraph level) and final Q&A accuracy compared to naive approaches.
- Shows that element-based chunking is generalizable without hyperparameter tuning, offering a robust solution for varying document layouts.
- Provides insights into the trade-offs between retrieval aggregation and LLM context limits, highlighting efficiency gains in indexing.
Method
- Document Parsing: Used Chipper vision-encoder-decoder to extract structural elements (titles, narrative text, tables) from SEC filings.
- Chunking Strategy: Merged structural elements into chunks respecting boundaries, avoiding splitting tables or titles, with a max character limit.
- Metadata Enrichment: Added keywords, summaries, and prefixes (first two sentences/table descriptions) to chunks using GPT-4 prompts.
- Indexing: Converted chunks to vectors using sentence-transformers and stored in Weaviate vector database.
- Retrieval: Retrieved top-10 chunks per query using cosine similarity.
- Generation: Prompted GPT-4 with query and retrieved chunks to generate answers.
- Evaluation: Compared generated answers to ground truth using manual review and GPT-4 similarity checks.
- Baselines: Compared against fixed token chunking (128, 256, 512) and aggregated retrieval results.
Results & findings
- Element-based chunking achieved a manual Q&A accuracy of 53.19%, outperforming the best fixed-size baseline (512 tokens) at 48.23%.
- Fixed-size chunking showed inconsistency; Base 128 had high page accuracy (72.34%) but the lowest paragraph-level ROUGE (0.383).
- Aggregating multiple chunking strategies improved retrieval scores (84.4% page accuracy) but caused Q&A failures due to context window limits.
- Element-based chunking required fewer total chunks (20,843) than aggregated baselines (112,155), reducing indexing costs.
- The 'Prefix & Table Description' metadata enrichment variant of element-based chunking achieved the highest paragraph-level ROUGE (0.514).
- GPT-4 automated evaluation showed high correlation with manual results but struggled with detailed explanatory answers.
- Structural chunking proved more generalizable across documents of varying lengths and layouts without tuning.
Data & experiments
- Dataset: FinanceBench, comprising 141 questions from 80 unique SEC filings (10-K, 10-Q, 8-K) after recovering documents from original links.
- Document Understanding: Uses the Chipper model from the Unstructured library to extract structural elements (titles, text, tables) from PDFs.
- Vector Database: Weaviate with sentence-transformer embeddings for indexing and approximate nearest neighbor retrieval.
- Baselines: Fixed token chunking (128, 256, 512 tokens) and aggregated retrieval from multiple chunking strategies.
- Evaluation Metrics: Page-level retrieval accuracy, paragraph-level ROUGE and BLEU scores, and Q&A accuracy (manual and GPT-4 automated).
- Generator: GPT-4 used for answer generation and automated evaluation of answer similarity.
Limitations & caveats
- Evaluation is limited to SEC filings; generalizability to other document types (e.g., biomedical) is untested.
- Automated evaluation using GPT-4 showed discrepancies with manual evaluation for detailed answers.
- Aggregating multiple chunking strategies improved retrieval but exceeded GPT-4 context limits, rendering it ineffective for Q&A.
- The study relies on the Unstructured library's Chipper model, which may have specific biases or limitations.
- Does not explore the impact of different embedding models or vector database configurations beyond the chosen setup.
- Recovery of only 80 out of 150 documents from FinanceBench may introduce selection bias.
Curator notes
Discovery status: round1_promoted_seed_for_round2. Source links: A Scalable Data-Driven Framework for Systematic Analysis of SEC 10-K Filings Using Large Language Models; A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges; Adaptive Query Routing: A Tier-Based Framework for Hybrid Retrieval Across Financial, Legal, and Medical Documents; BloombergGPT: A Large Language Model for Finance; ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering; FinDER: Financial Dataset for