The List / RAG, Search, and Knowledge Systems / paper
MultiFinRAG: An Optimized Multimodal Retrieval-Augmented Generation Framework for Financial Question Answering
Abstract
Financial documents such as 10-Ks, 10-Qs, and investor presentations span hundreds of pages and combine diverse modalities, including dense narrative text, structured tables, and complex figures. Answering questions over such content often requires joint reasoning across modalities, which strains traditional large language models (LLMs) and retrievalaugmented generation (RAG)[1] pipelines due to token limitations, layout loss, and fragmented cross-modal context. We introduce MultiFinRAG, a retrieval augmented generation framework purpose built for financial QA. MultiFinRAG first performs multimodal extraction by grouping table and figure images into batches and sending them to a lightweight, quantized open-source multimodal LLM, which produces both structured JSON outputs and concise textual summaries. These outputs, along with narrative text, are embedded and indexed with modality aware similarity thresholds for precise retrieval. A tiered fallback strategy then dynamically escalates from text only to text+table+image contexts when necessary, enabling cross modal reasoning while reducing irrelevant context. Despite running on commodity hardware, MultiFinRAG achieves 19 percentage points higher accuracy than ChatGPT-4o[2] (free-tier) on complex financial QA tasks involving text, tables, images, and combined multimodal reasoning.
In one sentence
Summary
The paper addresses the challenge of answering complex questions over long, multimodal financial documents like 10-Ks and 10-Qs, where traditional RAG pipelines fail due to token limits, layout loss, and fragmented cross-modal context. The authors propose MultiFinRAG, a system designed to preserve numerical and visual relationships by converting tables and figures into structured JSON and summaries using quantized open-source multimodal LLMs, while embedding narrative text for semantic search. This approach aims to enable precise retrieval and joint reasoning across text, tables, and images without relying on expensive proprietary models. The methodology involves a three-stage pipeline: multimodal extraction, semantic chunking, and tiered retrieval. Tables and images are batch-processed by Gemma-3 or LLaMA-3.2-Vision to generate descriptions and JSON, which are embedded alongside text chunks using BAAI/bge-base-en-v1.5. The system employs semantic chunk merging to reduce redundancy and uses modality-aware similarity thresholds in a FAISS index. A tiered fallback strategy retrieves text first, escalating to table and image contexts only if initial retrieval is insufficient, thereby optimizing context size and cost. Evaluation is conducted on 300 manually crafted questions spanning text, image, table, and combined modalities, using manual verification for accuracy. Results show MultiFinRAG with Gemma-3 achieves 75.3% accuracy on complex multimodal questions, surpassing ChatGPT-4o (free tier) by 19.3 percentage points and significantly outperforming baseline RAG systems. The framework reduces token usage by 40-60% and runs efficiently on commodity hardware. Limitations include reliance on manual evaluation, potential brittleness in OCR/layout parsing, and the current focus on single-document queries rather than cross-document or longitudinal analysis. The system is positioned as a cost-effective, open-source alternative for financial document understanding.
Research problem
Financial documents are lengthy and multimodal, causing traditional RAG systems to lose layout context and fail at cross-modal reasoning. The paper addresses the need for a retrieval system that can accurately extract and synthesize information from text, tables, and figures in regulatory filings like 10-Ks and 10-Qs.
Core contributions
- A multimodal RAG framework that converts tables and figures into structured JSON and summaries for precise indexing.
- A semantic chunking and merging strategy that reduces context size by 40-60% while preserving coherence.
- A tiered fallback retrieval mechanism that dynamically escalates from text-only to multimodal contexts based on similarity thresholds.
- Empirical demonstration of superior accuracy over ChatGPT-4o on complex financial QA tasks using open-source models.
- A benchmark dataset of 300 manually crafted questions covering text, image, table, and combined modalities.
Method
- Multimodal extraction: Batch processing of tables and images using quantized Gemma-3 or LLaMA-3.2-Vision to generate JSON and summaries.
- Semantic chunking: Sentence segmentation, sliding windows, and merging of chunks based on embedding similarity.
- Indexing: Embedding text, table summaries, and image summaries using BAAI/bge-base-en-v1.5 and storing in FAISS.
- Tiered retrieval: Text-only retrieval first, with fallback to table and image contexts if similarity or count thresholds are not met.
- Threshold calibration: Determining optimal similarity cut-offs for each modality via decision function on a holdout set.
- LLM generation: Final answer generation using the retrieved multimodal context with system prompts to prevent hallucination.
Results & findings
- MultiFinRAG with Gemma-3 achieves 75.3% accuracy on complex multimodal questions, outperforming ChatGPT-4o by 19.3 percentage points.
- For image-based questions, MultiFinRAG (Gemma-3) achieves 66.7% accuracy vs. 23.8% for ChatGPT-4o.
- For table-based questions, MultiFinRAG (Gemma-3) achieves 69.4% accuracy vs. 44.4% for ChatGPT-4o.
- Text-based accuracy for MultiFinRAG (Gemma-3) is 90.4%, slightly higher than ChatGPT-4o's 86.3%.
- Baseline RAG systems perform poorly on multimodal questions, often achieving 0% accuracy.
- The framework reduces token usage by 40-60% through semantic chunk merging.
- Gemma-3 outperforms LLaMA-3.2-Vision across all question types in the MultiFinRAG framework.
- The system operates efficiently on commodity hardware, with ~25 minutes processing time for a 200-page PDF.
Data & experiments
- Dataset: 300 manually crafted questions from SEC filings (10-K, 10-Q, 8-K, DEF 14A) covering text, image, table, and combined types.
- Baselines: Standard RAG with fixed chunks, MultiFinRAG with LLaMA-3.2-Vision, MultiFinRAG with Gemma-3, and ChatGPT-4o (free tier).
- Metrics: End-to-end QA accuracy evaluated via manual verification by domain experts.
- Hardware: Experiments conducted on Google Colab T4 GPU with quantized open-source models.
- Evaluation: Comparison of accuracy across different question types and model configurations.
Limitations & caveats
- Evaluation relies on manual verification, which may not scale to larger datasets.
- OCR and layout parsing can be brittle for low-quality scans or unusual formats.
- Current implementation focuses on single-document queries, lacking cross-document reasoning.
- Performance may vary with different PDF structures and document types.
- The framework's effectiveness depends on the quality of the multimodal LLM's extraction capabilities.
Curator notes
Discovery status: report_analysis_focus_promoted. Source links: Financial Report Chunking for Effective Retrieval Augmented Generation