Evaluating Retrieval-Augmented Generation Models for Financial Report Question and Answering

In one sentence

Summary

The paper addresses the challenge of enabling private investors to accurately analyze half-yearly and quarterly bank financial reports using Large Language Models (LLMs). It identifies a gap in tailored RAG solutions for this specific use case, where manual analysis is labor-intensive and standard LLMs suffer from hallucinations and outdated knowledge. The research aims to enhance context relevance, answer faithfulness, and answer relevance by developing and evaluating a custom RAG system designed for financial report question-answering. The authors employ a Design Science Research (DSR) methodology to build and test three distinct RAG model configurations using the Verba RAG application. The dataset consists of quarterly and half-yearly reports from five major European banks (Barclays, HSBC, Credit Suisse, Credit Agricole, and Banco Santander) for 2022 and 2023, along with irrelevant reports to test filtering. The system uses a 100-word chunk size with 50-token overlap, a hybrid WindowRetriever, and various embedding models (OpenAI ADA, MiniLM) and LLMs (GPT-4, Llama 3, Gemini 1.5 Pro). Evaluation is conducted manually on 10 questions per bank, covering both quantitative and qualitative aspects. Results indicate that Model One (OpenAI ADA embeddings with GPT-4 generation) achieved the highest performance in accuracy and relevance, while Model Three (MiniLM embeddings with GPT-4) scored significantly lower, highlighting the critical importance of high-quality embedding models. The study finds that well-structured reports yield better RAG performance than less coherent ones. Additionally, qualitative questions received higher scores than quantitative ones, suggesting current RAG systems are more proficient in handling descriptive data than precise numerical extraction. Limitations include difficulties with complex PDF layouts and the need for domain-specific terminology repositories.

Research problem

The paper addresses the difficulty private investors face in analyzing bank financial reports efficiently and accurately. It highlights that standard LLMs struggle with domain-specific knowledge, hallucinations, and outdated information. The specific problem is the lack of tailored RAG systems that can reliably extract and synthesize information from quarterly and half-yearly bank reports to support investment decisions.

Core contributions

• Develops a tailored RAG system using the Verba application for analyzing bank financial reports.
• Conducts empirical experiments comparing three distinct RAG model configurations with varying embedding and generation components.
• Evaluates RAG performance on context relevance, answer faithfulness, and answer relevance using manual assessment.
• Demonstrates the significant impact of embedding model quality on overall RAG system performance.
• Identifies that qualitative questions are answered more accurately than quantitative ones in this context.

Method

• Design Science Research (DSR) methodology for artifact development and evaluation.
• Document chunking: 100-word tokens with 50-token overlap to balance context and precision.
• Hybrid retrieval: WindowRetriever combining semantic and keyword search.
• Vector storage: Weaviate cloud service for efficient similarity search.
• Model configurations: Testing different combinations of embedding models (OpenAI ADA, MiniLM) and LLMs (GPT-4, Llama 3, Gemini 1.5 Pro).
• Manual evaluation: Human assessment of context relevance, answer faithfulness, and answer relevance.
• Filtering test: Inclusion of irrelevant reports to assess the system's ability to ignore noise.

Results & findings

• Model One (OpenAI ADA + GPT-4) achieved the highest performance in accuracy and relevance.
• Model Three (MiniLM + GPT-4) scored significantly lower, indicating embedding quality is crucial.
• Well-structured reports resulted in better RAG performance than less coherent reports.
• Qualitative questions received higher evaluation scores than quantitative questions.
• The study confirms RAG can reduce hallucinations but struggles with complex PDF layouts.
• Domain-specific terminology variations between banks pose challenges for retrieval.

Data & experiments

• Dataset: Quarterly and half-yearly reports from Barclays, HSBC, Credit Suisse, Credit Agricole, and Banco Santander for 2022-2023.
• Additional irrelevant reports from Bawag Bank, Erste Bank, and KB Bank to test data filtering.
• 10 questions per bank, split between quantitative (e.g., revenue, profit) and qualitative (e.g., risks, strategies) types.
• Three RAG configurations tested: varying embedding models (OpenAI ADA, MiniLM) and LLMs (GPT-4, Llama 3, Gemini 1.5 Pro).
• Manual evaluation of context relevance, answer faithfulness, and answer relevance.
• Chunking strategy: 100-word tokens with 50-token overlap.

Limitations & caveats

• Small sample size of only five banks limits the generalizability of findings.
• Manual evaluation is subjective and not scalable compared to automated metrics.
• The system struggles with complex PDF layouts, requiring future computer vision integration.
• Quantitative questions are answered less accurately, indicating a gap in numerical reasoning.
• Domain-specific terminology variations across banks are not fully resolved.
• The study focuses on European banks, which may differ from US or Asian reporting standards.

Curator notes

Discovery status: report_analysis_focus_promoted. Source links: Metadata-Driven Retrieval-Augmented Generation for Financial Question Answering