OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain

In one sentence

Summary

The paper addresses the lack of comprehensive, automatic evaluation benchmarks for Retrieval-Augmented Generation (RAG) systems in specialized vertical domains, specifically finance. Existing benchmarks often focus solely on final response quality or lack domain-specific granularity. OmniEval aims to provide an omnidirectional assessment that captures the nuances of financial queries by structuring the evaluation space into a matrix of 16 financial topics and 5 task classes, including extractive QA, multi-hop reasoning, and conversational QA. This positioning allows for a structured, multi-dimensional profile of RAG capabilities rather than a single aggregate score. The methodology involves constructing a diverse knowledge corpus from sources like BSCF, FinGLM, and official web pages, processed via LlamaIndex. Data generation utilizes a multi-agent pipeline powered by GPT-4 to create question-answer pairs aligned with the topic-task matrix, followed by automatic and manual quality inspections, achieving an 87.47% human acceptance rate. The evaluation system combines rule-based metrics (Rouge-L, MAP, MRR) with five fine-tuned LLM-based metrics (Accuracy, Completeness, Hallucination, Utilization, Numerical Accuracy) trained on Qwen2.5-7B. Experiments evaluate various retrievers (BGE-M3, GTE-Qwen2) and generators (Llama3.1, Qwen2.5) on both auto-generated and human-annotated test sets. Results indicate that RAG systems generally outperform close-book LLMs but still exhibit significant room for improvement in financial domains. Performance varies substantially across different topics and tasks, with multi-hop reasoning and conversational QA posing particular challenges. The study highlights that retrievers fine-tuned from LLMs (like GTE-Qwen2) perform better than those trained from scratch. The benchmark reveals imbalances in RAG capabilities across topics, suggesting that pre-training corpus popularity influences retrieval effectiveness. The paper provides open-source code and datasets, serving as a critical resource for advancing RAG reliability in finance.

Research problem

The paper identifies a gap in evaluating RAG systems in vertical domains like finance, where existing benchmarks are often limited to general QA or lack comprehensive, automatic evaluation frameworks that cover diverse scenarios and granular performance metrics. It specifically addresses the need for a benchmark that can assess both retrieval and generation components across a structured matrix of financial topics and task types.

Core contributions

• Introduction of OmniEval, an omnidirectional and automatic RAG benchmark tailored for the financial domain.
• Development of a matrix-based evaluation framework categorizing queries into 16 financial topics and 5 task classes.
• Creation of a multi-dimensional data generation pipeline using GPT-4 and human annotation, achieving high quality.
• Design of a multi-stage evaluation system combining rule-based and fine-tuned LLM-based metrics for comprehensive assessment.
• Open-sourcing of the benchmark code, datasets, and fine-tuned LLM evaluators to facilitate further research.

Method

• Matrix-based scenario definition using 16 financial topics and 5 task classes.
• Multi-agent data generation pipeline using GPT-4 for question-answer pair synthesis.
• Automatic and manual quality inspection agents to filter and correct generated data.
• Integration of rule-based metrics (Rouge-L, MAP, MRR) with LLM-based metrics (ACC, COM, HAL, UTL, NAC).
• Supervised fine-tuning of Qwen2.5-7B-Instruct for reliable LLM-based evaluation.
• Evaluation of retriever and generator components separately and in combination.

Results & findings

• GTE-Qwen2-1.5B retriever outperformed others, attributed to its LLM-based fine-tuning and parameter size.
• RAG systems significantly outperformed close-book LLMs, confirming the value of external knowledge in finance.
• Performance varied widely across topics, with 'Fund' and 'Banking' showing higher scores than 'AI' or 'Derivatives'.
• Multi-hop reasoning and conversational QA tasks were the most challenging for current RAG models.
• Fine-tuned Qwen2.5-7B evaluator achieved 74.4% accuracy against human annotations, surpassing prompting-based methods.
• Human annotation acceptance rate for auto-generated data was 87.47%, validating the generation pipeline.

Data & experiments

• Constructed a knowledge corpus from BSCF, FinGLM, Wikipedia, and official financial websites.
• Generated 11.4k auto-generated and 1.7k human-annotated test examples across 80 topic-task combinations.
• Evaluated four retrievers (BGE-M3, BGE-large-zh, GTE-Qwen2-1.5B, Jina-zh) and four LLMs (Qwen2.5, Llama3.1, Deepseek-v2, Yi15).
• Conducted experiments on auto-generated and human-annotated sets to assess retrieval and generation performance.
• Analyzed performance variations across specific topics and tasks to identify model weaknesses.

Limitations & caveats

• Data corpus distribution is limited by accessibility and licensing, potentially introducing bias.
• Limited human annotation data for training LLM evaluators may affect their reliability.
• Focus on Chinese financial data may limit generalizability to other languages.
• Auto-generated data may still contain subtle biases or errors despite quality checks.
• Evaluation focuses on static QA, not dynamic or real-time financial information retrieval.

Curator notes

Discovery status: round4_promoted. Source links: Multimodal Financial Foundation Models (MFFMs): Progress, Prospects, and Challenges; PyFi: Toward Pyramid-like Financial Image Understanding for VLMs via Adversarial Agents