The List / Multimodal and Multilingual Finance / paper
FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation
Abstract
Multimodal Large Language Models (MLLMs) have experienced rapid development in recent years. However, in the financial domain, there is a notable lack of effective and specialized multimodal evaluation datasets. To advance the development of MLLMs in the finance domain, we introduce FinMME, encompassing more than 11,000 high-quality financial research samples across 18 financial domains and 6 asset classes, featuring 10 major chart types and 21 subtypes. We ensure data quality through 20 annotators and carefully designed validation mechanisms. Additionally, we develop FinScore, an evaluation system incorporating hallucination penalties and multi-dimensional capability assessment to provide an unbiased evaluation. Extensive experimental results demonstrate that even state-of-the-art models like GPT-4o exhibit unsatisfactory performance on FinMME, highlighting its challenging nature. The benchmark exhibits high robustness with prediction variations under different prompts remaining below 1%, demonstrating superior reliability compared to existing datasets. Our dataset and evaluation protocol are available at https://huggingface.co/datasets/luojunyu/FinMME and https://github.com/luo-junyu/FinMME.
In one sentence
Summary
The paper addresses the critical lack of specialized evaluation frameworks for Multimodal Large Language Models (MLLMs) in the finance domain, where high knowledge density and complex visual data require rigorous assessment. Existing general benchmarks fail to capture the nuances of financial analysis, such as interpreting diverse chart types and performing domain-specific reasoning. The authors position FinMME as a comprehensive solution to bridge this gap, providing a structured, high-quality dataset designed to test and improve MLLM capabilities in real-world financial scenarios, including equity research, macroeconomic analysis, and asset allocation. FinMME comprises 11,099 samples spanning 18 core financial domains and 6 asset classes, featuring 10 major chart types and 21 subtypes. Data was curated from professional research reports and web sources, with a rigorous annotation pipeline involving 20 human annotators and LLM-based consistency checks to maintain error rates below 1%. The evaluation framework, FinScore, incorporates domain-normalized scoring and hallucination penalties to ensure unbiased assessment. Experiments evaluated 17 proprietary and open-source models, including GPT-4o, Claude 3.5 Sonnet, and Qwen2.5-VL, using multiple-choice and calculation questions across comprehensive perception, fine-grained perception, and cognitive reasoning dimensions. Results indicate that even leading models like GPT-4o achieve only around 46% average accuracy, with calculation tasks being the most challenging. Proprietary models generally outperform open-source alternatives, though Qwen2.5-VL-72B shows competitive performance in specific domains. The benchmark demonstrates high robustness, with prediction variations under different prompts remaining below 1%. However, the study acknowledges limitations, including a reliance on multiple-choice formats that may not fully capture complex financial analysis, potential biases in expert annotations, and a lack of audio/video content. The dataset is released to facilitate future research in financial multimodal reasoning.
Research problem
The financial domain lacks effective, specialized multimodal evaluation datasets for MLLMs, despite the sector's high knowledge density and reliance on complex visual data like charts and reports. Existing general benchmarks do not adequately test domain-specific reasoning, hallucination resistance, or the ability to interpret diverse financial visualizations, creating a gap in evaluating model readiness for professional financial applications.
Core contributions
- Introduction of FinMME, a large-scale dataset with over 11,000 high-quality financial multimodal samples across 18 domains and 6 asset classes.
- Development of a rigorous annotation pipeline using 20 annotators and LLM-based consistency checks to ensure high data quality and low error rates.
- Proposal of FinScore, an evaluation metric that combines domain-normalized performance with hallucination penalties to assess reliability in financial contexts.
- Comprehensive benchmarking of 17 state-of-the-art MLLMs, revealing significant performance gaps in financial reasoning compared to general tasks.
- Release of the dataset and evaluation code to the community to advance multimodal research in finance.
Method
- Data collection from professional research reports and web sources with automated deduplication and manual review.
- Annotation by 20 human annotators (12 junior, 8 experts) with LLM-based external and internal consistency checks.
- Expert review for resolving discrepancies between human and LLM annotations.
- FinScore calculation combining domain-normalized accuracy with hallucination penalty rates.
- Evaluation using multiple-choice and calculation questions across three cognitive levels.
- Stability analysis measuring standard deviations across multiple test runs.
Results & findings
- GPT-4o achieves an average score of 46.56% and a FinScore of 15.34, indicating unsatisfactory performance on complex financial tasks.
- Gemini Flash 2.0 leads proprietary models with an average score of 51.85 and FinScore of 20.10.
- Qwen2.5-VL-72B is the top open-source model, achieving an average score of 52.54 and FinScore of 20.87, outperforming some proprietary models.
- Calculation questions are the most difficult dimension, with top models scoring below 40% accuracy.
- Models perform better in traditional sectors like energy and metals but struggle in economics and fixed income.
- Hallucination penalties significantly impact FinScore, highlighting the need for reliability in financial applications.
Data & experiments
- Dataset contains 11,099 samples with 10 major chart types and 21 subtypes, covering equity, fixed income, derivatives, commodities, forex, and cross-asset classes.
- Data sourced from professional research reports and web pages, with strict copyright compliance and three-stage cleaning processes.
- Evaluation includes 17 models (6 proprietary, 11 open-source) tested on single-answer, multiple-answer, and calculation questions.
- Experiments measure performance across three cognitive levels: comprehensive perception, fine-grained perception, and cognitive reasoning.
- Stability analysis confirms low prediction variance (<1%) across different prompts, validating the benchmark's reliability.
Limitations & caveats
- Evaluation relies primarily on multiple-choice and calculation questions, which may not capture the full complexity of real-world financial analysis.
- Potential biases in expert annotations, as complex financial concepts posed interpretation difficulties even for knowledgeable annotators.
- Dataset may not cover all scenarios encountered in financial work due to the vast and evolving nature of the industry.
- Lacks integration with audio/video content and real-time data analysis, limiting its scope to static visual and textual data.
- Robustness findings may not generalize to noisy or distorted inputs, as the study focused on high-quality inputs.
Curator notes
Discovery status: round4_promoted. Source links: FinEvo: From Isolated Backtests to Ecological Market Games for Multi-Agent Financial Strategy Evolution; PyFi: Toward Pyramid-like Financial Image Understanding for VLMs via Adversarial Agents