The List / Multimodal and Multilingual Finance / paper
MME-Finance: A Multimodal Finance Benchmark for Expert-level Understanding and Reasoning
Abstract
To date, there is a notable lack of rigorous benchmarks that assess Multimodal Large Language Models (MLLMs) within the financial domain, a field characterized by specialized financial charts and complex domain-specific expertise. To address this gap, we introduce MME-Finance, the first comprehensive bilingual multimodal benchmark tailored for financial analysis. MME-Finance comprises 4,751 meticulously curated samples, encompassing 2,274 open-ended questions, 2,000 binary-choice questions, and 477 multi-turn questions. To mitigate bias when LLMs act as judges, we also created an evaluation framework that strengthens alignment with human judgments by embedding visual context into the multimodal assessment pipeline. A comprehensive evaluation of 31 popular MLLMs has been conducted to assess their perception, reasoning, and cognitive capabilities. Gemini2.5Pro achieves highest accuracy of 79.28% and 85.71% on the open-ended questions and multi-turn questions, respectively. Among open-source models, InternVL3-78B attains 71.24 % accuracy on the open-ended question, whereas Qwen2.5-VL-72B achieves an F1 score of 88.73 % on the binary-choice question. The results indicate that state-of-the-art MLLMs demonstrate considerable overall competence, yet exhibit significant deficiencies in fine-grained visual perception and the understanding of domain-specific financial images. Source code is available at https://github.com/HiThink-Research/MME-Finance.
In one sentence
Summary
The paper addresses the lack of rigorous benchmarks for Multimodal Large Language Models (MLLMs) in the financial domain, where unique challenges include specialized jargon, complex chart types like candlesticks, and high data density. Existing general benchmarks fail to capture these nuances, necessitating a domain-specific evaluation tool to guide the development of large financial models. The authors introduce MME-Finance, the first comprehensive bilingual (English and Chinese) multimodal benchmark designed to assess expert-level understanding and reasoning in finance. The benchmark comprises 4,751 curated samples, including open-ended, binary-choice, and multi-turn questions, covering six image types: candlestick charts, technical indicator charts, statistical charts, tables, documents, and mixed charts. Data collection simulates real-world usage via computer screenshots and mobile photographs. Annotation involves experts with 10+ years of experience. The evaluation pipeline uses an LLM-based evaluator (GPT-4o) that incorporates visual context to score open-ended responses, achieving high consistency with human judges. The study evaluates 19 mainstream MLLMs, including open-source models like Qwen2VL-72B and proprietary ones like GPT-4o, across perception, reasoning, and cognition tasks. Results indicate that state-of-the-art MLLMs exhibit significant deficiencies in fine-grained visual perception and domain-specific image understanding. GPT-4o achieved the highest accuracy on open-ended questions (79.28% in the abstract, though text reports 63.18% overall score normalized), while Qwen2VL-72B led open-source models. Models performed poorly on candlestick and technical indicator charts, as well as on mobile photographs due to resolution and angle issues. Spatial awareness and estimated numerical calculations were particularly challenging. The paper highlights that models excelling in general benchmarks do not necessarily perform well in finance, underscoring the need for specialized training and evaluation.
Research problem
The financial domain presents unique challenges for Multimodal Large Language Models (MLLMs), including specialized jargon, complex chart types (e.g., candlestick, technical indicators), and high data density. Current general multimodal benchmarks fail to assess these specific capabilities, leaving a gap in evaluating and guiding the development of financial MLLMs. There is also a lack of bilingual benchmarks to support diverse user bases.
Core contributions
- Introduction of MME-Finance, the first bilingual multimodal benchmark for financial analysis, containing 4,751 samples across English and Chinese.
- Development of a hierarchical ability structure assessing perception, reasoning, and cognition, tailored to financial tasks like investment advice and risk warning.
- Creation of a novel LLM-based evaluation framework that integrates visual context to improve consistency with human judgments for open-ended questions.
- Comprehensive evaluation of 19 mainstream MLLMs, revealing specific weaknesses in financial chart interpretation and spatial awareness.
- Provision of a Chinese version of the benchmark to facilitate evaluation in non-English financial contexts.
Method
- Data Collection: Sourced from financial platforms using screenshots and mobile photography to simulate real-world scenarios.
- Annotation: Questions generated via GPT-4o and refined by experts with 10+ years of financial experience.
- Ability Hierarchy: Tasks categorized into Perception (OCR, spatial awareness), Reasoning (numerical calculation), and Cognition (investment advice, risk warning).
- Evaluation Pipeline: MLLMs generate responses; GPT-4o evaluates responses against ground truth with visual context.
- Scoring: 6-level scoring system (0-5) normalized to percentage, with few-shot examples in prompts.
- Consistency Check: Spearman’s rank correlation calculated between LLM evaluator and human experts.
- Bilingual Support: Parallel evaluation in English and Chinese to assess cross-lingual capabilities.
Results & findings
- GPT-4o achieved the highest accuracy on open-ended questions (79.28%) and multi-turn questions (85.71%) in the abstract summary, though overall normalized scores were lower.
- Qwen2VL-72B was the top-performing open-source model, achieving 71.24% accuracy on open-ended questions and 88.73% F1 on binary-choice.
- Models struggled significantly with candlestick charts and technical indicator charts, indicating a lack of domain-specific visual understanding.
- Spatial awareness was the most challenging perception task, with top models achieving only ~30% accuracy.
- Estimated numerical calculation was harder than accurate calculation, highlighting reasoning limitations.
- Mobile photographs posed difficulties due to lower resolution and oblique angles, affecting OCR and entity recognition.
- Hallucination was prevalent; many models failed to identify 'Not Applicable' questions correctly.
- GPT-4o with image input showed the highest consistency with human evaluators (Spearman's 0.720).
Data & experiments
- Dataset: 4,751 samples (2,274 open-ended, 2,000 binary-choice, 477 multi-turn) in English and Chinese.
- Image Types: Candlestick charts, technical indicator charts, statistical charts, tables, documents, and mixed charts.
- Image Styles: Computer screenshots, mobile photographs, vertical/horizontal mobile screenshots.
- Models Evaluated: 19 MLLMs including Qwen2VL-72B, InternVL2-76B, GPT-4o, GPT-4o-mini, and others.
- Metrics: Accuracy scores normalized to 0-100%, Spearman’s rank correlation for evaluator consistency.
- Evaluation: LLM-as-a-judge with visual input, validated against human expert annotations.
Limitations & caveats
- The benchmark focuses on static images; dynamic video or real-time streaming data is not covered.
- Evaluation relies on LLM-as-a-judge, which may still have biases despite human consistency checks.
- The dataset size, while comprehensive, may not cover all niche financial instruments or rare chart patterns.
- Mobile photography challenges are specific to resolution and angle, which may vary across devices.
- The 'Not Applicable' task may not fully capture the complexity of hallucination in open-ended generation.
- Expert annotation is limited to a small panel, potentially introducing subjective bias in subjective tasks.
Curator notes
Discovery status: round4_promoted. Source links: Interpretable multimodal reasoning for robo-advisory: the FinErva framework; Multimodal Financial Foundation Models (MFFMs): Progress, Prospects, and Challenges; PyFi: Toward Pyramid-like Financial Image Understanding for VLMs via Adversarial Agents