FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answering

In one sentence

Summary

This paper introduces FAMMA, an open-source benchmark designed to evaluate large language models on complex financial reasoning tasks that require specialized knowledge, calculation, and multimodal input. Unlike existing benchmarks that focus on rudimentary text-based questions, FAMMA covers eight major finance subfields (e.g., derivatives, portfolio management) and includes questions in English, Chinese, and French, featuring charts, tables, and diagrams. The benchmark consists of FAMMA-Basic (1,945 questions from textbooks/exams) and FAMMA-LivePro (103 novel, contamination-free questions created by domain experts), with difficulty levels aligned to CFA standards. The research addresses the scarcity of high-quality, challenging financial reasoning benchmarks that test both advanced domain knowledge and sophisticated calculation capabilities. The authors constructed the dataset through a rigorous two-stage cleaning protocol and curated 1,273 reasoning trajectories from DeepSeek-R1 on FAMMA-Basic. They evaluated a suite of proprietary (GPT-o1, Claude, Gemini) and open-source (Qwen, DeepSeek-R1) models using zero-shot prompting, Program-of-Thoughts (PoT) for arithmetic, and Retrieval-Augmented Generation (RAG). Experiments included supervised fine-tuning of Qwen models on the distilled reasoning data and applying budget forcing at inference time. Evaluation metrics focused on Pass@1 accuracy across arithmetic vs. non-arithmetic questions, subfields, and languages, with GPT-4o used as a deterministic evaluator for open-ended responses. Results show that even frontier models like GPT-o1 struggle significantly, solving only 30% of hard LivePro questions. PoT prompting drastically improves arithmetic performance but fails to address non-arithmetic bottlenecks, where domain knowledge gaps account for 75% of errors. RAG provided no significant gains for strong reasoning models, suggesting they already encode sufficient textbook knowledge. Fine-tuning open-source models on distilled reasoning trajectories yielded systematic improvements, particularly for larger models (32B), and budget forcing offered complementary gains. The study highlights that closing the gap in financial reasoning requires deeper domain expertise rather than just numerical tool use or retrieval.

Research problem

The paper addresses the lack of high-quality benchmarks for evaluating LLMs on complex financial reasoning tasks. Existing benchmarks are often text-only, English-centric, and test only rudimentary knowledge. FAMMA aims to evaluate models on advanced, multimodal, multilingual financial problems that require both specialized domain knowledge and sophisticated calculation, reflecting real-world practitioner challenges.

Core contributions

• Introduction of FAMMA, a multilingual, multimodal benchmark covering 8 finance subfields with CFA-aligned difficulty.
• Creation of FAMMA-LivePro, a contamination-free set of 103 novel questions authored by domain experts.
• Development of a distillation pipeline to generate 1,273 high-quality reasoning trajectories from DeepSeek-R1.
• Comprehensive evaluation of 10+ state-of-the-art LLMs, revealing significant performance gaps in non-arithmetic financial reasoning.
• Demonstration that fine-tuning on distilled reasoning trajectories and budget forcing improves open-source model performance.

Method

• Dataset construction using STEM researchers and domain experts with two-stage quality control.
• Multimodal question creation involving charts, tables, and diagrams across 8 finance subfields.
• Distillation pipeline using DeepSeek-R1 to generate structured and free-form reasoning traces.
• Verification of reasoning traces using GPT-4o against ground-truth labels.
• Evaluation using zero-shot prompting, PoT for arithmetic, and RAG for knowledge retrieval.
• Supervised fine-tuning of Qwen2.5 models on distilled reasoning data using PyTorch FSDP.
• Budget forcing intervention to extend reasoning token length during inference.
• Error analysis categorizing model failures into context, knowledge, and ambiguity types.

Results & findings

• GPT-o1 solves only 30% of hard FAMMA-LivePro questions, showing significant difficulty with advanced reasoning.
• Program-of-Thoughts (PoT) prompting boosts arithmetic accuracy from ~25% to ~83% for GPT-o1 on LivePro.
• Non-arithmetic questions remain a bottleneck, with GPT-o1 achieving only 38.8% accuracy on hard items.
• Domain knowledge gaps account for 75% of errors in non-arithmetic questions, not context misinterpretation.
• RAG provides no significant performance gain for strong reasoning models like GPT-o1 and DeepSeek-R1.
• Fine-tuning Qwen models on distilled reasoning trajectories improves performance, especially for 32B models.
• Budget forcing offers complementary gains to fine-tuned models, scaling favorably with model size.
• Chinese remains the most challenging language for most models, while French performance is robust.

Data & experiments

• FAMMA-Basic: 1,945 questions from textbooks/exams, multilingual (English, Chinese, French), multimodal.
• FAMMA-LivePro: 103 novel questions by experts, held-out answers for contamination-free evaluation.
• OCR-converted text variants (FAMMA-Basic-Txt, LivePro-Txt) for text-only model evaluation.
• Evaluation of proprietary (GPT-o1, Claude, Gemini) and open-source (Qwen, DeepSeek) models.
• Experiments with Program-of-Thoughts (PoT), RAG, supervised fine-tuning, and budget forcing.
• Error analysis categorizing failures into context misinterpretation, domain knowledge gaps, and ambiguity.

Limitations & caveats

• FAMMA-LivePro contains only 103 questions, limiting statistical power for fine-grained analysis.
• OCR conversion may introduce errors, potentially affecting text-only model evaluations.
• The benchmark focuses on academic/professional exam-style questions, which may not reflect all real-world financial tasks.
• RAG experiments were limited to specific models and may not generalize to all retrieval strategies.
• Fine-tuning results are based on a small reasoning dataset (1,273 trajectories), which may not capture all reasoning patterns.
• The study does not evaluate long-term deployment or economic impact of using these models in practice.

Curator notes

Discovery status: round4_promoted. Source links: Interpretable multimodal reasoning for robo-advisory: the FinErva framework; Multimodal Financial Foundation Models (MFFMs): Progress, Prospects, and Challenges