Multimodal Financial Foundation Models (MFFMs): Progress, Prospects, and Challenges

In one sentence

Summary

The paper addresses the evolution from unimodal Financial LLMs to Multimodal Financial Foundation Models (MFFMs) capable of processing interleaved data types including text, audio, video, charts, and tabular financial records. It positions MFFMs as essential for handling the complexity of real-world financial data, such as earnings calls and regulatory filings, which contain rich multimodal signals beyond simple text. The authors argue that current language-centric models are insufficient for tasks requiring deep integration of diverse data sources, such as interpreting visual charts in reports or analyzing audio sentiment in conference calls. The authors provide a comprehensive taxonomy of multimodal financial data, covering Earnings Conference Calls (ECCs), Monetary Policy Conferences (MPCs), SEC filings (10-K, 10-Q), financial news, market time-series, and climate data. They discuss specific datasets like MDRM and MONOPOLY, highlighting challenges in data curation, alignment, and storage. The paper also introduces an agentic FinAI ecosystem, detailing tool agents (search, tutor, XBRL) and service agents (robo-advisors, compliance auditors) powered by frameworks like FinGPT and protocols like MCP and A2A. Experiments and case studies include the development of a 'Warren Buffett' agent via FinLoRA and the use of FinRL for trading strategies. Key findings emphasize the potential of MFFMs to democratize financial services and enhance decision-making through agentic workflows. However, significant challenges remain, including 'model cannibalism,' 'openwashing,' hallucination, and the high cost of training and inference. The paper calls for robust guardrail frameworks, better benchmarks like CFA exam evaluations, and adherence to openness standards. It concludes that while MFFMs offer promising prospects for alpha mining, risk modeling, and automated reporting, achieving 'FinAI readiness' requires solving critical issues in transparency, privacy, and regulatory compliance.

Research problem

The paper identifies the limitation of existing Financial LLMs (FinLLMs) that are primarily text-centric and unable to fully leverage the multimodal nature of financial data. It addresses the need for Multimodal Financial Foundation Models (MFFMs) that can process and integrate diverse data types such as audio, video, charts, and tabular data to improve financial analysis, decision-making, and service automation. The problem also encompasses the lack of standardized benchmarks, transparency in model development, and robust governance frameworks for these advanced models.

Core contributions

• First comprehensive survey of Multimodal Financial Foundation Models (MFFMs), covering data, applications, and models.
• Introduction of a data-centric approach to multimodal financial data, detailing types like ECCs, MPCs, and climate data.
• Proposal of an agentic FinAI ecosystem using FinAgents powered by open models and protocols like MCP and A2A.
• Comparison of MFFMs with LLMs, FinLLMs, and MLLMs to clarify development lifecycles and capabilities.
• Identification of key challenges including proprietary data constraints, training costs, and regulatory complexities.
• Discussion of ethical challenges and the need for guardrail frameworks to address hallucination and misinformation.

Method

• Literature review and synthesis of existing MFFM research, datasets, and applications.
• Categorization of multimodal financial data into text, audio, video, chart, tabular, and time-series.
• Analysis of agentic frameworks and protocols for building FinAgents.
• Evaluation of model openness and transparency issues in current LLM releases.
• Assessment of MFFM capabilities using benchmarks like CFA exams and financial QA datasets.
• Discussion of fine-tuning techniques and cost-efficiency strategies for financial models.

Results & findings

• MFFMs can process interleaved multimodal data, enabling deeper understanding of financial complexity.
• Existing FinLLMs like FinGPT and BloombergGPT show potential but are limited to language-centric approaches.
• Earnings Conference Calls (ECCs) and Monetary Policy Conferences (MPCs) contain critical multimodal signals.
• Agentic workflows using MCP and A2A protocols enable scalable financial applications like robo-advisors.
• Challenges include 'model cannibalism' and 'openwashing' reducing transparency and reproducibility.
• Hallucination and misinformation are significant risks requiring robust guardrail frameworks.
• Training and inference costs for MFFMs are high, necessitating efficient fine-tuning methods like LoRA.
• MFFMs have potential for alpha mining, risk modeling, and automated compliance auditing.

Data & experiments

• Survey of multimodal financial data types including text, audio, video, charts, tabular, and time-series data.
• Analysis of specific datasets: MDRM for earnings calls, MONOPOLY for monetary policy conferences, and SEC filings.
• Evaluation of MFFM performance on CFA exam questions to assess reasoning and financial knowledge.
• Case study of a 'Warren Buffett' agent fine-tuned using FinLoRA and QLoRA on multimodal data.
• Discussion of FinRL trading agents and hybrid LLM-RL approaches for automated stock trading.
• Review of benchmarks like FinBen, PIXIU, and FinFact for evaluating financial LLMs.

Limitations & caveats

• Paper is a position paper and survey, lacking original empirical results or new model training.
• Challenges in data curation and alignment for multimodal financial data are noted as significant barriers.
• High training and inference costs may limit accessibility for smaller institutions.
• Hallucination and misinformation risks require further development of guardrail frameworks.
• Transparency issues due to 'openwashing' and 'model cannibalism' hinder reproducibility.
• Regulatory complexities and proprietary data constraints limit widespread adoption.

Curator notes

Discovery status: round3_promoted_seed_for_round4. Source links: Deficiency of Large Language Models in Finance: An Empirical Examination of Hallucination; Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learning; FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis; Large Language Models for Financial and Investment Management: Applications and Benchmarks