Open FinLLM Leaderboard: Towards Financial AI Readiness

In one sentence

Summary

The paper addresses the critical need for robust, dynamic evaluation frameworks for Financial Large Language Models (FinLLMs) and FinAgents, noting that existing benchmarks like FinBen and FinanceBench are static and insufficient for tracking rapid model advancements. The authors propose an open FinLLM leaderboard, a collaborative platform designed to democratize access to financial AI readiness by providing a standardized, transparent, and continuously updated ecosystem for assessing model performance. This initiative aims to bridge the gap between academic research and industry deployment by fostering a community-driven approach to benchmarking, ensuring that models are evaluated on their real-world utility and reliability in complex financial contexts. The methodology involves a comprehensive testing pipeline that evaluates popular models, including GPT-4, LLaMA 3.1, Gemini, and Qwen2, using zero-shot settings on expert-validated datasets. The leaderboard covers seven task categories: Information Extraction, Textual Analysis, Question Answering, Text Generation, Risk Management, Forecasting, and Decision-Making, utilizing 42 financial datasets that include multimodal data such as text, tables, and XBRL filings. The evaluation process employs specific metrics like Accuracy, F1, ROUGE, and BERTScore, with scores normalized to a 0-100 scale for fair comparison. Additionally, the platform integrates Zero-Knowledge Proofs (ZKP) to ensure privacy-preserving verification and prevent leaderboard gaming, while offering interactive demos like the FinGPT Search Agent for side-by-side model comparisons. Key findings highlight the leaderboard's role in identifying model strengths and weaknesses across diverse financial tasks, revealing issues such as hallucinations in financial reasoning and difficulties with complex XBRL parsing. The paper demonstrates that while models perform well on standard text tasks, they struggle with precise numerical reasoning and context identification in structured financial documents. Use cases include enhancing legal consultations, simplifying financial document analysis for the general public, and supporting regulatory compliance. Limitations include the current static nature of some underlying datasets and the challenge of evaluating multimodal capabilities fully. The work serves as a foundational resource for surveying the state of FinLLM evaluation, emphasizing the shift from static benchmarks to dynamic, community-driven assessment ecosystems.

Research problem

Existing benchmarks for Financial Large Language Models (FinLLMs) are static and lack the momentum to continuously adapt to emerging models and tasks, limiting their utility for real-world financial applications. There is a need for a robust, open, and continuously updated evaluation framework that can assess both FinLLMs and FinAgents across a wide spectrum of financial tasks, including multimodal data, to ensure financial AI readiness and reliability.

Core contributions

• Creation of an open FinLLM leaderboard in collaboration with the Linux Foundation and Hugging Face to provide a transparent and standardized evaluation platform.
• Development of a comprehensive testing pipeline that evaluates zero-shot performance of major models (GPT-4, LLaMA 3.1, Gemini, Qwen2) across 42 financial datasets.
• Introduction of a hierarchical leaderboard structure covering seven key financial task categories: Information Extraction, Textual Analysis, Question Answering, Text Generation, Risk Management, Forecasting, and Decision-Making.
• Integration of Zero-Knowledge Proofs (ZKP) to ensure privacy-preserving verification and prevent leaderboard manipulation while maintaining evaluation integrity.
• Provision of interactive demos, including the FinGPT Search Agent, allowing users to perform side-by-side comparisons of model outputs and verify sources in real-time.
• Establishment of a community-driven ecosystem that encourages contributions of new datasets, tasks, and models to keep the leaderboard dynamic and responsive to industry needs.

Method

• Zero-shot evaluation of models on expert-validated financial datasets.
• Normalization of diverse evaluation metrics (Accuracy, F1, ROUGE, etc.) to a 0-100 scale.
• Integration of Retrieval-Augmented Generation (RAG) in the FinGPT Search Agent demo.
• Application of Zero-Knowledge Proofs (ZKP) for privacy-preserving verification of model performance.
• Side-by-side comparison interface for qualitative assessment of model outputs.
• Hierarchical leaderboard structure mapping tasks to business applications (Search Agent, AI Tutor, Compliance, Auditing).
• Min-max scaling for score normalization to ensure fair comparison across different task types.

Results & findings

• The leaderboard provides a standardized framework for ranking models based on multimodal performance in financial tasks.
• Integration with FinGPT Search Agent allows for interactive, source-verified model comparisons beyond numeric scores.
• Zero-Knowledge Proofs are implemented to protect sensitive dataset privacy and prevent leaderboard hacking.
• Models show varying strengths across task categories, with challenges in precise numerical reasoning and XBRL context identification.
• The platform supports continuous updates, encouraging community contributions to maintain relevance.
• Case studies reveal specific failure modes in LLMs, such as incorrect contextRef usage in XBRL files and calculation errors in financial ratios.
• The leaderboard aims to democratize access to financial intelligence, enhancing analytical capabilities for the general public.

Data & experiments

• Evaluation of 5 major models: GPT-4, LLaMA 3.1 (8B/70B), Gemini, Qwen2 (72B/7B), and Xuanyuan-70B.
• Use of 42 expert-validated financial datasets spanning text, tables, and XBRL formats.
• Zero-shot evaluation setting to test generalization ability on unseen financial tasks.
• Metrics include Accuracy, F1 Score, ROUGE, BERTScore, and Matthews Correlation Coefficient (MCC).
• Scores normalized to a 0-100 scale using min-max scaling for fair cross-task comparison.
• Case studies on XBRL parsing failures and financial ratio calculation errors using GPT-4o.

Limitations & caveats

• Some underlying datasets may still be static, limiting the dynamic nature of the evaluation.
• Multimodal capabilities are not fully evaluated, with a focus primarily on text and structured data.
• The leaderboard's effectiveness depends on continuous community contributions, which may vary.
• Zero-shot evaluation may not fully capture the performance of fine-tuned models in specific financial domains.
• Hallucinations and numerical reasoning errors remain significant challenges for current FinLLMs.
• Privacy-preserving verification via ZKP adds complexity to the evaluation process.

Curator notes

Discovery status: round4_promoted. Source links: Interpretable LLMs for Credit Risk: A Systematic Review and Taxonomy; Multimodal Financial Foundation Models (MFFMs): Progress, Prospects, and Challenges