The List / Categories
Financial LLM benchmarks evaluate different capabilities: text classification, numerical reasoning, document retrieval and sequential decisions. Scores from these tasks are not interchangeable.
For trading-oriented evaluations, also inspect transaction costs, execution assumptions and market regimes in the trading guide.
FinBen is a comprehensive open-source benchmark evaluating 15 LLMs across 24 financial tasks, revealing that while models excel at information extraction, they struggle with complex reasoning, forecasting, and trading, with GPT-4 leading in IE and trading while Gemini leads in generation.
This empirical study benchmarks ChatGPT and GPT-4 against domain-specific models and fine-tuned baselines across eight financial NLP datasets, revealing that generalist LLMs excel in sentiment analysis and numerical reasoning but lag in structured prediction tasks like named entity recognition and relation extraction.
PIXIU introduces FinMA, an open-source financial LLM fine-tuned on the FIT dataset, and FLARE, a benchmark covering financial NLP and stock prediction tasks, demonstrating that domain-specific instruction tuning significantly improves performance on most financial NLP tasks compared to general and proprietary models.
RiskLabs is a multimodal framework that leverages LLMs to extract features from earnings calls, news, and time series data to significantly improve financial risk prediction compared to traditional baselines.
The paper introduces CALM, a fine-tuned LLM for credit scoring, demonstrating that LLMs can generalize across diverse financial risk tasks but exhibit significant demographic biases requiring ethical oversight.
BizFinBench introduces a business-driven Chinese financial benchmark with 6,781 real-world queries and the IteraJudge evaluation framework, revealing that while proprietary models lead in reasoning and extraction, no single model dominates all financial tasks, highlighting significant gaps in complex cross-concept reasoning.
FinanceReasoning is a rigorous benchmark for evaluating large reasoning models in financial numerical tasks, demonstrating that combining reasoner and programmer models with structured knowledge retrieval significantly improves accuracy on complex financial calculations.
The paper proposes MANAGER, a multimodal LLM framework that integrates external financial knowledge and adaptive graph-based fusion of text, video, and audio from Monetary Policy Conference calls to outperform baselines in predicting asset price movements and volatility.
FinDABench is a comprehensive benchmark evaluating LLMs on financial data analysis across foundational, reasoning, and technical skills, revealing that even GPT-4 achieves only 32.37% accuracy in zero-shot settings, highlighting significant gaps in domain-specific analytical capabilities.
The paper demonstrates that using Sentence-BERT embeddings to encode non-semantic categorical features in general ledger journal entries significantly improves anomaly detection performance compared to traditional one-hot encoding, particularly when paired with Logistic Regression and Neural Networks.
The paper introduces FLAME, a comprehensive Chinese financial LLM evaluation system comprising certification and scenario benchmarks, demonstrating that Baichuan4-Finance outperforms other models in professional knowledge and practical application tasks.
This survey systematically categorizes financial foundation models into language, time-series, and visual-language types, reviewing their architectures, training strategies, datasets, and applications while identifying key challenges in data, algorithms, and infrastructure.
This position paper identifies five critical biases in financial LLM evaluation—look-ahead, survivorship, narrative, objective, and cost—and proposes a Structural Validity Framework with a checklist to enforce rigorous, deployment-ready standards.
This study evaluates the open-source financial LLM FinMA-7B-full on the FLARE benchmark, demonstrating strong performance in sentiment analysis and classification but significant weaknesses in numerical reasoning, entity recognition, and summarization compared to general-purpose baselines.
RealFin introduces a bilingual benchmark evaluating LLMs' ability to detect under-specified financial questions, revealing that general models over-commit to guesses while finance-specialized models fail to identify missing premises.
This paper demonstrates that LLMs can effectively detect money laundering in financial graphs via in-context learning on serialized subgraphs, achieving 63.7% accuracy while providing interpretable rationales, though scalability remains a challenge.
The paper demonstrates that data-centric post-training strategies, specifically high-quality Chain-of-Thought distillation for SFT and difficulty-aware sampling for RL, enable an 8B-parameter financial LLM to outperform larger and specialized open-source competitors across general, sentiment, and numerical reasoning benchmarks.