FLaME: A Holistic Benchmark for Financial Language Models

In one sentence

Summary

This paper addresses the lack of rigorous, standardized evaluation for Large Language Models (LLMs) in specialized financial domains. Existing benchmarks are fragmented, lacking holistic criteria, which has led to an underestimation of LM capabilities in Finance NLP (FinNLP). The authors argue that without comprehensive evaluation, deploying LMs in financial systems risks severe errors due to hallucinations or flawed reasoning. FLaME (Financial Language Model Evaluation) is proposed as the first holistic benchmarking suite designed to provide standardized, multi-metric, and reproducible assessments of LM performance on core FinNLP tasks, explicitly acknowledging the incompleteness of current evaluation methods. The methodology involves a unified inference hub supporting proprietary and open-weight models, evaluated across 20 curated datasets covering six core NLP tasks: text classification, information retrieval, sentiment analysis, question answering, text summarization, and causal analysis. The study assesses 23 models, including foundation LMs (e.g., Llama 3, Qwen) and reasoning-reinforced models (e.g., DeepSeek R1, o1-mini). Evaluation metrics include accuracy, F1, and BERTScore, with a focus on deterministic decoding for consistency. The authors employ a scenario-based taxonomy categorizing data by task, domain, and language to map the landscape of available FinNLP resources and identify gaps in data coverage, particularly for non-English languages and complex causal tasks. Results indicate that no single model outperforms all others across every task; DeepSeek R1 leads in multi-step QA, while Claude 3.5 Sonnet excels in sentiment and retrieval. Numeric reasoning and causal detection remain significant bottlenecks, with low F1 scores indicating struggles with precise numeric mapping and cross-sentence references. Summarization tasks show higher tractability. The study highlights a strong cost-performance trade-off, where mid-scale open-weight models like DeepSeek-V3 and Llama 3.1 70B offer efficient alternatives to expensive proprietary models. Limitations include the exclusion of multi-modal models and a current focus on English, with calls for future work on domain-adaptive training and expanded multilingual coverage.

Research problem

The paper identifies a critical gap in evaluating Large Language Models (LLMs) for specialized, knowledge-intensive financial tasks. Existing benchmarks are fragmented, lack standardization, and fail to provide holistic assessments, leading to an erroneous belief in the lower bounds of LM performance in Finance NLP (FinNLP). This lack of rigorous evaluation poses risks for deploying LMs in financial systems, where errors can lead to significant economic and legal harm.

Core contributions

• Introduction of FLaME, the first holistic benchmarking suite for Financial Language Model Evaluation, providing standardized, multi-metric, and reproducible evaluation pipelines.
• Comprehensive empirical study of 23 foundation and reasoning-reinforced LMs across 20 core financial NLP datasets, covering six task categories.
• Development of a scenario-based taxonomy for FinNLP that categorizes datasets by task, domain, and language, highlighting gaps in current data availability.
• Open-source release of the FLaME framework, datasets, and results, including a public leaderboard to encourage continuous community updates and collaboration.
• Detailed error analysis and meta-analysis revealing specific weaknesses in numeric reasoning, causal detection, and the cost-performance trade-offs of various model families.

Method

• Unified inference hub supporting cloud APIs, cloud-hosted open-weight models, and local inference backends.
• Two-stage evaluation: generation by primary LM, followed by extraction and verification by a separate LM or rule-based parser.
• Scenario-based taxonomy mapping datasets by task, domain, and language attributes.
• Deterministic decoding strategies (temperature 0.0) to minimize randomness and ensure reproducibility.
• Multi-metric assessment including accuracy, F1, and BERTScore, aggregated by task and domain.
• Cost-efficiency analysis comparing inference costs across different model families and deployment modes.
• Quality assurance checks on datasets for label consistency, domain relevance, and data leakage.

Results & findings

• No single LM dominates all tasks; DeepSeek R1 leads in multi-step QA, while Claude 3.5 Sonnet excels in sentiment and retrieval.
• Numeric reasoning tasks (e.g., FNXL) show very low F1 scores (<0.06), indicating significant struggles with precise numeric labeling.
• Summarization tasks yield high BERTScores (0.75-0.82), suggesting they are more tractable for foundation LMs than complex reasoning.
• Larger parameter sizes do not guarantee better performance; mid-scale models like DeepSeek-V3 offer strong cost-performance balance.
• Reasoning-reinforced models (e.g., DeepSeek R1, o1-mini) show superior performance in complex QA but at higher inference costs.
• Causal detection and classification remain challenging due to data scarcity, requiring external knowledge or specialized modules.
• Open-weight models like Llama 3.1 70B and DeepSeek-V3 provide viable, cost-effective alternatives to expensive proprietary models.

Data & experiments

• Evaluated 23 LMs, including proprietary (GPT-4o, Claude 3.5, Gemini) and open-weight (Llama 3, DeepSeek, Qwen) models.
• Curated 20 datasets covering text classification, information retrieval, sentiment analysis, QA, summarization, and causal analysis.
• Used deterministic decoding (temperature 0.0) to ensure consistent and reproducible results for benchmarking.
• Assessed performance using task-specific metrics: accuracy, F1, precision, recall, and BERTScore for generation.
• Conducted cost analysis of inference, comparing expenses of proprietary APIs against open-weight local/cloud inference.
• Performed error analysis focusing on numeric reasoning gaps, language drift, and causal data scarcity.

Limitations & caveats

• Excludes multi-modal models, focusing solely on text-based NLP capabilities.
• Current taxonomy and datasets are predominantly in English, limiting applicability to global markets.
• Causal analysis and numeric reasoning tasks suffer from data scarcity, potentially skewing performance assessments.
• Some datasets may have inherent biases or limitations not fully captured by the benchmark metrics.
• The benchmark is a 'living' resource, meaning results may change as new models and datasets are added.
• Reliance on deterministic decoding may not reflect real-world usage where variability is often desired.

Curator notes

Evaluates many models over financial tasks.