RealFin: How Well Do LLMs Reason About Finance When Users Leave Things Unsaid?

In one sentence

Summary

The paper addresses the critical gap in financial LLM evaluation by introducing RealFin, a benchmark designed to test whether models can recognize when a question lacks sufficient information for a justified answer. Unlike traditional benchmarks that assume a closed-world setting with complete premises, RealFin systematically removes essential assumptions from professional exam-style questions in English (CFA) and Chinese (CPA), creating underdetermined problems that appear linguistically plausible but are logically unsolvable without clarification. This approach targets the real-world scenario where users leave implicit assumptions unstated, requiring models to exhibit epistemic caution rather than blind computation. The dataset comprises 2,020 paired questions, including full-condition and condition-missing variants, covering six reasoning types and multiple financial sub-domains. The authors evaluate 15 LLMs, including general-purpose, finance-specialized, and reasoning-enhanced models, using three formulations: standard answering, recognizing missing information, and a None-of-the-Above (NOTA) setting to force genuine inference. Experiments are conducted in a zero-shot setting with greedy decoding, measuring accuracy and confidence scores to assess both answer correctness and the model's ability to identify logical underdetermination. Results show that general-purpose models often improve in accuracy on missing-condition questions by implicitly filling in gaps with learned defaults, leading to high confidence but low reasoning accuracy. In contrast, finance-specialized models frequently fail to answer even full-condition questions in Chinese due to rigid rule-matching, while reasoning-enhanced models show mixed results. The NOTA evaluation reveals a significant gap between answer accuracy and reasoning accuracy, indicating that many correct answers are driven by pattern recognition rather than true logical deduction. The study highlights a structural failure in current LLMs to withhold commitment when information is insufficient, a key requirement for reliable financial reasoning.

Research problem

Current financial LLM benchmarks operate under a closed-world assumption, evaluating only whether models can answer well-posed questions. This paper addresses the lack of evaluation for epistemic caution: the ability to recognize when a financial problem is under-specified due to missing premises, such as implicit assumptions, regulatory standards, or valuation models. It investigates whether LLMs can distinguish between solvable problems and those requiring clarification, a critical capability for high-stakes financial decision-making.

Core contributions

• Introduction of RealFin, a bilingual benchmark evaluating financial reasoning under missing conditions.
• Creation of a 2,020-question dataset with paired full-condition and condition-missing variants from CFA and CPA materials.
• Development of three evaluation formulations: standard answering, missing information recognition, and NOTA-based inference.
• Comprehensive evaluation of 15 LLMs across general, finance-specialized, and reasoning-enhanced categories.
• Quantitative analysis of the gap between answer accuracy and reasoning accuracy, revealing systematic over-commitment in general models.

Method

• Dataset construction: Manual removal of essential premises from CFA and CPA exam questions.
• Condition types: Removed macro assumptions, valuation models, financial covenants, and regulatory standards.
• Evaluation formulations: Standard QA, missing information recognition, and NOTA-based inference.
• Model evaluation: Zero-shot inference with greedy decoding and fixed chat templates.
• Metrics: Accuracy on answer selection, reasoning accuracy (identifying missing premises), and confidence calibration.
• Cross-lingual analysis: Comparing performance on English CFA and Chinese CPA to identify language-specific biases.
• Adversarial masking: Using NOTA to test if models rely on pattern recognition or genuine logical deduction.

Results & findings

• General models (e.g., GPT-5.1-mini) show improved accuracy on missing-condition questions by guessing defaults, not recognizing missing info.
• Finance-specialized models (e.g., FinGPT-7B) perform poorly, often failing to answer even full-condition questions in Chinese.
• Reasoning-enhanced models (e.g., Fin-R1-7B) show better reasoning accuracy but still struggle with Chinese CPA under-specification.
• NOTA evaluation reveals a large gap between answer accuracy and reasoning accuracy, especially in general models.
• GPT-OSS-120B shows the highest confidence (95.66%) but lowest reasoning accuracy (20.71%) on English NOTA questions.
• Chinese CPA performance is significantly lower than English CFA across all model types, indicating language-specific challenges.
• Complex calculations and comprehensive judgment are the most difficult question types for all models.
• Parameter scaling (20B to 120B) does not resolve the structural failure to identify missing premises.

Data & experiments

• Dataset: 2,020 bilingual questions (English CFA, Chinese CPA) with full-condition and condition-missing variants.
• Question types: Financial concepts, simple/complex calculations, summary/judgment, knowledge transfer, and statistical methods.
• Models evaluated: 15 LLMs including GPT-5.1-mini, Claude-Sonnet-3.5, Fin-R1-7B, and DianJin-R1-32B.
• Experimental setup: Zero-shot evaluation with greedy decoding (temperature=0) and fixed max tokens.
• Metrics: Answer accuracy, reasoning accuracy (identifying missing premises), and confidence scores.
• NOTA setting: Replacing correct options with 'None of the Above' to test genuine inference vs. guessing.

Limitations & caveats

• Dataset is limited to CFA and CPA exam questions, which may not cover all real-world financial scenarios.
• Evaluation is zero-shot, which may not reflect performance with fine-tuning or in-context learning.
• Chinese CPA performance is significantly lower, potentially due to language-specific challenges or data scarcity.
• The NOTA setting may not fully capture the complexity of real-world abstention decisions.
• Model evaluation is limited to 15 LLMs, which may not represent the full landscape of available models.
• The study focuses on logical underdetermination, not other forms of financial reasoning errors like numerical miscalculation.

Curator notes

Discovery status: round2_promoted. Source links: A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist; CFGPT: Chinese Financial Assistant with Large Language Model; DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning; FinBERT: Financial Sentiment Analysis with Pre-trained Language Models; XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters