FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning

In one sentence

Summary

The paper addresses the lack of realistic, open-domain benchmarks for evaluating LLM agents' financial search capabilities, noting that existing datasets either bypass search or lack the complexity of real analyst workflows. FinSearchComp is introduced as the first fully open-source agent benchmark for this purpose, designed to stress-test information gathering, coordination, and grounded reasoning in high-stakes financial contexts. It comprises three task families: Time-Sensitive Data Fetching (e.g., real-time prices), Simple Historical Lookup (e.g., point-in-time financials), and Complex Historical Investigation (e.g., multi-period synthesis), covering global and Greater China markets. The benchmark includes 635 questions curated by 70 professional financial experts from institutions like Citadel and J.P. Morgan, ensuring high difficulty and reliability through a rigorous multi-stage quality-assurance pipeline. Data sources include official filings, regulatory websites, and professional databases, with answers validated via cross-referencing and expert arbitration. The evaluation protocol uses an LLM-as-a-Judge approach with rubric-guided judging and tolerance bands for numerical answers, validated against human labels with 95% agreement. The dataset spans 10 distinct topics, including stocks, indices, currencies, and macroeconomics, with separate subsets for Global and Greater China markets to assess cross-lingual and cross-regional generalization. Experiments evaluate 21 models, revealing that web-enabled agents with financial plugins significantly outperform those without. Grok 4 (web) leads the global subset with 68.9% accuracy, approaching human expert levels (75.0%), while DouBao (web) leads the Greater China subset. However, all models remain substantially below human performance, with common failure modes including shallow search depth, retrieval of stale information, misalignment with reporting calendars, and errors in unit/currency normalization. The study highlights that while LLMs are making progress, they still struggle with freshness awareness, multi-source reconciliation, and temporal reasoning required for reliable decision support.

Research problem

The paper identifies a critical gap in evaluating LLM-based agents for financial tasks: existing benchmarks either do not require open-domain search, lack the complexity of real-world analyst workflows, or are not fully open-source. Financial search is a demanding proving ground because it requires handling time-sensitive data, reconciling conflicting evidence, and integrating domain-specific knowledge, yet no open dataset currently evaluates these end-to-end capabilities. The research problem is to create a realistic, expert-level benchmark that tests an agent's ability to fetch, verify, and synthesize financial information across diverse markets and task complexities.

Core contributions

• Introduction of FinSearchComp, the first fully open-source, end-to-end agent benchmark for open-domain financial search and reasoning.
• Creation of 635 expert-curated questions spanning three analyst-style task families: Time-Sensitive Data Fetching, Simple Historical Lookup, and Complex Historical Investigation.
• Development of a rigorous quality-assurance pipeline involving 70 professional financial experts and multi-stage verification to ensure high difficulty and reliability.
• Release of a curated benchmark dataset with deterministic gold answers and a fully open-source evaluation harness using LLM-as-a-Judge with rubric-guided judging.
• Comprehensive evaluation of 21 models, providing insights into the impact of web search, financial plugins, and model origin on performance.
• Identification of recurring failure modes in current LLM agents, such as shallow search, stale evidence, and temporal reasoning errors.

Method

• Task Design: Three task families mirroring analyst workflows: Time-Sensitive Data Fetching, Simple Historical Lookup, and Complex Historical Investigation.
• Data Collection: Questions derived from real-world analyst scenarios, using official filings, regulatory websites, and professional databases.
• Quality Control: Multi-stage verification by 70 financial experts, including blind review and senior arbitration for discrepancies.
• Evaluation Protocol: LLM-as-a-Judge with rubric-guided judging, tolerance bands for numerical answers, and expert spot checks.
• Model Evaluation: Testing 21 models with and without web search and financial plugins to assess the impact of tool use.
• Cross-Validation: Cross-referencing data from multiple sources to ensure accuracy and mitigate ambiguity in financial metrics.
• Error Analysis: Categorizing failures into shallow search, stale evidence, temporal misalignment, and calculation errors.

Results & findings

• Grok 4 (web) achieves the highest score on the Global subset (68.9%), outperforming GPT-5-Thinking (web) by 5.0 percentage points.
• DouBao (web) leads the Greater China subset, demonstrating the impact of model origin and tool availability on performance.
• Human experts achieve 75.0% accuracy on the Global subset and 88.3% on the Greater China subset, showing a significant gap between current LLMs and human experts.
• Equipping agents with web search and financial plugins substantially improves performance across all task types.
• Common failure modes include insufficient search depth, retrieval of outdated information, and errors in unit/currency normalization.
• LLM-as-a-Judge evaluation shows 95% agreement with human-verified labels, confirming the robustness of the evaluation protocol.
• Complex Historical Investigation tasks are the most challenging, requiring multi-period synthesis and provenance reconciliation.

Data & experiments

• Dataset: 635 questions across Global and Greater China subsets, covering 10 topics (stocks, indices, currencies, macroeconomics, etc.).
• Task Types: Time-Sensitive Data Fetching (134 global, 110 domestic), Simple Historical Lookup (119 global, 100 domestic), Complex Historical Investigation (84 global, 88 domestic).
• Data Sources: Official company filings, regulatory authority websites, professional financial databases, and public APIs.
• Expert Annotation: 70 financial experts (50 annotators, 20 senior reviewers) from top institutions like Citadel, J.P. Morgan, and Citigroup.
• Evaluation: LLM-as-a-Judge with rubric-guided judging and tolerance bands; validated against human labels with 95% agreement.
• Models Evaluated: 21 models (web-enabled products and API endpoints), including Grok 4, GPT-5-Thinking, and DouBao.

Limitations & caveats

• The benchmark focuses on search and reasoning, not on generating investment recommendations or trading strategies.
• Performance may vary based on the specific tools and plugins available to the models, which are not standardized across all evaluations.
• The LLM-as-a-Judge evaluation, while validated, may still introduce biases or errors compared to human judgment.
• The dataset covers only a subset of financial topics and markets, potentially limiting generalizability to other domains.
• The dynamic nature of time-sensitive data requires careful handling of evaluation timing and data latency.
• The complexity of some tasks may be too high for current LLMs, limiting the benchmark's utility for assessing incremental improvements.

Curator notes

Discovery status: round1_promoted_seed_for_round2. Source links: A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges; BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs; ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering; FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models; FinQA: A Dataset of Numerical Reasoning over Financial Data; FinRobot: An Open-Source AI Agent Platform for Financial Applica