Finance Agent Benchmark: Evaluating Language Model Agents as Financial Assistants

In one sentence

Summary

This paper addresses the critical need for robust, domain-specific benchmarks to evaluate Large Language Model (LLM) agents in real-world financial analysis. It argues that previous benchmarks fail to capture the interactive, multi-step reasoning and tool-use requirements of industry tasks, leaving performance in high-stakes financial settings uncertain. The authors introduce the Finance Agent Benchmark, constructed in consultation with experts from banks, hedge funds, and private equity firms, to provide a standardized testbed for measuring the progress of LLM-driven finance agents. The benchmark comprises 537 expert-authored questions across nine task categories, ranging from simple quantitative retrieval to complex financial modeling and market analysis. Questions are grounded in recent SEC filings (post-2024) to prevent data contamination. The authors developed an agentic evaluation harness equipped with tools including Google Search, EDGAR database access, HTML parsing, and information retrieval. Evaluation employs a refined LLM-as-judge methodology with rubric-based assessment and contradiction detection to ensure accurate grading of multi-step reasoning trajectories. Experiments on 20 models reveal that current AI capabilities are significantly limited for autonomous financial tasks. The best-performing model, OpenAI o3, achieved only 46.8% class-balanced accuracy, with no model surpassing 50%. A clear logarithmic relationship exists between cost and accuracy, with diminishing returns beyond $1 per query. While models are significantly faster and cheaper than human experts (avg $3.79 vs $25.66), the low accuracy underscores the need for further advancements before reliable deployment in high-stakes finance. The study also analyzes tool usage patterns, finding that more exploratory models tend to perform better, while high tool usage without precision leads to failure.

Research problem

The paper addresses the lack of robust, domain-specific benchmarks for evaluating LLM agents in real-world financial analysis. It highlights that existing benchmarks often fail to capture the interactive environments, nuanced reasoning, and tool-use capabilities required for industry-specific tasks, leading to uncertainty about AI reliability in high-stakes financial decisions.

Core contributions

• Introduction of the Finance Agent Benchmark, a comprehensive dataset of 537 expert-authored financial research questions grounded in recent SEC filings.
• Development of a nine-category taxonomy of financial tasks, ranging from simple retrieval to complex financial modeling, validated by industry experts.
• Creation of an agentic evaluation harness equipped with tools for EDGAR search, web search, and document parsing to simulate real-world analyst workflows.
• Implementation of a refined LLM-as-judge evaluation framework with rubric-based assessment and contradiction detection to ensure accurate grading of multi-step reasoning.
• Comprehensive empirical evaluation of 20 state-of-the-art LLMs, establishing performance baselines and analyzing cost-accuracy trade-offs and tool usage patterns.

Method

• Dataset Creation: Collaboration with industry experts to identify common financial tasks and generate 537 questions with ground-truth answers and reasoning steps.
• Taxonomy Development: Categorization of tasks into nine levels of complexity and frequency in investment banking contexts.
• Validation: Multi-stage peer review by experts to verify calculations, content validity, and accuracy of questions and answers.
• Agentic Harness: Implementation of a ReAct-based framework allowing LLMs to interleave reasoning and action using specialized tools.
• Tool Integration: Integration of Google Search, EDGAR Search, HTML Parsing, and Information Retrieval tools to enable autonomous research.
• Evaluation: Rubric-based LLM-as-judge scoring, breaking down expert answers into key points and checking for presence/absence and contradictions.
• Error Handling: Automatic retry for rate-limit errors and classification of agent errors (e.g., token limits, incorrect arguments) as model failures.

Results & findings

• Best model (OpenAI o3) achieved only 46.8% class-balanced accuracy, with no model surpassing 50%.
• Average cost per query for top models was $3.79, significantly lower than human expert cost of $25.66.
• Average time per query for top models was ~3.1 minutes, compared to 16.8 minutes for human experts.
• Clear logarithmic relationship observed between cost and accuracy, with diminishing returns beyond $1 per query.
• Models handling simple retrieval performed better than those tackling complex financial modeling or market analysis.
• More exploratory models (e.g., Claude 3.7 Sonnet) tended to perform better by making more tool calls.
• High tool usage without precision (e.g., GPT-4o Mini) correlated with high error rates and poor performance.
• LLaMA 4 Maverick showed disproportionate use of retrieval operations and hallucinated document existence.

Data & experiments

• Dataset: 537 expert-authored questions covering nine financial task categories, validated by experts from banks, hedge funds, and private equity.
• Data Source: Questions are grounded in public SEC filings (EDGAR) published no earlier than 2024 to ensure recency and prevent training data contamination.
• Evaluation Harness: Agentic framework with tools for Google Search, EDGAR Search, HTML Parsing, and Information Retrieval.
• Models Evaluated: 20 LLMs including OpenAI o3, Claude 3.7 Sonnet, GPT-4.1, Llama 4, and others.
• Metrics: Class-balanced accuracy, naive accuracy, time per query, cost per query, and tool usage statistics.
• Evaluation Method: LLM-as-judge with rubric-based scoring and contradiction detection to assess multi-step reasoning and factual accuracy.

Limitations & caveats

• Dataset emphasizes short passages or final outputs, with limited coverage of structured tabular data reasoning.
• Citation accuracy and source reliability were not evaluated, only the correctness of the final answer.
• Agentic harness is relatively simple (ReAct-based) and may not represent the complexity of commercial RAG systems.
• Models only had access to public SEC filings, lacking integration with private knowledge bases or proprietary databases.
• Evaluation focuses on US-centric SEC filings, potentially limiting generalizability to other markets.
• LLM-as-judge evaluation, while refined, may still have biases or errors compared to human grading.

Curator notes

Requires recent SEC filing use and expert-authored tasks.