Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Risk

In one sentence

Summary

The paper addresses the critical gap in evaluating Large Language Model (LLM) agents in finance, arguing that traditional benchmarks focusing on accuracy, F1 scores, or return-based metrics provide an illusion of reliability. It posits that these metrics fail to capture essential safety risks such as hallucinated facts, temporal staleness, and adversarial prompt manipulation, which can lead to significant financial losses in high-stakes environments. The authors contend that the primary evaluation criterion should shift from "how well" an agent performs to "how safely" it can fail, emphasizing the need for risk-aware auditing in safety-critical financial applications. To operationalize this shift, the authors introduce the Safety-Aware Evaluation Agent (SAEA), a modular auditing recipe grounded in risk-engineering principles and Basel’s operational-risk taxonomy. SAEA conducts a three-level audit: model-level (intrinsic vulnerabilities like hallucination and over-confidence), workflow-level (error propagation and prompt sensitivity in multi-step chains), and system-level (resilience to API failures and external shocks). The authors evaluate six state-of-the-art LLM agents (including GPT-4o, Claude-3.5-Sonnet, and various open-weight models) across three high-impact tasks: finance management (cryptocurrency), webshop automation, and transactional services. They categorize agent trajectories as "safe" or "unsafe" and measure risk severity across nine dimensions, including hallucination, temporal accuracy, and adversarial robustness. The results demonstrate that SAEA effectively distinguishes between safe and unsafe trajectories, uncovering hidden weaknesses that conventional benchmarks miss. For instance, agents often exhibit high confidence in erroneous actions or fail to detect adversarial inputs despite high accuracy on standard tasks. The paper concludes with actionable recommendations for researchers and regulators, urging the adoption of risk-aware metrics, the publication of stress scenarios, and the treatment of a "safety budget" as a primary success criterion. It highlights that while engineering guardrails are useful, they are insufficient without rigorous, scenario-based stress testing to ensure long-term reliability and compliance in financial systems.

Research problem

Standard benchmarks for financial LLM agents focus on accuracy and performance metrics, ignoring safety and risk. This creates a dangerous gap where agents may perform well on static tests but fail catastrophically in real-world deployment due to hallucinations, adversarial attacks, or temporal staleness. The paper seeks to redefine evaluation criteria to prioritize risk mitigation over point-estimate performance.

Core contributions

• Proposes a paradigm shift from performance-centric to risk-centric evaluation for financial LLM agents.
• Introduces a three-level risk taxonomy (model, workflow, system) grounded in finance-risk theory and safety engineering.
• Develops SAEA, a modular, open auditing recipe for stress-testing LLM agents under realistic failure modes.
• Provides empirical evidence that six state-of-the-art LLM agents exhibit hidden safety risks despite high benchmark scores.
• Offers actionable recommendations for researchers, practitioners, and regulators to adopt risk-aware metrics and stress testing.

Method

• SAEA shadows the agent's I/O stream to capture trajectories for analysis.
• Adaptive metric selection based on task type and trajectory characteristics.
• Evaluation agents assess specific risk dimensions (e.g., hallucination, temporal accuracy) using predefined prompts.
• Metric aggregation combines individual risk scores into a composite risk profile.
• Trajectories are labeled 'safe' or 'unsafe' based on ground truth to validate SAEA's discriminative power.
• Risk severity is quantified on a 0-100 scale, with lower scores indicating better safety.
• Post-hoc analysis traces root causes of failures through structured audit logs.
• Comparison of risk profiles across different model sizes and types (API vs. open-weight).

Results & findings

• SAEA assigns significantly lower risk scores to 'safe' trajectories compared to 'unsafe' ones, validating its ability to detect risk.
• High-performing agents on standard benchmarks still exhibit severe vulnerabilities, such as hallucination and adversarial susceptibility.
• Smaller models (e.g., Llama-3.1-8b) showed less stable evaluations, assigning higher risk scores to safe trajectories.
• Open-weight models exhibited similar risk profiles to API-based models, suggesting consistency in SAEA's findings.
• Agents often displayed illusory confidence, proceeding with high-risk actions without acknowledging uncertainties.
• Adversarial prompts could trigger significant error propagation and hallucination, even in top-tier models.
• The paper highlights that accuracy does not equate to safety, as agents can arrive at correct answers for wrong reasons.

Data & experiments

• Evaluates six LLM agents: GPT-4o, Claude-3.5-Sonnet, Qwen3-235b-a22b, DeepSeek-R1, Llama-3.3-70b, and Llama-3.1-8b.
• Tests agents on three high-impact financial tasks: finance management (crypto), webshop automation, and transactional services.
• Uses trajectories from the R-Judge dataset, categorized as 'safe' or 'unsafe' based on ground truth labels.
• Measures risk severity across nine dimensions: hallucination, temporal accuracy, confidence, adversarial robustness, explanation clarity, error propagation, prompt sensitivity, response degradation, and stress testing.
• Standardizes scores to a 0-100 scale, where lower scores indicate lower risk presence or greater robustness.

Limitations & caveats

• SAEA is an auditing recipe, not a comprehensive benchmark dataset, limiting direct comparison with other benchmarks.
• The evaluation relies on a limited set of tasks and models, potentially missing broader failure modes.
• Risk scoring is subjective and depends on the evaluation agent's interpretation, which may vary.
• The paper acknowledges that engineering guardrails are necessary but argues they are insufficient without stress testing.
• Real-world deployment risks may include factors not captured in the simulated environments (e.g., extreme market events).
• The study does not provide a solution for mitigating the identified risks, only for detecting them.

Curator notes

Discovery status: round1_promoted_seed_for_round2. Source links: A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist; A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges; A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law; AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework; BloombergGPT: A Large Language Model for Finance; Deficiency of Large Language Models in Finance: An Empirical Examinat