AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language Models

In one sentence

Summary

The paper addresses the critical failure mode of behavioral instability in LLM-based trading agents, where direct action emission leads to extreme run-to-run variance, inconsistent decision sequences even under deterministic decoding, and irrational action flipping due to stateless architectures and sensitivity to continuous-to-discrete mappings. To resolve this, the authors propose AlphaForgeBench, a framework that repositions LLMs as quantitative researchers generating executable alpha factors and strategy code, thereby decoupling reasoning from execution to ensure deterministic and reproducible evaluation aligned with real-world quantitative workflows. The benchmark utilizes a two-stage dataset construction process: Stage 1 curates 633 real-world single-asset queries from brokerage reports and open-source repositories, while Stage 2 generates 270 structured queries across a 3x3 difficulty taxonomy (Logic Translation, Logic Completion, Goal-Oriented Generation) to enable fine-grained diagnostic evaluation. Experiments evaluate six frontier LLMs (including Gemini, Claude, GPT, DeepSeek, and Grok) by generating Python strategy functions and executing them in a unified backtest engine across seven assets (BTC, ETH, AAPL, GOOGL, MSFT, NVDA, TSLA) over a five-year period, measuring performance via Sharpe Ratio, Annual Return, Maximum Drawdown, and other risk-adjusted metrics. Results demonstrate that the code-generation paradigm eliminates execution-induced instability, with intra-query variance an order of magnitude smaller than inter-query variance, and temperature-invariant model rankings. The study reveals distinct "risk personalities" among models (e.g., Gemini favors aggressive returns, DeepSeek favors conservative risk control) and identifies a clear performance hierarchy where Gemini-3-Pro leads in return-oriented metrics. The benchmark effectively exposes complementary cognitive capabilities, showing that code-translation skill and strategic reasoning are dissociable, with Level 3 tasks providing the highest discriminative power for assessing true financial reasoning and alpha discovery.

Research problem

Existing financial LLM benchmarks that evaluate direct trading actions suffer from severe behavioral instability, including extreme run-to-run variance, inconsistent action sequences under deterministic decoding, and irrational action flipping. These issues stem from the stateless autoregressive nature of LLMs and their sensitivity to continuous-to-discrete action mappings, rendering current evaluations unreliable and non-reproducible for assessing true financial reasoning capabilities.

Core contributions

• Systematic analysis of the critical flaws in existing financial trading benchmarks, demonstrating that LLM instability renders traditional performance metrics unreliable.
• Introduction of AlphaForgeBench, a novel benchmark that repositions LLMs from stochastic execution agents to quantitative researchers generating executable alpha factors and strategy code.
• Development of a two-stage dataset construction pipeline combining real-world ecological validity with a 3x3 difficulty-stratified synthetic query set for controlled diagnostic evaluation.
• Extensive empirical validation across six frontier LLMs, confirming that the code-generation paradigm eliminates execution-induced instability and provides a rigorous, reproducible metric for assessing financial logic synthesis.

Method

• Reconceptualizes LLMs as quantitative researchers generating executable alpha factors and strategy code rather than direct trading actions.
• Two-stage dataset construction: real-world curation followed by LLM-augmented structured query generation with a 3x3 difficulty taxonomy.
• Standardized prompt construction including system instructions, strategy queries, and factor-library references.
• Code generation pipeline where LLMs output self-contained Python functions conforming to a specific interface contract.
• Unified backtest engine executing generated code on historical OHLCV data with precomputed technical indicators.
• Evaluation metrics covering return generation, risk exposure, and risk-adjusted efficiency, reported as mean ± standard deviation over multiple runs.
• Stratified analysis by asset class, difficulty level, and model to reveal capability differences and failure modes.
• Temperature ablation to assess the stability of evaluation outcomes under different decoding strategies.

Results & findings

• AlphaForgeBench eliminates execution-induced instability, with intra-query variance an order of magnitude smaller than inter-query variance.
• Model rankings are temperature-invariant, with near-identical performance polygons for tau=0 and tau=0.7, unlike direct-trading benchmarks.
• Gemini-3-Pro-Preview leads in return-oriented metrics (SR=0.628, ARR=0.208), while DeepSeek-V3.2 favors conservative risk control.
• A clear performance hierarchy exists: Gemini-3-Pro is top-tier, followed by Gemini-Flash and Claude, with others trailing.
• Level 3 (Goal-Oriented) tasks provide the highest discriminative power, with inter-model SR spread widening from 0.029 at Level 1 to 0.405 at Level 3.
• Cross-level ranking reversals reveal that DeepSeek excels at code translation (Level 1) but struggles with open-ended strategy design (Level 3).
• Code generation reliability is high, with Pass@1 rates exceeding 96% across all models and difficulty levels.
• Model risk personalities are persistent and asset-agnostic, with Gemini favoring aggressive logic and GPT-5.2 favoring conservative strategies.

Data & experiments

• Stage 1 dataset: 633 real-world single-asset queries curated from brokerage reports, quantitative platforms (WorldQuant, JoinQuant), and open-source repositories.
• Stage 2 dataset: 270 structured queries generated via LLM-augmented processes, organized into a 3x3 level-grade taxonomy (Logic Translation, Logic Completion, Goal-Oriented Generation).
• Evaluation of six frontier LLMs: claude-sonnet-4.5, deepseek-v3.2, gemini-3-flash-preview, gemini-3-pro-preview, gpt-5.2, and grok-4.1-fast.
• Backtesting across seven assets: two cryptocurrencies (BTCUSDT, ETHUSDT) and five US equities (AAPL, GOOGL, MSFT, NVDA, TSLA) over a five-year period (2021-2026).
• Metrics include Annual Rate of Return (ARR), Sharpe Ratio (SR), Maximum Drawdown (MDD), Calmar Ratio (CR), Sortino Ratio (SoR), and Volatility (VOL).
• Temperature ablation experiments comparing greedy (tau=0) and stochastic (tau=0.7) decoding to assess evaluation stability.

Limitations & caveats

• The evaluation is restricted to single-asset trading, excluding portfolio construction effects and multi-asset diversification benefits.
• The backtest engine assumes frictionless execution, ignoring transaction costs, slippage, and market impact which are critical in real-world trading.
• The dataset focuses on technical indicators and price data, potentially underrepresenting the role of fundamental analysis and alternative data.
• The benchmark evaluates code generation and backtesting, but does not assess the LLM's ability to adapt strategies in real-time live markets.
• The study uses a fixed set of seven assets, which may not fully represent the diversity of global financial markets.
• The 'risk personalities' observed may be specific to the models and time period studied, requiring further validation across different market regimes.

Curator notes

Discovery status: institutional_trading_focus_promoted. Source links: large language models execution trading