QuantCode-Bench: A Benchmark for Evaluating the Ability of Large Language Models to Generate Executable Algorithmic Trading Strategies

In one sentence

Summary

This paper addresses the gap in evaluating Large Language Models' ability to generate executable algorithmic trading strategies, a task requiring domain-specific financial logic, API knowledge, and semantic alignment beyond standard code generation. The authors introduce QuantCode-Bench, a benchmark of 400 tasks derived from Reddit, TradingView, StackExchange, GitHub, and synthetic sources, designed to test strategy generation for the Backtrader framework. The evaluation pipeline enforces four nested criteria: syntactic correctness, successful backtest execution, presence of trades, and semantic validation via an LLM judge, distinguishing technical executability from functional trading behavior. Experiments compare state-of-the-art models in single-turn and agentic multi-turn settings. Results show that while compilation rates are near-perfect for frontier models, performance drops significantly at the trade and judge stages in single-turn mode, with top models achieving only 70-76% Judge Pass. In the agentic setting, iterative feedback allows models to repair local errors, boosting success rates to 95-98%. Error analysis reveals that failures are rarely syntactic; instead, they stem from incorrect operationalization of trading logic, improper API usage (e.g., Line object handling), and failure to match the task's semantic intent, particularly when conditions do not activate on data. The study concludes that trading strategy generation is a distinct class of domain-specific code generation where success requires aligning natural language descriptions with financial logic and observable data behavior. Limitations include reliance on a single framework (Backtrader), the use of an LLM judge which may have biases, and the lack of evaluation on profitability or risk robustness. The benchmark highlights that general-purpose models often outperform code-specialized models in semantic interpretation, suggesting that instruction-following capabilities are critical for financial code generation.

Research problem

The paper investigates the ability of Large Language Models to generate executable algorithmic trading strategies from textual descriptions, a task that requires mastering domain-specific financial logic, specialized API usage, and semantic alignment, which is underexplored compared to general code generation benchmarks.

Core contributions

• Introduction of QuantCode-Bench, a benchmark with 400 tasks for evaluating LLMs in generating Backtrader trading strategies.
• Development of a multi-stage evaluation pipeline checking syntax, execution, trade presence, and semantic alignment via an LLM judge.
• Comprehensive comparison of state-of-the-art models in single-turn and agentic multi-turn settings.
• Detailed error analysis identifying dominant failure modes like logic operationalization and API misuse rather than syntax.
• Release of the benchmark dataset and code to support future research in financial code generation.

Method

• Task definition: Generate Backtrader code from English descriptions.
• Four-stage validation: Compilation, Backtest execution, Trade presence, LLM Judge semantic alignment.
• LLM Judge evaluates indicator usage, entry/exit logic, and task adherence.
• Single-turn setting: One-shot generation without feedback.
• Agentic multi-turn setting: Up to 10 iterations with structured error feedback.
• Error analysis classifying failures by stage and specific technical/semantic causes.

Results & findings

• Frontier models achieve near 100% compilation but only 70-76% Judge Pass in single-turn settings.
• Agentic multi-turn settings improve performance to 95-98% for top models like Claude Opus 4.6.
• Compilation is no longer the bottleneck; failures occur at execution, trade generation, and semantic alignment.
• Most common failure is strategies that compile and backtest but place no trades due to strict conditions.
• General-purpose models often outperform code-specialized models in semantic interpretation tasks.
• Iterative feedback effectively repairs technical errors but struggles with fundamental semantic misunderstandings.

Data & experiments

• Dataset of 400 tasks from Reddit, TradingView, StackExchange, GitHub, and synthetic sources.
• Evaluation using the Backtrader framework on historical market data.
• Two settings: single-turn generation and agentic multi-turn with iterative feedback.
• Metrics include compilation rate, backtest success, trade presence, and LLM Judge Pass rate.
• Error analysis categorizing failures by stage and type (e.g., signal activation, API errors).

Limitations & caveats

• Evaluated only on the Backtrader framework, limiting transferability to other libraries like LEAN or Zipline.
• Relies on an LLM judge for semantic validation, which may introduce biases or miss subtle errors.
• Does not evaluate profitability, risk robustness, or economic quality of the generated strategies.
• Focuses on executable code generation rather than investment performance or alpha generation.
• Synthetic tasks may not fully capture the complexity of real-world trading ideas.

Curator notes

Discovery status: institutional_trading_focus_promoted. Source links: large language models execution trading