StockBench: Can Large Language Models Beat the Stock Market?

In one sentence

Summary

This paper introduces STOCKBENCH, a benchmark designed to evaluate Large Language Model (LLM) agents in realistic, dynamic stock trading environments, addressing the gap left by static financial question-answering benchmarks. The benchmark focuses on sequential decision-making over a multi-month horizon, requiring agents to process daily market signals including prices, fundamentals, and news to make buy, sell, or hold decisions. The primary goal is to assess both profitability and risk management capabilities in a contamination-free setting using recent market data from March to June 2025, ensuring no overlap with model training corpora. The experimental setup involves a back-trading environment with 20 high-weight Dow Jones Industrial Average (DJIA) stocks. Agents operate through a four-step workflow: portfolio overview, in-depth stock analysis, decision generation, and execution validation. The study evaluates a diverse set of proprietary and open-source LLMs, including GPT-5, Claude-4, Qwen3, and Kimi-K2, against a passive equal-weight buy-and-hold baseline. Performance is measured using cumulative return, maximum drawdown, and the Sortino ratio, with a composite ranking derived from z-scores of these metrics to balance profit and risk. Results indicate that while most LLM agents achieve lower maximum drawdowns than the passive baseline, indicating effective downside risk management, many fail to significantly outperform the simple buy-and-hold strategy in terms of cumulative return. Kimi-K2 emerges as the top performer, achieving positive returns and robust risk profiles, whereas reasoning-tuned models do not consistently outperform instruction-tuned counterparts. The study highlights that strong performance on static financial QA does not guarantee effective trading behavior, and notes limitations such as the exclusion of trading costs, slippage, and high-frequency trading dynamics.

Research problem

The paper addresses the lack of rigorous benchmarks for evaluating LLM agents in dynamic, real-world financial trading environments. Existing benchmarks primarily focus on static question-answering, failing to capture the sequential decision-making, market dynamics, and risk management required for actual trading. The research problem is to determine if current LLMs can effectively translate their reasoning capabilities into profitable and risk-aware trading strategies in a contamination-free, multi-month setting.

Core contributions

• Introduction of STOCKBENCH, a contamination-free benchmark for evaluating LLM agents in realistic, multi-month stock trading environments.
• Development of a minimal yet realistic agent workflow that simulates retail investor decision-making processes including portfolio overview and in-depth analysis.
• Comprehensive evaluation of state-of-the-art proprietary and open-source LLMs against a passive buy-and-hold baseline using financial metrics like return, drawdown, and Sortino ratio.
• Open-sourcing of the benchmark implementation to facilitate reproducibility and future research on LLM-driven financial agents.

Method

• Back-trading environment simulation with strict temporal separation to prevent data leakage.
• Four-step agent workflow: Portfolio Overview, In-Depth Stock Analysis, Decision Generation, Execution and Validation.
• Daily decision-making process with actions: increase, decrease, or hold position.
• Execution logic converting dollar targets to share quantities based on opening prices, with liquidity checks.
• Composite ranking using z-scores of Return, Drawdown, and Sortino Ratio to balance profit and risk.
• Ablation studies on input modalities (news, fundamentals) and portfolio size scalability.

Results & findings

• Kimi-K2 achieved the highest composite rank with a 1.9% return and -11.8% max drawdown, outperforming the passive baseline.
• Most LLM agents achieved lower maximum drawdowns than the passive baseline (-15.2%), indicating better downside risk management.
• Reasoning-tuned models (e.g., Qwen3-Think) did not consistently outperform instruction-tuned models (e.g., Qwen3-Ins), suggesting reasoning ability alone is insufficient for trading.
• GPT-5 and GPT-OSS models generally underperformed, with GPT-OSS-120B showing high variance and poor returns in downturns.
• Ablation studies showed that removing news and fundamental data significantly reduced cumulative returns for most models.
• Model rankings shifted significantly between market downturns and upturns, indicating sensitivity to market conditions.

Data & experiments

• Evaluation window: March 3, 2025, to June 30, 2025 (82 trading days), ensuring no data leakage from model training.
• Investment targets: Top 20 stocks by weight from the Dow Jones Industrial Average (DJIA), covering technology, finance, and manufacturing.
• Data inputs: Daily opening prices, fundamental metrics (market cap, P/E, dividend yield), and up to five recent news articles per stock.
• Models tested: GPT-5, Claude-4-Sonnet, Qwen3 (235B/30B), Kimi-K2, GLM-4.5, DeepSeek-V3/V3.1, and GPT-OSS variants.
• Baseline: Equal-weight buy-and-hold strategy allocating initial capital equally across all selected stocks.
• Metrics: Final Return, Maximum Drawdown, Sortino Ratio, and a composite z-score ranking.

Limitations & caveats

• Evaluation is limited to daily trading, excluding high-frequency or intraday strategies.
• The four-month window may not capture long-term market cycles or rare events.
• Trading costs, slippage, and liquidity limits are not modeled, making trading easier than in reality.
• Focus is restricted to large-cap DJIA stocks, limiting generalizability to other asset classes.
• Performance may vary significantly across different market conditions (downturn vs. upturn).

Curator notes

Useful for checking whether LLM trading conclusions survive stricter evaluation.