The List / Agents and Multi-Agent Systems / paper
LiveTradeBench: Seeking Real-World Alpha with Large Language Models
Abstract
Large language models (LLMs) achieve strong performance across benchmarks--from knowledge quizzes and math reasoning to web-agent tasks--but these tests occur in static settings, lacking real dynamics and uncertainty. Consequently, they evaluate isolated reasoning or problem-solving rather than decision-making under uncertainty. To address this, we introduce LiveTradeBench, a live trading environment for evaluating LLM agents in realistic and evolving markets. LiveTradeBench follows three design principles: (i) Live data streaming of market prices and news, eliminating dependence on offline backtesting and preventing information leakage while capturing real-time uncertainty; (ii) a portfolio-management abstraction that extends control from single-asset actions to multi-asset allocation, integrating risk management and cross-asset reasoning; and (iii) multi-market evaluation across structurally distinct environments--U.S. stocks and Polymarket prediction markets--differing in volatility, liquidity, and information flow. At each step, an agent observes prices, news, and its portfolio, then outputs percentage allocations that balance risk and return. Using LiveTradeBench, we run 50-day live evaluations of 21 LLMs across families. Results show that (1) high LMArena scores do not imply superior trading outcomes; (2) models display distinct portfolio styles reflecting risk appetite and reasoning dynamics; and (3) some LLMs effectively leverage live signals to adapt decisions. These findings expose a gap between static evaluation and real-world competence, motivating benchmarks that test sequential decision making and consistency under live uncertainty.
In one sentence
Summary
The paper addresses the critical gap in evaluating Large Language Models (LLMs) for financial decision-making, noting that static benchmarks fail to capture real-world dynamics, uncertainty, and sequential adaptation. It introduces LiveTradeBench, a live trading environment that streams real-time market prices and news to eliminate information leakage inherent in offline backtesting. The framework employs a portfolio management abstraction, requiring agents to output continuous allocation vectors across multiple assets rather than discrete buy/sell actions, thereby integrating risk management and cross-asset reasoning into a single decision loop. The experimental setup involves a 50-day live evaluation of 21 mainstream LLMs across six families (including GPT-5, Claude, and Llama) in two distinct markets: U.S. equities (15 stocks/ETFs) and Polymarket prediction markets (10 binary contracts). Agents operate within a ReAct-style framework equipped with tool use for data filtering and a memory module storing recent observations. Performance is measured using cumulative return, Sharpe ratio, maximum drawdown, win rate, and volatility. The study compares these live results against static benchmark scores to assess the transferability of general intelligence to financial execution. Key findings indicate a disconnect between general LLM capabilities and trading performance; top models on LMArena do not necessarily achieve superior trading outcomes. Models display distinct portfolio styles reflecting varying risk appetites, with some effectively leveraging live news signals to adapt allocations. The study highlights that success in one market (e.g., stocks) does not generalize to another (e.g., prediction markets), suggesting that LLMs require market-specific reasoning strategies. The work provides a valuable benchmark for assessing the robustness of LLM agents under live uncertainty, though it notes limitations regarding the long-term sustainability of strategies and the impact of transaction costs not fully modeled in the allocation abstraction.
Research problem
Current LLM evaluations rely on static benchmarks that lack real-time dynamics and uncertainty, failing to test sequential decision-making under live market conditions. Existing trading benchmarks often suffer from information leakage via offline backtesting or oversimplify trading as single-asset discrete actions, neglecting portfolio-level risk management. This paper seeks to evaluate LLM agents' true trading competence in realistic, evolving markets.
Core contributions
- Introduction of LiveTradeBench, a live trading environment streaming real-time prices and news to prevent information leakage.
- A portfolio management abstraction that requires LLMs to output continuous allocation vectors, integrating risk and cross-asset reasoning.
- Multi-market evaluation across structurally distinct environments: U.S. stocks and Polymarket prediction markets.
- Comprehensive 50-day live evaluation of 21 LLMs, revealing a disconnect between general reasoning scores and trading performance.
- Analysis of distinct portfolio styles and risk appetites exhibited by different LLM families in live trading scenarios.
Method
- Live data streaming of market prices and news to ensure real-time evaluation.
- Portfolio management abstraction with continuous allocation vectors summing to 1.0.
- ReAct-style agent framework with tool use for data filtering and memory for historical context.
- Long-only constraint for U.S. stocks; binary YES/NO allocation for Polymarket.
- Evaluation metrics: CR, SR, MDD, WR, and volatility.
- Unified prompting and parsing pipeline across 21 LLM families.
- Comparison of live performance against static benchmark rankings.
Results & findings
- High LMArena scores do not correlate with superior live trading outcomes.
- Qwen2.5-72B-Instruct achieved the highest cumulative return (5.15) in U.S. stocks.
- GPT-o3 achieved the highest Sharpe ratio (2.57) in U.S. stocks.
- Polymarket performance was highly volatile, with some models suffering >50% drawdowns.
- Sharpe ratio correlation between stocks and Polymarket was near zero, indicating no cross-market generalization.
- Models exhibited distinct portfolio styles, reflecting different risk appetites and reasoning dynamics.
- Some LLMs effectively adapted allocations based on live news signals, while others failed to react appropriately.
Data & experiments
- Live streaming data from U.S. stock market (15 equities/ETFs) and Polymarket prediction markets (10 binary contracts).
- 50-day live evaluation period from August 18, 2025, to October 24, 2025.
- Evaluation of 21 LLMs across 6 families: OpenAI, Anthropic, Google, xAI, Meta, Qwen, DeepSeek, Moonshot.
- Metrics: Cumulative return, Sharpe ratio, maximum drawdown, win rate, and volatility.
- Comparison of live trading performance against static benchmark scores (LMArena).
- Analysis of cross-market generalization and model-specific portfolio styles.
Limitations & caveats
- Evaluation period is limited to 50 days, potentially missing long-term regime changes.
- Transaction costs are not fully modeled in the allocation abstraction.
- Short positions are not allowed, limiting the scope of strategies.
- Polymarket liquidity and slippage may not be fully captured.
- Results may vary with different market conditions or longer horizons.
- LLM output parsing errors could impact execution fidelity.
Curator notes
Discovery status: round1_promoted_seed_for_round2. Source links: A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist; AlphaAgent: LLM-Driven Alpha Mining with Regularized Exploration to Counteract Alpha Decay; Automate Strategy Finding with LLM in Quant investment; Can LLM-Based Financial Investing Strategies Outperform?; Can Large Language Models Trade? Testing Financial Theories with LLM Agents in Market Simulations; ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answerin