The List / Agents and Multi-Agent Systems / paper
When Agents Trade: Live Multi-Market Trading Benchmark for LLM Agents
Abstract
Although Large Language Model (LLM)-based agents are increasingly used in financial trading, it remains unclear whether they can reason and adapt in live markets, as most studies test models instead of agents, cover limited periods and assets, and rely on unverified data. To address these gaps, we introduce Agent Market Arena (AMA), the first lifelong, real-time benchmark for evaluating LLM-based trading agents across multiple markets. AMA integrates verified trading data, expert-checked news, and diverse agent architectures within a unified trading framework, enabling fair and continuous comparison under real conditions. It implements four agents, including InvestorAgent as a single-agent baseline, TradeAgent and HedgeFundAgent with different risk styles, and DeepFundAgent with memory-based reasoning, and evaluates them across GPT-4o, GPT-4.1, Claude-3.5-haiku, Claude-sonnet-4, and Gemini-2.0-flash. Live experiments on both cryptocurrency and stock markets demonstrate that agent frameworks display markedly distinct behavioral patterns, spanning from aggressive risk-taking to conservative decision-making, whereas model backbones contribute less to outcome variation. AMA thus establishes a foundation for rigorous, reproducible, and continuously evolving evaluation of financial reasoning and trading intelligence in LLM-based agents.
In one sentence
Summary
The paper addresses the critical gap in evaluating LLM-based trading agents by introducing Agent Market Arena (AMA), the first lifelong, real-time benchmark for assessing financial reasoning and adaptability. Existing benchmarks often test static models or rely on unverified, noisy data over limited periods. AMA provides a unified framework with verified, expert-checked news and price data across equities and cryptocurrencies, enabling continuous, fair comparison of diverse agent architectures under live market conditions. This setup allows for rigorous evaluation of how agents perceive, plan, and act in dynamic, high-stakes environments, moving beyond simple prediction tasks to sequential decision-making. The experimental design deploys four distinct agent frameworks—InvestorAgent, TradeAgent, HedgeFundAgent, and DeepFundAgent—across five leading LLM backbones (GPT-4o, GPT-4.1, Claude-3.5-haiku, Claude-sonnet-4, Gemini-2.0-flash). These agents operate in a live environment trading Tesla (TSLA), BioMarin (BMRN), Bitcoin (BTC), and Ethereum (ETH) for two months. The Market Intelligence Stream aggregates and verifies multi-source data, while the Agent Execution Protocol standardizes inputs and outputs. Performance is tracked via cumulative return, Sharpe ratio, volatility, and maximum drawdown, with a transparent leaderboard providing real-time analytics. Results indicate that LLM-based agents can outperform buy-and-hold strategies, but agent architecture is the dominant factor shaping performance, outweighing the choice of LLM backbone. InvestorAgent achieved high Sharpe ratios (e.g., 6.47 on TSLA with GPT-4.1), while DeepFundAgent showed balanced adaptability. TradeAgent and HedgeFundAgent exhibited aggressive risk-taking with higher volatility. The study highlights that specific agent-model pairings yield optimal results, suggesting complementarity. Limitations include the short evaluation period and the specific asset selection, though the framework is designed for continuous evolution. The work establishes a reproducible foundation for studying financial intelligence in autonomous agents.
Research problem
The paper addresses the lack of rigorous, real-time evaluation frameworks for LLM-based trading agents. Current benchmarks often test static models rather than autonomous agents, rely on unverified or noisy data, and cover limited time periods and asset classes. This makes it difficult to assess whether agents can truly reason, adapt, and perform profitably in live, dynamic financial markets.
Core contributions
- Introduction of Agent Market Arena (AMA), the first lifelong, real-time benchmark for LLM-based trading agents.
- Development of a verified, multi-source data pipeline integrating expert-checked news and price data to ensure input quality.
- Implementation of four diverse agent architectures (single-agent, multi-agent, role-based, memory-based) for comparative analysis.
- Live deployment across equities and cryptocurrencies, providing empirical evidence on agent adaptability and performance.
- Transparent leaderboard and analytics interface for continuous, reproducible evaluation of trading intelligence.
Method
- Market Intelligence Stream (MIS) aggregates and summarizes multi-source data with expert verification.
- Agent Execution Protocol (AEP) standardizes inputs, outputs, and hyperparameters for fair comparison.
- Four agent designs: single-agent (InvestorAgent), multi-role (TradeAgent), hierarchical (HedgeFundAgent), memory-based (DeepFundAgent).
- Daily trading decisions (Buy/Sell/Hold) executed synchronously.
- Performance metrics calculated daily and updated on a live dashboard.
- Qualitative analysis of agent behavior during specific market events (e.g., BTC volatility).
Results & findings
- Agent architecture is the dominant factor in trading performance, more so than the LLM backbone.
- InvestorAgent with GPT-4.1 achieved a Sharpe ratio of 6.47 on TSLA, outperforming Buy & Hold.
- DeepFundAgent showed balanced performance with a Sharpe ratio of 1.96 on BMRN.
- TradeAgent and HedgeFundAgent exhibited aggressive risk-taking with higher volatility.
- Switching LLMs within a fixed agent framework caused less performance variation than switching agents.
- Specific agent-model pairings (e.g., TradeAgent with Gemini-2.0-flash) showed notable complementarity.
- Agents demonstrated ability to adapt to market events, such as hedging during peak optimism.
Data & experiments
- Live trading data for Tesla (TSLA), BioMarin (BMRN), Bitcoin (BTC), and Ethereum (ETH) over two months.
- Multi-source news and price data aggregated from APIs and social media, verified for accuracy and bias.
- Four agent frameworks: InvestorAgent, TradeAgent, HedgeFundAgent, DeepFundAgent.
- Five LLM backbones: GPT-4o, GPT-4.1, Claude-3.5-haiku, Claude-sonnet-4, Gemini-2.0-flash.
- Metrics: Cumulative Return, Annualized Return, Annualized Volatility, Sharpe Ratio, Maximum Drawdown.
- Comparison against Buy & Hold baseline and analysis of agent-model complementarity.
Limitations & caveats
- Evaluation period is limited to two months, which may not capture long-term market cycles.
- Asset selection is limited to two stocks and two cryptocurrencies, potentially lacking diversity.
- Results may vary with different market conditions or longer time horizons.
- The study focuses on daily trading frequency, which may not reflect high-frequency trading dynamics.
- Reliance on specific LLM APIs introduces potential biases or changes in model behavior over time.
Curator notes
Discovery status: round3_promoted_seed_for_round4. Source links: FLAG-Trader: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Trading; Plutus: Benchmarking Large Language Models in Low-Resource Greek Finance