Agent Trading Arena: A Study on Numerical Understanding in LLM-Based Agents

In one sentence

Summary

The paper addresses the limitation of static backtesting in financial LLM research by introducing the Agent Trading Arena, a closed-loop, zero-sum virtual stock market. This environment forces LLM-based agents to interact via bid-ask mechanisms, directly influencing price dynamics and simulating realistic market friction, liquidity constraints, and slippage. The study positions this framework as a necessary step to bridge the gap between training and evaluation in dynamic financial environments, moving beyond passive historical analysis to active, adaptive decision-making where agents must compete against each other and adapt to shifting market conditions in real-time. The proposed method, ArenaTrader, employs a three-stage pipeline: market analysis, trade execution, and strategic reflection. Agents process multi-modal inputs, including plain-text time-series data and visualizations like line charts and bar graphs, while also accessing a shared 'Chat Pool' for noisy or informative rumors. A key component is the reflection module, which allows agents to evaluate past strategies, contrast top and bottom performers, and refine future actions. Experiments were conducted on NASDAQ and CSI datasets using various LLMs (GPT-4o, Gemini-1.5, Qwen-VL) and compared against baselines such as Buy & Hold, SMA, MACD, and specialized models like StockFormer and TimesNet. Metrics included Total Return, Sharpe Ratio, and Win Rate. Results indicate that LLMs struggle with numerical reasoning on plain-text data, often overfitting to local patterns, whereas visual inputs substantially improve performance. The combination of visual data and the reflection module yielded the best results, with GPT-4o achieving a Sharpe Ratio of 0.348 on NASDAQ, significantly outperforming the benchmark and traditional quantitative models. The agent demonstrated robustness, particularly in high-volatility regimes, without task-specific retraining. Limitations include the synthetic nature of the zero-sum environment, which may not fully capture external macroeconomic shocks, and the reliance on vision-capable LLMs, which may not be universally accessible or cost-effective for high-frequency trading applications.

Research problem

Existing LLM-based financial agents rely on static historical backtesting, which fails to capture the interactive nature of real markets where trading actions influence prices. This paper addresses the lack of dynamic, multi-agent simulation frameworks that can evaluate LLMs' ability to adapt to real-time market feedback, liquidity constraints, and competitive strategies.

Core contributions

• Design of the Agent Trading Arena, a closed-loop, zero-sum virtual market simulating bid-ask interactions and price impact.
• Demonstration that LLMs struggle with textual numerical reasoning but excel with visual chart inputs.
• Introduction of a reflection module that iteratively refines trading strategies based on performance feedback.
• Empirical validation showing LLM agents outperform traditional quantitative baselines and specialized models on NASDAQ and CSI datasets.

Method

• Closed-loop simulation where agent orders directly impact market prices via bid-ask matching.
• Multi-modal input processing combining plain-text time-series data with line/bar charts.
• Shared Chat Pool for inter-agent communication, including rumors and noisy signals.
• Three-stage agent pipeline: market analysis, action decision, and strategy optimization.
• Reflection module for evaluating past strategies and archiving unsuccessful tactics.
• Zero-sum competition forcing agents to adapt to opponent behaviors and market shifts.

Results & findings

• GPT-4o achieved a Total Return of 15.99% on NASDAQ and 48.93% on CSI, outperforming all baselines.
• Sharpe Ratio of 0.348 on NASDAQ benchmark, significantly higher than the benchmark's 0.189.
• Visual inputs improved Total Return for GPT-4o from 33.65% (text-only) to 47.69% (visual+reflection).
• LLMs overfit to local patterns in text but capture global trends effectively with charts.
• The reflection module boosted Sharpe Ratios and Win Rates, indicating more stable and consistent trading.
• Agents performed robustly in high-volatility periods without task-specific retraining.

Data & experiments

• Simulations on NASDAQ (2 months) and CSI (1 year) datasets using real-world price and volume data.
• Comparison against baselines: Buy & Hold, SMA, ZMR, MACD, StockFormer, TimesNet, and StockMixer.
• Evaluation of multiple LLM backbones including GPT-4o, Gemini-1.5, Qwen-VL, and LLaMa-3.
• Ablation studies on input modalities (text vs. visual) and the impact of the reflection module.
• Metrics: Total Return, Sharpe Ratio, Win Rate, Mean Daily Return, and Return Volatility.

Limitations & caveats

• The zero-sum environment is synthetic and may not fully replicate real-world macroeconomic shocks.
• Reliance on vision-capable LLMs may limit accessibility and increase inference costs.
• The simulation isolates external knowledge, which may reduce generalizability to real-world fundamental analysis.
• Performance may vary significantly with different LLM backbones and prompt engineering.
• The study focuses on stock trading, limiting applicability to other asset classes without adaptation.

Curator notes

Discovery status: institutional_trading_focus_promoted. Source links: LLM option trading financial markets