The List / Agents and Multi-Agent Systems / paper
StockSim: A Dual-Mode Order-Level Simulator for Evaluating Multi-Agent LLMs in Financial Markets
Abstract
We present StockSim, an open-source simulation platform for systematic evaluation of large language models (LLMs) in realistic financial decision-making scenarios. Unlike previous toolkits that offer limited scope, StockSim delivers a comprehensive system that fully models market dynamics and supports diverse simulation modes of varying granularity. It incorporates critical real-world factors, such as latency, slippage, and order-book microstructure, that were previously neglected, enabling more faithful and insightful assessment of LLM-based trading agents. An extensible, role-based agent framework supports heterogeneous trading strategies and multi-agent coordination, making StockSim a uniquely capable testbed for NLP research on reasoning under uncertainty and sequential decision-making. We open-source all our code at https: //github.com/harrypapa2002/StockSim.
In one sentence
Summary
The paper addresses the critical gap in standardized, realistic evaluation infrastructure for Large Language Models (LLMs) in financial decision-making. Existing benchmarks often suffer from data leakage or abstract away crucial market microstructure details like latency and slippage. StockSim introduces a unified, open-source platform that supports two complementary simulation modes: an order-level execution mode that emulates real-time limit-order book dynamics, queueing, and market impact, and a candlestick-level mode for scalable, high-throughput testing. The system features an asynchronous architecture with role-based multi-agent coordination, allowing heterogeneous LLMs to act as specialist analysts (technical, fundamental, news) coordinated by a trader agent, thereby enabling rigorous assessment of reasoning under uncertainty and sequential decision-making in dynamic financial environments. The methodology leverages a modular exchange simulation engine that integrates real-time market data from providers like Polygon.io and Alpha Vantage, alongside external news and fundamental streams. The engine processes agent actions via RabbitMQ, supporting limit, market, and stop orders with realistic latency and slippage modeling. Experiments were conducted on NVIDIA stock data from April to June 2025, comparing GPT-o3 and GPT-o4-mini agents using identical prompts. The evaluation framework computes comprehensive performance metrics including ROI, Sharpe ratio, Sortino ratio, win rate, and profit factor. The system was also stress-tested for scalability with deterministic agents, demonstrating linear scaling up to 150 concurrent agents on consumer hardware, ensuring reproducibility and consistent simulation outputs across runs. Results indicate distinct trading strategies between the evaluated models: GPT-o3 adopted a selective, high-conviction approach with fewer trades, higher profit per trade, and superior risk-adjusted returns (Sharpe 5.97 vs 2.62), while GPT-o4-mini exhibited a more active, lower-efficiency trading style. The platform successfully captures these behavioral differences, highlighting its utility for NLP research on agent coordination and reasoning. Limitations include the platform's computational demands for large-scale multi-agent simulations and the inherent simplification of market complexities such as liquidity constraints and full market impact. Despite these caveats, StockSim provides a vital, production-grade testbed for bridging NLP research with real-world deployment requirements in financial markets.
Research problem
The NLP community lacks standardized, open-source platforms for rigorously evaluating LLMs in realistic financial trading scenarios. Current benchmarks often suffer from data leakage due to static datasets appearing in training corpora, or they abstract away critical market microstructure details like latency, slippage, and order-book dynamics, leading to inflated performance metrics and poor generalization to unseen scenarios.
Core contributions
- Introduction of StockSim, a unified, open-source simulation platform supporting both order-level and candlestick-level execution modes.
- Implementation of a realistic market microstructure engine that models latency, slippage, queue dynamics, and market impact.
- Development of an extensible, role-based multi-agent framework allowing heterogeneous LLMs to coordinate as specialist analysts.
- Provision of a comprehensive evaluation suite with customizable performance metrics and interactive visualization tools.
- Demonstration of distinct trading behaviors between GPT-o3 and GPT-o4-mini, validating the platform's ability to differentiate agent strategies.
Method
- Asynchronous architecture using RabbitMQ for reliable message delivery between agents and the simulation engine.
- Order-level execution engine emulating limit-order book dynamics with price-time priority matching.
- Candlestick-level execution engine for scalable testing using aggregated OHLCV data with simulated intra-bar paths.
- Modular agent design allowing separate technical, fundamental, and news analysts coordinated by a trader.
- Real-time computation of technical indicators (SMA, EMA, RSI, ATR, VWAP) and market embeddings.
- Extensible evaluator module for custom performance metrics and detailed trade history logging.
- Integration of external data providers via lightweight Python adapters for news and fundamentals.
Results & findings
- GPT-o3 achieved a Sharpe Ratio of 5.97 and a Win Rate of 1.0, significantly outperforming GPT-o4-mini.
- GPT-o4-mini had a lower Sharpe Ratio (2.62) and higher trade frequency, indicating a less efficient strategy.
- GPT-o3 generated $27,120.75 in realized P&L compared to $5,427.80 for GPT-o4-mini.
- GPT-o3 executed only 9 trades with a profit factor of 999.0, while GPT-o4-mini executed 31 trades with a factor of 2.37.
- StockSim scales linearly up to ~150 agents, with CPU usage rising from 8% to 27% and memory from 0.8 GB to 2 GB.
- The platform ensures deterministic outputs for identical configurations, validating reproducibility.
- Interactive visualizations allow detailed inspection of order execution prices and corresponding LLM reasoning.
Data & experiments
- Market data sourced from Polygon.io and Alpha Vantage, including OHLCV bars and order-level events.
- External data streams including news headlines, earnings calendars, and fundamental metrics integrated via API adapters.
- Experiments conducted on NVIDIA (NVDA) stock data from April 28, 2025, to June 28, 2025.
- Comparison of GPT-o3 and GPT-o4-mini agents using identical prompts and daily trading constraints.
- Scalability tests with deterministic agents up to 500 concurrent instances to assess system resource usage.
- Evaluation metrics include ROI, Sharpe Ratio, Sortino Ratio, Win Rate, Profit Factor, and Max Drawdown.
Limitations & caveats
- Requires substantial computational resources for large-scale multi-agent simulations.
- Market simulation cannot fully capture all complexities of actual trading, such as deep liquidity constraints.
- Current evaluation metrics may not capture all aspects of decision quality relevant to financial applications.
- Focus on financial markets may limit generalizability to other sequential decision-making domains.
- Scalability becomes super-linear beyond 150 agents, with significant CPU and memory spikes.
- Relies on external data providers which may have licensing or availability constraints.
Curator notes
Discovery status: institutional_trading_focus_promoted. Source links: LLM option trading financial markets; large language models order book trading