When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments

In one sentence

Summary

This paper addresses the limitation of static backtesting in finance by introducing StockAgent, a multi-agent framework that simulates dynamic stock market environments using Large Language Models. The research aims to understand how external factors—such as macroeconomic policies, company fundamentals, and social sentiment—affect trading decisions and market dynamics. It specifically investigates the reliability of LLM-based agents in simulating realistic investor behaviors, including personality traits and risk preferences, while ensuring that agents do not rely on prior knowledge of test data to prevent information leakage. The methodology employs a multi-agent system where LLMs (GPT-3.5 and Gemini) drive individual investor agents with distinct personalities (Conservative, Aggressive, etc.). The simulation incorporates a transaction module with a random clock page replacement algorithm to manage order books and prevent deadlocks, and a Bulletin Board System (BBS) for agents to share trading tips. Experiments involve simulating one year of trading for two anonymized stocks, incorporating real-world events like interest rate hikes and financial report releases. The study evaluates performance across different external condition ablations (e.g., removing BBS or loan options) and compares the trading outcomes and profitability of different LLM backbones. Results indicate that external factors significantly impact agent trading behavior and profitability, with the BBS module notably influencing market volatility and agent decisions. The simulation reveals inherent tendencies in LLMs, such as risk aversion or aggression, which affect strategy generation. The framework successfully avoids test set leakage by restricting agents to real-time information. However, limitations include the simplified market mechanics, the use of only two LLMs, and the synthetic nature of the stock data, which may limit direct applicability to live trading without further validation against real market microstructure.

Research problem

The paper addresses the inadequacy of traditional backtesting methods, which suffer from static nature and retrospective bias, in capturing the dynamic interplay of market participants and external factors. It seeks to determine if LLM-based agents can realistically simulate investor behavior and market dynamics in response to macroeconomic events, policy changes, and social sentiment. A key problem is preventing test set leakage, where agents might exploit prior knowledge of market data, thereby invalidating the simulation's realism.

Core contributions

• Development of StockAgent, a multi-agent LLM framework for simulating stock trading with distinct investor personalities.
• Implementation of a BBS module to simulate social sentiment and information diffusion among agents.
• Introduction of a random clock page replacement algorithm to manage concurrent trading orders and prevent deadlocks.
• Demonstration of how external factors (monetary policy, financial reports) influence agent trading behavior and profitability.
• Analysis of LLM inherent tendencies (e.g., risk preference) on trading strategies and market outcomes.
• Mitigation of test set leakage by restricting agents to real-time information without prior market data access.

Method

• Multi-agent system architecture with Investment Agent, Transaction, and BBS modules.
• LLM-driven decision making using GPT-3.5 and Gemini for buy/sell/hold/loan actions.
• Random clock page replacement algorithm for order book management and deadlock prevention.
• Prompt engineering for agent reasoning, including Chain of Thought for financial analysis.
• Simulation of market events such as interest rate changes and quarterly financial reports.
• Ablation studies to isolate the impact of BBS, loans, and financial data on agent behavior.
• Profitability and performance metrics calculated based on agent asset values and trading logs.

Results & findings

• Agents with different personalities exhibit distinct trading behaviors and profitability levels.
• The BBS module significantly influences market volatility and agent decision-making through social sentiment.
• External macroeconomic events cause measurable shifts in agent trading strategies and stock prices.
• LLM-based agents show inherent risk preferences that affect their long-term profitability.
• The framework successfully prevents test set leakage, ensuring agents rely only on real-time information.
• Removing loan options or financial statements reduces agent profitability and market efficiency.
• Simulated stock prices react to financial reports and policy changes in ways consistent with behavioral finance theories.

Data & experiments

• Simulation of two anonymized US stocks (one established, one IPO) over 264 trading days.
• Use of GPT-3.5-turbo and Gemini-pro-1.0 as backbone LLMs for agent decision-making.
• Incorporation of real-world macroeconomic events (interest rate hikes, reserve ratio reductions) from 2014-2019.
• Ablation studies removing specific external factors (BBS, loans, financial statements) to isolate their impact.
• Evaluation of agent profitability, trading volume, and stock price movements under various scenarios.
• Comparison of trading strategies and outcomes across different agent personality types.

Limitations & caveats

• Simulation uses simplified market mechanics, lacking real-world order book depth and latency.
• Only two LLMs were tested, limiting the generalizability of findings to other models.
• Stock data is anonymized and synthetic, reducing direct applicability to live trading.
• Agent personalities are predefined, potentially missing the complexity of real investor heterogeneity.
• The BBS module is a simplified representation of social media, lacking noise and misinformation.
• Long-term market impacts and black swan events are not fully captured in the simulation.

Curator notes

Discovery status: round2_promoted. Source links: A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist; Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges; Learning to Trade Like an Expert: Cognitive Fine-Tuning for Stable Financial Reasoning in Language Models; LiveTradeBench: Seeking Real-World Alpha with Large Language Models; Revolutionizing Finance with LLMs: An Overview of Applications and Insights