AlphaQuanter: An End-to-End Tool-Augmented Agentic Reinforcement Learning Framework for Stock Trading

In one sentence

Summary

The paper addresses the limitations of existing LLM-based trading agents, which often suffer from inefficiency, inconsistent signals in multi-agent debate pipelines, and a lack of end-to-end optimization for coherent strategy learning. AlphaQuanter proposes a single-agent framework that uses reinforcement learning to learn a dynamic policy over a transparent, tool-augmented decision workflow. This allows the agent to autonomously orchestrate tools and proactively acquire information on demand, establishing a verifiable and traceable reasoning process that overcomes the black-box nature of traditional deep RL and the prompt-sensitivity of current LLM methods. The method models the trading task as a tool-augmented Markov Decision Process where the agent selects between query actions (to gather market, fundamental, sentiment, and macro data) and decision actions (BUY, SELL, HOLD). Training employs a composite reward function combining outcome scores based on smoothed future returns and process scores that penalize inefficient tool use and format violations. Experiments are conducted on five large-cap U.S. stocks (GOOGL, MSFT, META, NVDA, TSLA) using a backtesting protocol over a six-month test period. The framework is evaluated against seven categories of baselines, including passive strategies, rule-based systems, deep RL, and various multi/single-agent LLM setups, using metrics like Annualized Rate of Return (ARR), Sharpe Ratio, and Maximum Drawdown. Results show that AlphaQuanter, particularly the 7B variant, achieves state-of-the-art performance, significantly outperforming multi-agent frameworks and prompt-only single agents in ARR and risk-adjusted returns. The RL training enables the agent to learn sophisticated, expert-like heuristics, such as prioritizing technical indicators over low-frequency fundamentals, while maintaining high faithfulness in its decision traces. However, the framework is limited to single-asset decisions, relies on a predefined toolset without novel analysis generation, and faces context-length challenges due to accumulating tool outputs, highlighting the need for better memory mechanisms.

Research problem

Existing LLM-based trading agents face critical limitations: multi-agent frameworks often produce inconsistent signals and suffer from inefficiency, while prompt-based methods lack robustness and end-to-end optimization. Traditional deep RL methods are black-box and cannot integrate heterogeneous external signals on demand. The paper addresses the need for a framework that enables evidence-grounded, traceable, and robust trading decisions through autonomous tool orchestration and reinforcement learning.

Core contributions

• Proposes AlphaQuanter, a single-agent framework with ReAct-like reasoning chains that ensure decision consistency and traceable tool-use trajectories grounded in external evidence.
• Designs an end-to-end reinforcement learning approach that trains the agent to actively acquire useful information and select supporting evidence, directly optimizing the entire decision-making process for long-term profitability.
• Introduces a composite reward function combining outcome scores (based on smoothed future returns) and process scores (penalizing inefficient tool use and format violations) to guide training.
• Demonstrates that AlphaQuanter achieves state-of-the-art performance on key financial metrics and improves decision-trace faithfulness compared to multi-agent and prompt-only baselines.
• Provides empirical evidence that specialized RL training paradigms can enable small LLMs to learn robust trading policies that surpass powerful zero-shot models like GPT-4o.

Method

• Single-agent ReAct-like workflow: Interleaves reasoning traces with discrete actions (query or decision) in an iterative loop.
• Tool-augmented MDP: State includes initial context, query history, and results; actions include querying four data categories or making a trading decision.
• Outcome reward: Based on exponentially weighted forward returns, with asymmetric penalties for incorrect actions in strong trends.
• Process reward: Penalizes excessive verbosity, inefficient tool use, and malformed tool calls to encourage structured reasoning.
• RL training: Uses GRPO algorithm with Qwen2.5-3B/7B backbones, optimizing the policy for long-term profitability.
• Backtesting simulation: Applies daily decisions to a portfolio with transaction fees and slippage buffers to measure risk-adjusted returns.

Results & findings

• AlphaQuanter-7B achieves an average ARR of 34.94% and Sharpe Ratio of 0.65, outperforming all baselines including GPT-4o-based multi-agent systems.
• Single-agent architecture consistently outperforms multi-agent debate frameworks, which often inject noise and amplify hallucinations.
• Prompt-based single agents fail to beat the market baseline on average, indicating that prompt-only reasoning is insufficient for reliable trading decisions.
• RL training significantly improves decision-trace faithfulness, with higher alignment and conciseness scores compared to multi-agent and prompt-only single-agent methods.
• The 7B model learns a selective, discriminative policy, prioritizing technical indicators (trend, momentum, volume) while downweighting low-frequency fundamental data.
• Ablation studies show that removing process rewards causes ARR drops of over 40%, highlighting their critical role in guiding effective decision-making.
• AlphaQuanter maintains low token cost per decision (0.15x the cost of multi-agent baselines) while achieving superior performance.

Data & experiments

• Dataset: Five large-cap U.S. stocks (GOOGL, MSFT, META, NVDA, TSLA) with non-overlapping train/valid/test splits to prevent leakage.
• Data Sources: Market data (OHLCV, technical indicators), fundamental data (financial statements, insider transactions), sentiment data (news, social media), and macroeconomic indicators.
• Baselines: Market (buy-and-hold), Rule (MACD, ZMR), RL (A2C, PPO), LM (Chronos-2), LLM (FinMem, TradingAgents), Multi-Agent (TradingAgents with various backbones), and Single-Agent (AlphaQuanter without RL).
• Evaluation Metrics: Annualized Rate of Return (ARR), Sharpe Ratio (SR), Maximum Drawdown (MDD), and human-evaluated faithfulness metrics (Alignment, Grounding, Conciseness).
• Backtesting Protocol: Daily-trained policy applied sequentially over a six-month historical period to measure portfolio performance.

Limitations & caveats

• Limited to single-asset decision making; does not address portfolio-level optimization.
• Capabilities are bounded by the predefined toolset; cannot generate novel analytical methods.
• Context length can become excessively long due to accumulating tool outputs, requiring better memory mechanisms.
• Evaluated only on five large-cap U.S. stocks; generalizability to other markets or assets is untested.
• Backtesting environment may not fully capture real-world market dynamics and liquidity constraints.

Curator notes

Discovery status: trading_agent_focus_promoted. Source links: FinMem: A Performance-Enhanced Large Language Model Trading Agent with Layered Memory and Character Design