FLAG-Trader: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Trading

In one sentence

Summary

The paper addresses the limitation of LLMs in sequential financial decision-making by proposing FLAG-Trader, a framework that treats a partially fine-tuned LLM as a policy network within a reinforcement learning setup. It converts market states into structured text prompts, allowing the LLM to leverage pre-trained reasoning capabilities while adapting to financial domains through parameter-efficient fine-tuning of only the top layers. This hybrid approach aims to combine the multimodal understanding of LLMs with the reward-driven optimization of RL, overcoming the static nature of standard LLM inference and the feature-engineering burden of traditional RL. The method employs an actor-critic architecture where the policy and value networks share trainable LLM layers while keeping base layers frozen. Training utilizes Proximal Policy Optimization (PPO) with a reward function based on the Sharpe ratio of cumulative profits. Experiments evaluate the model on five US stocks (MSFT, JNJ, UVV, HON, TSLA) and Bitcoin, comparing it against buy-and-hold, LLM-agentic baselines (InvestorBench), and various large proprietary and open-source models. The 135M-parameter SmolLM2 backbone is fine-tuned using structured prompts containing price history, account status, and recent decision metrics. Results show FLAG-Trader consistently outperforms baselines in Cumulative Return and Sharpe Ratio, notably enabling the small open-source model to surpass larger proprietary models like GPT-4 and GPT-o1-preview in several assets. The framework demonstrates convergence to stable policies and robustness across different market conditions. However, limitations include high computational costs during fine-tuning, potential biases from structured prompts, and a lack of explicit risk-sensitive constraints beyond Sharpe ratio optimization, which may limit generalization in highly volatile or non-stationary markets.

Research problem

Large language models (LLMs) excel at static financial reasoning but struggle with multi-step, goal-oriented sequential decision-making in interactive trading environments. Traditional reinforcement learning (RL) approaches often fail to integrate multimodal data (text and time-series) effectively and rely heavily on manual feature engineering. The paper seeks to design a unified framework that seamlessly integrates LLMs' reasoning capabilities with RL's reward-driven optimization to tackle the challenges of financial sequential decision-making.

Core contributions

• Proposes FLAG-Trader, a unified architecture integrating linguistic processing via LLMs with gradient-driven RL policy optimization.
• Introduces a parameter-efficient fine-tuning module where a partially fine-tuned LLM acts as the policy network, balancing domain adaptation and knowledge retention.
• Designs a hybrid RL component that incorporates external environment reward gradients into policy updates using PPO, aligning LLM outputs with trading performance metrics.
• Demonstrates that a small-scale (135M parameter) open-source LLM can outperform much larger proprietary models (e.g., GPT-4) in financial trading tasks through effective RL fine-tuning.
• Validates the framework's ability to improve LLM performance not only in trading but also on other financial-domain tasks through shared representation learning.

Method

• Converts numerical market states (prices, indicators) and account status into structured text prompts.
• Uses a partially fine-tuned LLM with frozen base layers and trainable top layers as the shared backbone.
• Implements an actor-critic architecture where policy and value heads share trainable LLM parameters.
• Applies Proximal Policy Optimization (PPO) to update policy and value heads and trainable LLM layers.
• Uses Sharpe ratio of cumulative profits as the primary reward signal.
• Employs Generalized Advantage Estimation (GAE) for advantage calculation in PPO.

Results & findings

• FLAG-Trader consistently outperforms the buy-and-hold strategy and LLM-agentic baselines across multiple assets.
• The 135M-parameter SmolLM2 model achieved superior Sharpe Ratios and Cumulative Returns compared to GPT-4 and GPT-o1-preview on several stocks.
• For Bitcoin, FLAG-Trader achieved a Sharpe Ratio of 1.734, significantly higher than the baseline and many larger models.
• The framework demonstrates convergence to a stable optimal policy, becoming less sensitive to initial prompts over time.
• Results highlight that well-designed RL training strategies can bridge the performance gap typically associated with model scale.

Data & experiments

• Evaluates on five US stocks (MSFT, JNJ, UVV, HON, TSLA) and one cryptocurrency (Bitcoin).
• Compares against buy-and-hold strategy, LLM-agentic baseline (InvestorBench), and various proprietary (GPT-4, GPT-o1) and open-source models (Llama-3, Qwen).
• Uses warm-up periods (e.g., July-Sept 2020 for stocks) and test periods (e.g., Oct 2020-May 2021) to simulate trading conditions.
• Measures performance using Cumulative Return (CR), Sharpe Ratio (SR), Annualized Volatility (AV), and Maximum Drawdown (MDD).
• Employs Proximal Policy Optimization (PPO) with Generalized Advantage Estimation (GAE) for training.

Limitations & caveats

• Computationally expensive during fine-tuning, particularly for large-scale market datasets.
• Relies on structured prompts which may introduce biases in decision-making.
• Optimizes for Sharpe ratio without explicit risk-sensitive constraints, potentially ignoring tail risks.
• Challenges with long-term generalization due to non-stationary market conditions.
• Limited to single-asset trading in current experiments, not multi-asset portfolio optimization.

Curator notes

Discovery status: round2_promoted. Source links: A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist; FinBERT: A Pretrained Language Model for Financial Communications; FinBERT: Financial Sentiment Analysis with Pre-trained Language Models; InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning; LiveTradeBench: Seeking Real-World Alpha with Large Language Models; Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking; TradingGP