MM-DREX: Multimodal-Driven Dynamic Routing of LLM Experts for Financial Trading

In one sentence

Summary

The paper addresses the challenge of non-stationary financial markets where traditional quantitative models and static LLM agents fail to adapt to regime shifts. MM-DREX proposes a decoupled architecture that separates market state perception from strategy execution. It employs a Vision-Language Model (VLM) to jointly analyze candlestick chart patterns and temporal price features, enabling the system to detect complex visual signals like head-and-shoulders formations alongside numerical trends. This multimodal input allows for a more robust understanding of market conditions than unimodal approaches, addressing the limitation of existing LLM trading systems that lack fine-grained, adaptive decision mechanisms. The core method involves a dynamic router trained via a hybrid Supervised Fine-Tuning and Reinforcement Learning (SFT-RL) paradigm. The router allocates weights to four heterogeneous experts: Trend, Reversal, Breakout, and Positioning. Each expert implements specific technical strategies (e.g., MACross, RSI reversal, Turtle breakout). The system is evaluated on a novel multimodal dataset spanning US equities, Chinese A-shares, ETFs, futures, and cryptocurrencies, covering bull, bear, and crisis periods. Experiments compare MM-DREX against 15 baselines, including deep reinforcement learning models (PPO, SAC) and other LLM agents (FinAgent, FinMem), using metrics like total return, Sharpe ratio, and maximum drawdown. Results show MM-DREX significantly outperforms baselines, achieving a 47.5% return in US equities and a Sharpe ratio of 1.83, surpassing the best RL and LLM competitors. Ablation studies confirm that visual modalities provide critical incremental value, with performance degrading substantially when candlestick data is removed. The model demonstrates robustness during black-swan events like the 2020 COVID crash and 2022 rate hikes, maintaining lower drawdowns than the S&P 500. Limitations include the computational cost of the 72B VLM backbone and the need for future work on real-time latency, transaction costs, and meta-learning for expert evolution to handle extreme long-horizon dependencies.

Research problem

Existing quantitative trading models and LLM-based agents struggle with non-stationary markets due to static strategies and unimodal data reliance. Traditional methods fail to adapt to regime shifts, while current LLM solutions lack dynamic adjustment mechanisms and fine-grained decision capabilities, often resulting in homogeneous expert designs and poor performance during market volatility or structural breaks.

Core contributions

• Proposes MM-DREX, a multimodal LLM framework decoupling market perception from strategy execution.
• Introduces a VLM-powered dynamic router that fuses candlestick visual patterns with temporal features.
• Designs four heterogeneous trading experts (Trend, Reversal, Breakout, Position) with specialized action spaces.
• Develops a hybrid SFT-RL training paradigm to optimize router classification and expert policy simultaneously.
• Constructs a comprehensive multimodal dataset across five asset classes for LLM trading benchmarking.
• Provides an interpretability module for real-time auditing of routing logic and expert behavior.

Method

• Vision-Language Model (Qwen2.5-VL-72B) as the shared backbone for multimodal encoding.
• Dynamic router uses LoRA adapters to output softmax weights for four experts based on fused features.
• Four experts implement specific technical strategies: MACross, RSI, Turtle breakout, etc.
• SFT stage trains router on ternary market trend classification (Up/Down/Consolidation).
• RL stage optimizes expert policies using PPO with a reward function combining return, Sharpe, and drawdown.
• Group reward encourages expert differentiation to avoid strategy convergence.
• Input features include resized candlestick charts, normalized time-series indicators, and textual summaries.

Results & findings

• MM-DREX achieved 47.5% total return in US equities, outperforming FinAgent (39.34%) and PPO (29.34%).
• Sharpe ratio of 1.83 in US equities and 2.15 in A-shares, exceeding all baselines.
• Dynamic routing significantly reduced maximum drawdown compared to uniform or best-expert-only strategies.
• Removing visual modalities caused a 21.9% drop in total return and 25.8% drop in Sharpe ratio.
• Outperformed S&P 500 during six black-swan events, including COVID-19 and 2022 rate hikes.
• LLM baselines like FinAgent failed in futures and crypto, highlighting MM-DREX's cross-market adaptability.
• Hybrid SFT-RL training accelerated convergence and improved router market classification accuracy.

Data & experiments

• Multimodal dataset covering US stocks, Chinese A-shares, ETFs, futures, and crypto (62 assets).
• Includes OHLCV data, technical indicators, candlestick charts, and textual market summaries.
• Temporal split: 60% training, 20% validation, 20% testing across all markets.
• Compared against 15 baselines: traditional indicators, ML (LGBM, LSTM), RL (PPO, SAC), and LLMs (FinAgent).
• Evaluated on Total Return, Sharpe Ratio, and Maximum Drawdown across diverse market regimes.
• Ablation studies on routing mechanisms (dynamic vs. uniform vs. best-expert) and modalities (visual vs. temporal).

Limitations & caveats

• High computational cost due to the 72B parameter VLM backbone.
• Experiments do not explicitly model transaction costs or liquidity constraints in the main results.
• Expert strategies are fixed archetypes; future work is needed for meta-learning expert evolution.
• Reliance on historical data may not capture unprecedented structural breaks or novel market dynamics.
• Latency and real-time deployment challenges are acknowledged but not addressed in current experiments.
• Interpretability module provides audit trails but does not guarantee causal correctness of routing decisions.

Curator notes

Discovery status: institutional_trading_focus_promoted. Source links: LLM option trading financial markets; large language models execution trading