Can LLM-Based Financial Investing Strategies Outperform?

In one sentence

Summary

This paper addresses the lack of robust evaluation for LLM-based financial investing strategies, which often suffer from survivorship, look-ahead, and data-snooping biases due to narrow timeframes and selective stock choices. The authors propose FINSABER, a comprehensive backtesting framework designed to mitigate these biases by utilizing a 20-year history of S&P 500 constituents (including delisted stocks) and integrating multi-source unstructured data. The study critically assesses the generalizability of LLM agents by comparing them against traditional rule-based, machine learning, and reinforcement learning baselines under rigorous, bias-aware conditions. The experimental design employs a two-step pipeline: a selection-based module to generate unbiased stock universes and a timing-based module to execute daily trading signals. The authors evaluate prominent LLM agents, specifically FinMem and FinAgent, alongside benchmarks such as Buy-and-Hold, Moving Average Crossover, and RL agents (PPO, SAC). Experiments cover two decades (2004–2024) across multiple market regimes, using metrics like Sharpe Ratio, Maximum Drawdown, and Alpha/Beta decomposition. The study also includes statistical validation via paired t-tests and behavioral diagnostics through underwater plots to analyze drawdown profiles. Results indicate that previously reported LLM advantages vanish under broader, longer evaluations, with simple strategies like Buy-and-Hold often outperforming LLMs in risk-adjusted returns. LLM strategies exhibit regime-specific flaws: they are overly conservative in bull markets, missing gains, and overly aggressive in bear markets, incurring severe drawdowns. The agents fail to generate statistically significant alpha, suggesting their performance is driven by market beta rather than skill. The paper concludes that future LLM strategies must prioritize trend detection and adaptive risk management over architectural complexity, noting that high API costs further diminish their practical viability.

Research problem

Existing evaluations of LLM-based investing strategies are flawed by survivorship, look-ahead, and data-snooping biases, leading to overstated performance claims. There is a lack of comprehensive, long-term, and broad-universe backtesting frameworks that accurately assess the robustness and generalizability of LLM agents in real-world financial conditions.

Core contributions

• Proposes FINSABER, a bias-mitigating backtesting framework for LLM investing strategies.
• Demonstrates that LLM advantages reported in prior studies deteriorate significantly under longer, broader evaluations.
• Reveals that LLM strategies underperform passive benchmarks in bull markets and suffer heavy losses in bear markets.
• Provides empirical guidance that regime-aware risk control is more critical than model complexity for LLM investors.

Method

• FINSABER framework with multi-source data integration (news, filings, prices).
• Two-step pipeline: selection-based stock filtering followed by timing-based trading.
• Bias mitigation via historical constituent lists and rolling-window evaluations.
• Evaluation of FinMem (GPT-4o-mini/4o) and FinAgent (GPT-4o-mini).
• Comparison against Buy-and-Hold, SMA/WMA Crossover, ATR, Bollinger Bands.
• Comparison against RL agents (A2C, PPO, SAC, TD3) implemented via FinRL.
• CAPM regression for alpha and beta decomposition.
• Paired t-tests for statistical significance of performance differences.

Results & findings

• LLM strategies fail to outperform Buy-and-Hold in risk-adjusted metrics over long horizons.
• FinMem and FinAgent show high annual volatility and significant maximum drawdowns.
• Statistical tests confirm LLM underperformance is significant when biases are controlled.
• LLMs are overly conservative in bull markets, missing out on passive gains.
• LLMs are overly aggressive in bear markets, amplifying downside risk.
• FinAgent incurs significantly higher API costs than FinMem without superior performance.
• Simple rule-based strategies often match or exceed LLM performance in Sharpe ratios.

Data & experiments

• 20-year backtest period (2004–2024) using historical S&P 500 constituents including delisted stocks.
• Evaluation of FinMem and FinAgent against rule-based, ML, and RL baselines.
• Composite setup using unbiased stock selection methods (Random, Momentum, Volatility).
• Statistical validation via paired t-tests and CAPM-based alpha/beta decomposition.
• Behavioral analysis using underwater plots to visualize drawdown profiles across market regimes.

Limitations & caveats

• Evaluation focuses only on long-only positions, ignoring short-selling strategies.
• LLM performance may be affected by data leakage from pretraining on historical data.
• Results are specific to US equities (S&P 500) and may not generalize to other markets.
• Transaction costs are simplified and may not reflect real-world slippage or impact.
• The study does not explore fine-tuned LLMs, only off-the-shelf API models.
• High API costs for LLMs may limit practical deployment scalability.

Curator notes

Highlights bull-market and bear-market behavior issues.