The List / Agents and Multi-Agent Systems / paper
Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking
Abstract
Large Language Models (LLMs) have demonstrated notable capabilities across financial tasks, including financial report summarization, earnings call transcript analysis, and asset classification. However, their real-world effectiveness in managing complex fund investment remains inadequately assessed. A fundamental limitation of existing benchmarks for evaluating LLM-driven trading strategies is their reliance on historical back-testing, inadvertently enabling LLMs to"time travel"-leveraging future information embedded in their training corpora, thus resulting in possible information leakage and overly optimistic performance estimates. To address this issue, we introduce DeepFund, a live fund benchmark tool designed to rigorously evaluate LLM in real-time market conditions. Utilizing a multi-agent architecture, DeepFund connects directly with real-time stock market data-specifically data published after each model pretraining cutoff-to ensure fair and leakage-free evaluations. Empirical tests on nine flagship LLMs from leading global institutions across multiple investment dimensions-including ticker-level analysis, investment decision-making, portfolio management, and risk control-reveal significant practical challenges. Notably, even cutting-edge models such as DeepSeek-V3 and Claude-3.7-Sonnet incur net trading losses within DeepFund real-time evaluation environment, underscoring the present limitations of LLMs for active fund management. Our code is available at https://github.com/HKUSTDial/DeepFund.
In one sentence
Summary
The paper addresses the critical limitation of existing financial LLM benchmarks that rely on historical back-testing, which allows models to "time travel" by leveraging future information embedded in their training corpora, leading to inflated performance estimates. To solve this, the authors introduce DeepFund, a live fund investment benchmarking tool that evaluates LLMs in real-time market conditions, ensuring no information leakage from post-cutoff data. The framework positions LLMs as active fund managers rather than static analysts, requiring them to make dynamic trading decisions under uncertainty. DeepFund employs a multi-agent architecture where a single LLM backend assumes three roles: a Financial Planner that orchestrates tasks, an Analyst Team comprising specialized agents (Technical, Fundamental, Insider, Company News, Macro Economic, Policy) that generate directional signals, and a Portfolio Manager that synthesizes these signals to execute Buy/Sell/Hold decisions. The system connects to live market data via APIs (Yahoo Finance, Alpha Vantage) and evaluates nine flagship LLMs (e.g., GPT-4.1, DeepSeek-V3, Grok 3) over a 24-day period in March-April 2025, focusing on Berkshire Hathaway’s top five holdings. Performance is measured using standard financial metrics including Cumulative Return, Sharpe Ratio, and Maximum Drawdown. Empirical results reveal that most LLMs struggle in live trading, with only Grok 3 mini Beta achieving a positive cumulative return (+1.1%), while models like DeepSeek-V3 and Claude 3.7 Sonnet incurred significant losses. The study highlights distinct trading "personalities": Grok adopted a prudent, low-frequency strategy with high cash reserves and diversification, whereas DeepSeek exhibited high-frequency, momentum-driven behavior with low cash reserves, leading to vulnerability during market downturns. The findings underscore the current limitations of LLMs in active fund management and demonstrate the necessity of live benchmarking to assess true predictive power and risk control capabilities.
Research problem
Existing benchmarks for evaluating LLMs in finance rely on historical back-testing, which creates an information leakage problem where models may have seen the test data during pre-training. This "time travel" effect leads to overly optimistic performance estimates that do not reflect true predictive power in real-time markets. The paper seeks to rigorously assess LLMs' ability to manage complex fund investments under live, leakage-free conditions.
Core contributions
- Introduction of DeepFund, a live fund benchmarking tool that prevents information leakage by using real-time data published after model pretraining cutoffs.
- Development of a multi-agent architecture that mimics human fund management processes, assigning LLMs to roles of Financial Planner, Analyst Team, and Portfolio Manager.
- Empirical evaluation of nine state-of-the-art LLMs in a live market environment, revealing significant performance disparities and practical challenges in active fund management.
- Detailed analysis of LLM trading behaviors and "personalities," contrasting prudent strategies (Grok) with aggressive, loss-incurring strategies (DeepSeek).
- Provision of an open-source framework and web-based interface for continuous, reproducible evaluation of LLMs in financial trading.
Method
- Multi-agent orchestrator-worker paradigm using LangChain to coordinate LLM-based agents.
- Financial Planner agent dynamically selects relevant Analyst Team members based on market conditions.
- Specialized Analyst agents (Technical, Fundamental, Insider, Company News, Macro, Policy) generate Bullish/Bearish/Neutral signals with justifications.
- Portfolio Manager agent synthesizes signals, manages risk control (position sizing), and executes Buy/Sell/Hold decisions.
- Dual-memory architecture: short-term operational memory (FundState) and long-term historical memory for context and learning.
- Real-time data ingestion via modular API gateway connecting to financial data providers.
- Evaluation metrics calculated daily, including Cumulative Return, Sharpe Ratio, and Maximum Drawdown.
Results & findings
- Only Grok 3 mini Beta achieved a positive cumulative return (+1.1%) among the nine evaluated models.
- DeepSeek-V3 incurred the largest drawdown (-14.5%) and a cumulative return of -5.7%, highlighting the risks of aggressive trading.
- Most models, including GPT-4.1 (-5.9%) and Claude 3.7 Sonnet (-3.7%), incurred net trading losses, outperformed only by a passive Buy & Hold strategy in some cases.
- Grok demonstrated superior risk management with a maximum drawdown of 5.5% and a win rate of 61%, while maintaining high cash reserves.
- DeepSeek adopted a high-frequency strategy with low cash reserves (<40%), leading to inability to mitigate losses during the tariff-driven market downturn.
- Signal validity was high (96% for signals, 98% for decisions), indicating the framework reliably generates actionable outputs.
- Grok showed better integration of policy and technical signals, while DeepSeek leaned heavily on Neutral signals and company news.
Data & experiments
- Live market data from Yahoo Finance and Alpha Vantage, including stock prices, financial statements, insider transactions, and macroeconomic indicators.
- Evaluation period of 24 trading days (March 17 to April 17, 2025), covering significant events like the FOMC meeting and US tariff announcements.
- Test universe consisting of Berkshire Hathaway’s top five holdings: Apple (AAPL), American Express (AXP), Bank of America (BAC), Coca-Cola (KO), and Chevron (CVX).
- Nine flagship LLMs evaluated: GPT-4.1, Llama 4 Scout, Gemini 2.5 Flash, Claude 3.7 Sonnet, Grok 3 mini Beta, DeepSeek-V3, Qwen2.5-Max, Doubao-1.5-pro, and GLM-4-Air.
- Metrics include Cumulative Return, Sharpe Ratio, Maximum Drawdown, Win Rate, Beta, and Alpha, compared against a Buy & Hold baseline and S&P 500.
Limitations & caveats
- The evaluation period was short (24 days) and occurred during a volatile market, potentially skewing results.
- The framework simplifies trading context by ignoring transaction fees, market restrictions, and hybrid strategies.
- The study is limited to the US stock market and a specific set of large-cap stocks.
- LLM inference costs and latency are not fully optimized, though the framework is designed for reproducibility.
- The 'Buy & Hold' baseline may not be a fair comparison for active trading strategies in all market conditions.
- The analysis of 'trading personalities' is qualitative and based on a small sample of models.
Curator notes
Discovery status: round1_promoted_seed_for_round2. Source links: A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist; Assessing Look-Ahead Bias in Stock Return Predictions Generated by GPT Sentiment Analysis; FLAG-Trader: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Trading; FinBen: A Holistic Financial Benchmark for Large Language Models; FinMem: A Performance-Enhanced Large Language Model Trading Agent with Layered Memory and Character Design; FinNLI: Novel Dataset for Multi-Genre Fin