Will LLMs be Professional at Fund Investment? DeepFund: A Live Arena Perspective

In one sentence

Summary

The paper addresses the inadequacy of current LLM financial benchmarks, which primarily test document understanding rather than dynamic trading capabilities. The authors identify four systemic failures in existing evaluation methods: data leakage due to pre-training contamination on historical data, navel-gazing where models excel at theory but fail in practice, over-intervention via heavy prompt engineering, and maintenance difficulties due to the need for continuous data updates. This creates an unfair and misleading assessment landscape where models may appear proficient by recalling memorized narratives rather than demonstrating genuine predictive reasoning or adaptive decision-making in live market conditions. To resolve these issues, the authors propose DeepFund, a comprehensive live arena platform featuring a three-phase workflow. The system employs a multi-agent framework comprising an Agent Planner, specialized Analysts (Technical, Fundamental, Insider, and Media), and an Agent Manager that synthesizes insights into buy, hold, or sell decisions. The platform operates in a time-controlled live environment, providing day-by-day market data, news, and financial information to ensure models only access information available at the decision point, thereby eliminating data leakage. It includes a modular model integration interface for various LLMs and a web-based visualization tool for monitoring performance metrics across diverse market regimes and investment horizons. DeepFund aims to provide a realistic, fair, and standardized assessment of LLM capabilities in fund investment by mimicking real-world investment processes without human interference. The platform supports diverse portfolio management across multiple asset classes and evaluates performance through traditional financial metrics alongside LLM-specific criteria. By enabling forward testing and continuous updates, DeepFund bridges the gap between theoretical financial knowledge and practical application. The code is publicly available, encouraging community extension with new signal analyses and data sources, ultimately contributing to the development of more reliable AI-based financial decision-making tools.

Research problem

Current benchmarks for evaluating LLMs in finance fail to assess their ability to manage assets or identify trading opportunities in dynamic markets. Existing methods suffer from data leakage (models trained on historical data tested on the same data), navel-gazing (focus on theory over practice), over-intervention (reliance on human prompt engineering), and maintenance hardness (inability to update with fresh data).

Core contributions

• Identification of four critical flaws in existing LLM financial benchmarks: data leakage, navel-gazing, over-intervention, and maintenance-hard.
• Development of DeepFund, a live arena platform for evaluating LLM-based trading strategies in a real-time environment.
• Implementation of a multi-agent framework with specialized roles (Planner, Analysts, Manager) to mimic real-world investment processes.
• Creation of a time-controlled live environment that prevents data leakage by providing chronological, up-to-date market information.
• Provision of a web interface for visualizing and comparing LLM performance across different market conditions and investment metrics.

Method

• Three-phase workflow: initial data preparation, interim multi-agent analysis, and final decision making.
• Live Trading Environment providing chronological, up-to-date market data to prevent data leakage.
• Multi-Agent Framework with Agent Planner, specialized Analysts, and Agent Manager.
• Model Integration Interface for seamless integration of different foundation LLMs.
• Performance Monitoring and Visualization module tracking decision logs and financial metrics.
• Standardized evaluation metrics designed for assessing financial decision-making capabilities.

Results & findings

• Existing benchmarks suffer from data leakage, navel-gazing, over-intervention, and maintenance-hard issues.
• DeepFund implements a multi-agent framework to realize real-world investment decision processes.
• The platform provides a web interface for visualizing LLM performance with fund investment metrics.
• DeepFund aims to provide a more realistic and fair assessment of LLM capabilities in fund investment.
• The code is publicly available at https://github.com/HKUSTDial/DeepFund.

Data & experiments

• Live trading environment providing day-by-day market data, news, and financial information.
• Multi-agent framework with Technical, Fundamental, Insider, and Media analyst roles.
• Evaluation of LLMs on fund investment tasks including buy, hold, and sell decisions.
• Performance monitoring using traditional financial metrics and LLM-specific evaluation criteria.
• Web-based visualization for comparative analysis across different market conditions.

Limitations & caveats

• The platform is still under development, and the web interface is not final.
• The effectiveness of the multi-agent framework depends on the quality of the underlying LLMs.
• The live environment requires continuous data updates, which may be resource-intensive.
• The platform's ability to generalize across different market regimes is not fully validated.
• The evaluation metrics may not capture all aspects of real-world investment performance.

Curator notes

Discovery status: round1_promoted_seed_for_round2. Source links: A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist; BloombergGPT: A Large Language Model for Finance; FinBen: A Holistic Financial Benchmark for Large Language Models; FinGPT: Open-Source Financial Large Language Models; FinMem: A Performance-Enhanced Large Language Model Trading Agent with Layered Memory and Character Design; FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking; FinRobot: An Open-Source AI Agent Pl