INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent

In one sentence

Summary

The paper addresses the lack of standardized benchmarks for evaluating LLM-based agents in diverse financial decision-making contexts. It introduces InvestorBench, a framework that extends the FINMEM architecture to support multi-asset trading (stocks, crypto, ETFs) using a layered memory system with varying decay rates to manage information sensitivity. The research positions this as a solution to the fragmentation of existing financial agent frameworks, which often focus on single asset classes or rely on proprietary data, thereby limiting comparative analysis. The methodology involves constructing three distinct market environments using open-source data sources like Yahoo Finance, SEC EDGAR, and CoinMarketCap, supplemented by news sentiment data. The authors evaluate 13 different LLMs, ranging from small open-source models to large proprietary ones, as backbones for the agent. Experiments are conducted across single-asset trading tasks and a multi-asset portfolio management task, using metrics such as Cumulative Return, Sharpe Ratio, Annualized Volatility, and Maximum Drawdown. The experimental design includes a warm-up phase for memory initialization and a test phase for performance evaluation. Key findings indicate that proprietary models (e.g., GPT-4, GPT-o1-preview) generally achieve superior risk-adjusted returns compared to open-source and domain-specific models, particularly in complex or volatile markets. Larger open-source models (e.g., Llama-3.1-70B, Qwen2.5-72B) outperform smaller counterparts, confirming that reasoning capability scales with parameter size. The study highlights that domain-specific fine-tuning does not guarantee better trading performance if the model lacks robust general reasoning. Limitations include the reliance on historical data which may not reflect future market dynamics and the potential for overfitting to specific market regimes, as evidenced by performance variations in bullish versus bearish conditions.

Research problem

The field lacks a comprehensive, open-source benchmark for evaluating LLM-based agents across diverse financial decision-making tasks. Existing frameworks are often siloed by asset class (e.g., only stocks or only crypto) or rely on proprietary data, preventing standardized comparison of agent reasoning and sequential decision-making capabilities in volatile, multi-modal financial environments.

Core contributions

• Introduction of InvestorBench, the first benchmark designed to evaluate LLM agents across stocks, cryptocurrencies, and ETFs.
• Development of a unified agent framework with a layered memory system that adapts to varying information decay rates.
• Creation of open-source, multi-modal market environments using data from Yahoo Finance, SEC EDGAR, and crypto news sources.
• Comprehensive evaluation of 13 LLMs, providing baseline performance metrics for both proprietary and open-source models in financial trading.

Method

• LLM-modulo agent architecture with Brain, Perception, Profile, Memory, and Action modules.
• Layered Long-Term Memory with vector databases and varying decay rates for information prioritization.
• Working Memory with immediate and extended reflection mechanisms.
• Dynamic risk preference adjustment based on historical market momentum.
• POMDP formulation for sequential decision-making with daily PnL rewards.
• Use of vLLM for efficient LLM deployment across different GPU configurations.
• Sentiment analysis of news data using GPT-3.5-turbo.

Results & findings

• Proprietary LLMs (GPT-4, GPT-o1-preview) consistently outperformed open-source and domain-specific models in stock trading.
• Larger open-source models (>67B parameters) showed significantly better performance and robustness than smaller models.
• Domain-specific models like Palmyra-Fin-70B did not provide a decisive advantage over general-purpose large models in trading tasks.
• Cryptocurrency trading required large models to capture noisy, sentiment-driven signals effectively.
• ETF investing favored proprietary models due to the need for deep strategic reasoning and long-term context.
• In extreme market conditions, proprietary models maintained better risk-adjusted returns than open-source alternatives.
• Buy-and-Hold strategy often outperformed agents in strongly bullish markets, highlighting the difficulty of active trading.

Data & experiments

• Stock environment: OHLCV data from Yahoo Finance, 10-K/10-Q reports from SEC EDGAR, and news from Reuters/Zhou et al. for 7 stocks (2020-2021).
• Cryptocurrency environment: OHLCV from CoinMarketCap and news from cryptonews/cryptopotato for BTC and ETH (2023).
• ETF environment: NIFTY dataset with news headlines and sentiment (2019-2020).
• Evaluation of 13 LLMs including GPT-4, GPT-o1-preview, Llama-3.1, Qwen2.5, and Palmyra-Fin.
• Metrics: Cumulative Return, Sharpe Ratio, Annualized Volatility, and Maximum Drawdown.
• Additional experiments on multi-asset portfolio management and extreme market conditions.

Limitations & caveats

• Performance may vary significantly across different market regimes (bullish vs. bearish).
• Reliance on historical data limits generalizability to unseen market conditions.
• Domain-specific models may be optimized for other tasks (e.g., report analysis) rather than trading.
• Open-source models may struggle with noisy signals in highly volatile markets like crypto.
• The benchmark focuses on single-asset and compact portfolio tasks, limiting scope for large-scale portfolio optimization.

Curator notes

Discovery status: round4_promoted. Source links: Multimodal Financial Foundation Models (MFFMs): Progress, Prospects, and Challenges; When Agents Trade: Live Multi-Market Trading Benchmark for LLM Agents