InvestorBench: A Benchmark for Large Language Model Agents in Financial Decision-Making

In one sentence

Summary

The paper addresses the lack of standardized benchmarks for LLM-based agents in financial decision-making by introducing InvestorBench, a unified framework that extends the FINMEM architecture to handle diverse asset classes. The agent employs a layered memory system with varying decay rates to process multi-modal data, including OHLCV prices, SEC filings, and news sentiment, enabling sequential Buy/Sell/Hold decisions within a Partially Observable Markov Decision Process (POMDP) formulation. This approach aims to mimic human cognitive processes by integrating immediate observations with long-term historical insights to adapt to market volatility. The benchmark evaluates thirteen LLMs, ranging from small open-source models to large proprietary ones, across three distinct environments: single-stock trading (MSFT, TSLA, etc.), cryptocurrency trading (BTC, ETH), and ETF investing. Data sources include Yahoo Finance, SEC EDGAR, and specialized crypto/ETF news datasets. Experiments measure performance using Cumulative Return (CR), Sharpe Ratio (SR), Annualized Volatility, and Maximum Drawdown, comparing agent outputs against passive Buy & Hold baselines. The setup includes a warm-up phase for memory calibration and a test phase for evaluation, ensuring a rigorous assessment of reasoning and risk management capabilities. Results indicate that proprietary models like GPT-4 and GPT-o1-preview consistently achieve superior CR and SR compared to open-source and domain-specific fine-tuned models, particularly in volatile or complex market conditions. While large open-source models (e.g., Llama-3.1-70B) perform competitively in stable stock markets, they struggle in cryptocurrency and ETF tasks where deep reasoning and broad pre-training knowledge are critical. The study highlights that domain-specific fine-tuning alone does not guarantee trading superiority, suggesting that general reasoning capabilities and model scale are more decisive factors for agentic financial decision-making than specialized financial pre-training.

Research problem

The field lacks a comprehensive, standardized benchmark for evaluating LLM-based agents in diverse financial decision-making contexts. Existing frameworks are often niche-specific (e.g., only stocks or only crypto) and rely on proprietary data, making comparative analysis difficult. This paper addresses the need for a unified, open-source platform to assess the reasoning, memory, and sequential decision-making capabilities of LLM agents across multiple asset classes.

Core contributions

• Introduction of InvestorBench, the first benchmark specifically designed for evaluating LLM-based agents in diverse financial decision-making contexts.
• Development of a unified, flexible agent framework based on FINMEM that supports multi-modal data processing and layered memory for stocks, crypto, and ETFs.
• Creation of open-source, multi-source market environments mirroring real-world conditions, including data from Yahoo Finance, SEC EDGAR, and crypto news APIs.
• Comprehensive evaluation of 13 LLMs (proprietary, open-source, and domain-specific) to establish baselines for financial agentic performance.
• Provision of standardized evaluation metrics (CR, SR, AV, MDD) to facilitate robust comparison of agent performance across different market scenarios.

Method

• Agent architecture based on FINMEM with Brain, Perception, Profile, Memory, and Action modules.
• Layered Long-Term Memory with vector databases and varying decay rates for short, mid, and long-term information retention.
• Working Memory for immediate observation, summarization, and reflection on trading decisions.
• Multi-modal data integration: OHLCV prices, textual news, and financial reports processed into structured formats.
• Self-adaptive risk preference in the Profile module, adjusting risk aversion based on historical market momentum.
• POMDP formulation for sequential decision-making with reward function based on daily Profit & Loss (PnL).
• Evaluation using Cumulative Return, Sharpe Ratio, Annualized Volatility, and Maximum Drawdown.
• Deployment using vLLM with GPU scaling for different model sizes (RTX A6000 for small, A100 for large).

Results & findings

• Proprietary models (GPT-4, GPT-o1-preview) significantly outperform open-source and domain-specific models in stock trading, achieving higher CR and SR.
• Domain-specific fine-tuned models (e.g., Palmyra-Fin) did not provide a decisive advantage over general-purpose large models in sequential trading tasks.
• Large open-source models (>67B parameters) show improved robustness and performance compared to smaller models, supporting the scaling hypothesis.
• In cryptocurrency trading, only large-sized open-source and proprietary models could effectively capture signals; smaller models underperformed baselines.
• For ETF investing, proprietary models were essential for robust reasoning, outperforming open-source models due to the need for deep strategic comprehension.
• GPT-4o achieved the highest average CR in stock trading among tested models, demonstrating strong adaptability to mixed market conditions.
• Open-source models struggled in volatile environments, highlighting the importance of reasoning capabilities for handling noisy financial signals.

Data & experiments

• Stock market environment: OHLCV data from Yahoo Finance, 10-K/10-Q reports from SEC EDGAR, and news from Refinitiv/Zhou et al. for seven stocks (MSFT, TSLA, AAPL, etc.).
• Cryptocurrency environment: OHLCV from CoinMarketCap and news from cryptonews, cryptopotato, and cointelegraph for BTC and ETH.
• ETF environment: News-Informed Financial Trend Yield (NIFTY) dataset with curated headlines and sentiment for ETFs.
• Evaluation of 13 LLMs including GPT-4, GPT-o1-preview, Llama-3.1, Qwen2.5, and Palmyra-Fin across three distinct trading tasks.
• Metrics include Cumulative Return, Sharpe Ratio, Annualized Volatility, and Maximum Drawdown, compared against Buy & Hold baselines.
• Experiments conducted with a warm-up phase for memory calibration and a test phase for performance evaluation, repeated over five epochs.

Limitations & caveats

• Evaluation is limited to specific time periods and assets, which may not generalize to all market conditions.
• News sentiment is generated by GPT-3.5, which may introduce biases or inaccuracies.
• The study focuses on single-asset trading; portfolio management is not comprehensively evaluated.
• Open-source models may require further fine-tuning or prompt engineering to match proprietary performance.
• The benchmark relies on historical data, which may not fully capture future market dynamics or black swan events.
• Computational resources required for large models may limit accessibility for some researchers.

Curator notes

Important for measuring investment agents beyond static QA.