Signal or Noise in Multi-Agent LLM-based Stock Recommendations?

In one sentence

Summary

This paper addresses the critical gap in rigorous out-of-sample validation for multi-agent LLM equity systems, specifically testing whether the strong-buy recommendations from the deployed MarketSenseAI platform generate alpha beyond random stock selection. The research focuses on eliminating look-ahead bias by generating all signals live at each observation date, ensuring that the system's outputs are not influenced by future data or knowledge leakage from pre-training corpora. The study aims to quantify the economic value of these signals and decompose the internal reasoning process to understand the source of any observed edge. The methodology employs a fixed-cohort design on S&P 500 (19 months) and S&P 100 (35 months) universes, using a Monte Carlo null distribution of 10,000 random same-sized portfolios to test selection skill. The system uses four specialist agents (News, Fundamentals, Dynamics, Macro) whose outputs are synthesized into a thesis and ordinal recommendation. To interpret the model, the authors apply Non-Negative Least Squares (NNLS) to decompose thesis embeddings into agent contributions, validating the results with cosine diagnostics and Information Coefficient (IC) analysis restricted to the actionable buy/strong-buy universe. This approach isolates the predictive content of the continuous agent weights from the discrete recommendation label. Results show the S&P 500 strong-buy portfolio earns a +25.2% compound excess return over the equal-weight benchmark, ranking at the 99.7th percentile of the Monte Carlo null (p=0.003), with a statistically significant cross-sectional ICIR of +0.489. The S&P 100 cohort shows a similar directional excess (+30.5%) but lacks formal significance due to small position counts. NNLS attribution reveals an adaptive-integration mechanism where agent contributions rotate with market regimes: Fundamentals leads in the S&P 500, while Macro dominates in the S&P 100. The system exhibits downside protection, with strong-buy picks showing lower left-tail risk than hold signals. Limitations include the short time horizon, lack of sell-side implementation data, and the need for validation across more diverse market regimes and international universes.

Research problem

The paper investigates whether a deployed multi-agent LLM equity system (MarketSenseAI) generates stock selection signals that add value over passive benchmarks and random selection, and whether the internal structure of the agents reveals the source of this edge. It specifically addresses the scarcity of rigorous, out-of-sample portfolio-level validation for LLM-based trading systems, aiming to eliminate look-ahead bias and knowledge leakage through live signal generation.

Core contributions

• First portfolio-level Monte Carlo validation of a deployed multi-agent LLM equity system, testing strong-buy selections against random same-sized portfolios from the same universe.
• Novel application of Non-Negative Least Squares (NNLS) to decompose LLM thesis embeddings into agent contribution weights, revealing an adaptive-integration mechanism rather than a dominant-agent effect.
• Demonstration that agent contributions rotate with market regime (Fundamentals vs. Macro) and sector composition, providing interpretability into the source of alpha.
• Evidence that the discrete ordinal recommendation carries genuine predictive content (significant ICIR) and functions as an effective universe filter upstream of portfolio construction.

Method

• Multi-agent architecture: Four specialist agents (News, Fundamentals, Dynamics, Macro) generate independent text analyses.
• Synthesis agent: Combines agent outputs into a free-text equity thesis and a five-point ordinal recommendation.
• Embedding extraction: Thesis and agent texts encoded using OpenAI text-embedding-3-small (D=1536).
• NNLS Attribution: Solves non-negative least squares to decompose thesis embeddings into weighted sums of agent embeddings, normalizing weights to sum to one.
• Cosine Diagnostics: Validates NNLS weights against thesis-agent cosine similarities to ensure directional agreement while correcting for inter-agent collinearity.
• Monte Carlo Simulation: Generates 10,000 random portfolios per date to create a null distribution for testing selection skill.
• Information Coefficient: Computes Spearman rank correlation between signals and forward returns, with ICIR testing for statistical significance across dates.

Results & findings

• S&P 500 strong-buy portfolio earns +2.18%/month vs +1.15% benchmark, a +25.2% compound excess, ranking at the 99.7th percentile of Monte Carlo null (p=0.003).
• S&P 100 cohort shows +30.5% compound excess but lacks formal significance (p=0.17) due to small average selection size (~10 stocks/month).
• Cross-sectional ICIR for the ordinal recommendation is statistically significant on S&P 500 (ICIR=+0.489, p=0.024).
• NNLS attribution shows no single agent dominates; contributions rotate with regime (Fundamentals leads S&P 500, Macro leads S&P 100).
• Dynamics agent acts as an episodic momentum signal, leading on 5 of 19 S&P 500 dates despite negative aggregate pooled IC.
• Strong-buy signals exhibit downside protection, with lower left-tail risk (7.5% exceedance at -10% loss) compared to hold signals (10.2%).
• Portfolio beta is below unity (0.865), and excess returns are larger in down-market months, inconsistent with a pure risk-loading explanation.

Data & experiments

• Two fixed cohorts: S&P 500 (467 stocks, 19 months, Sep 2024–Mar 2026) and S&P 100 (94 stocks, 35 months, May 2023–Mar 2026).
• Live generation of signals at each first-Friday-of-the-month observation date to prevent look-ahead bias and knowledge leakage.
• Monte Carlo null distribution of 10,000 random equal-weight portfolios drawn without replacement from the same universe on the same dates.
• Information Coefficient (IC) and ICIR analysis restricted to the actionable buy and strong-buy universe to measure cross-sectional predictive power.
• Downside risk analysis using empirical CDFs and upside/downside return ratios for strong-buy vs. hold signals.

Limitations & caveats

• Short time horizon: S&P 500 cohort covers only 19 months, limiting statistical power and regime diversity.
• S&P 100 results lack formal significance due to small average position counts (~10 stocks/month), increasing variance.
• Sell-side signals are not implemented or validated due to lack of short interest, borrow availability, and liquidity data.
• Results may be specific to the recent market regime (Sep 2024–Mar 2026) and may not generalize to sustained bear markets.
• NNLS attribution assumes a linear relationship between agent embeddings and thesis embeddings, which may not capture all non-linear interactions.
• The system's edge may be sensitive to the specific LLM backbones and prompt engineering used, which are not fully detailed.

Curator notes

Discovery status: trading_agent_focus_promoted. Source links: ATLAS: Adaptive Trading with LLM AgentS Through Dynamic Prompt Optimization and Multi-Agent Coordination; From Earnings Calls to Investment Reports: Evaluating Role-based Multi-Agent LLM Systems; Toward Expert Investment Teams:A Multi-Agent LLM System with Fine-Grained Trading Tasks