The List / Trading, Investment, and Portfolio Management / paper
Your AI, Not Your View: The Bias of Large Language Models in Investment Analysis
Abstract
In finance, Large Language Models (LLMs) face frequent knowledge conflicts arising from discrepancies between their pre-trained parametric knowledge and real-time market data. These conflicts are especially problematic in real-world investment services, where a model’s inherent biases can misalign with institutional objectives, leading to unreliable recommendations. Despite this risk, the intrinsic investment biases of LLMs remain underexplored. We propose an experimental framework to investigate emergent behaviors in such conflict scenarios, offering a quantitative analysis of bias in LLM-based investment analysis. Using hypothetical scenarios with balanced and imbalanced arguments, we extract the latent biases of models and measure their persistence. Our analysis, centered on sector, size, and momentum, reveals distinct, model-specific biases. Across most models, a tendency to prefer technology stocks, large-cap stocks, and contrarian strategies is observed. These foundational biases often escalate into confirmation bias, causing models to cling to initial judgments even when faced with increasing counter-evidence. A public leaderboard benchmarking bias across a broader set of models is available at https://linqalpha.com/leaderboard.
In one sentence
Summary
The paper addresses the critical problem of knowledge conflict in financial LLMs, where pre-trained parametric biases clash with real-time market data, leading to unreliable investment recommendations. The authors argue that existing evaluations fail to capture how LLMs handle conflicting signals, a common scenario in active portfolio management and research. They position their work as a necessary diagnostic tool for trustworthiness, aiming to quantify how internal priors override objective evidence in decision-making processes. The methodology employs a three-stage framework using 427 S&P 500 stocks. First, balanced buy/sell evidence with uniform 5% intensity is generated by a neutral model to create knowledge conflicts. Second, bias is elicited by measuring decision preferences in balanced contexts, revealing sector, size, and momentum biases. Third, bias persistence is verified by introducing imbalanced counter-evidence in volume and intensity. The study evaluates six major LLMs, including GPT-4.1 and Llama4-Scout, using metrics like bias scores and decision flip rates, supported by statistical tests and entropy analysis of model uncertainty. Results show consistent biases toward technology sectors, large-cap stocks, and contrarian strategies. Models exhibit strong confirmation bias, with flip rates dropping sharply when even a small amount of supporting evidence is present, despite a majority of counter-evidence. For instance, models with high initial bias struggled to reverse decisions even with 10% intensity increments in counter-evidence. The study highlights that these biases can lead to systematic overvaluation of popular assets and underperformance in momentum strategies. Limitations include the use of synthetic evidence and a focus on hypothetical scenarios rather than live trading, though the findings provide a crucial baseline for auditing financial AI systems.
Research problem
The paper investigates the intrinsic investment biases of Large Language Models (LLMs) when faced with knowledge conflicts between their parametric training data and external market signals. It specifically asks what biases exist toward financial factors like sector, size, and momentum, and how these biases manifest as confirmation bias when models encounter conflicting evidence. The core issue is that LLMs may override institutional objectives with their own ingrained preferences, leading to unreliable and distorted investment recommendations.
Core contributions
- Proposes a systematic three-stage experimental framework to elicit, quantify, and verify latent biases in LLMs for investment analysis.
- Provides quantitative evidence of distinct, model-specific biases toward technology sectors, large-cap stocks, and contrarian investment strategies.
- Demonstrates that LLMs exhibit strong confirmation bias, stubbornly adhering to initial judgments even when faced with overwhelming counter-evidence.
- Establishes a public leaderboard benchmark for comparing bias profiles across multiple LLMs, facilitating transparency in financial AI deployment.
Method
- Evidence Generation: Uses a neutral LLM to create balanced buy/sell arguments with fixed 5% intensity for each stock.
- Bias Elicitation: Presents balanced evidence to LLMs and calculates bias scores based on the frequency of buy vs. sell decisions.
- Volume Verification: Tests decision flip rates by varying the ratio of supporting to counter-evidence (e.g., 1:2, 2:3).
- Intensity Verification: Tests decision flip rates by increasing the intensity of counter-evidence while maintaining volumetric parity.
- Entropy Analysis: Computes Shannon entropy from action token probabilities to measure model uncertainty during biased decisions.
- Statistical Testing: Applies t-tests and Chi-Square tests to validate the significance of sector, size, and momentum biases.
Results & findings
- Most models exhibit a strong positive bias toward the Technology sector, with statistically significant gaps between highest and lowest favored sectors.
- A consistent size bias is observed, with models preferring large-cap stocks (Q1) and showing negative bias toward small-cap stocks (Q3/Q4).
- All evaluated models show a statistically significant preference for contrarian strategies over momentum strategies.
- Decision flip rates drop sharply when supporting evidence is introduced, even if counter-evidence is volumetrically superior (e.g., 2 vs 3).
- Models with higher initial bias scores show lower adaptability, struggling to reverse decisions even with 10% intensity increments in counter-evidence.
- Entropy analysis reveals a correlation between bias strength and internal uncertainty, with biased models showing lower flexibility.
Data & experiments
- Curated dataset of 427 prominent S&P 500 stocks with continuous listing over the past five years to ensure robust parametric knowledge.
- Synthetic evidence generation using Gemini-2.5-Pro to create balanced buy/sell arguments with uniform 5% price impact intensity.
- Three-stage experiment: bias elicitation in balanced contexts, verification via volumetric imbalance, and verification via intensity imbalance.
- Evaluation of six LLMs (GPT-4.1, Llama4-Scout, DeepSeek-V3, etc.) using bias scores, decision flip rates, and Shannon entropy metrics.
- Statistical validation using independent-samples t-tests and Chi-Square tests to confirm the significance of observed biases.
Limitations & caveats
- Uses synthetic evidence generated by LLMs, which may not fully capture the complexity and nuance of real-world market data.
- Focuses on hypothetical scenarios rather than live trading or historical backtesting, limiting external validity.
- The study is confined to 427 S&P 500 stocks, potentially missing biases in less liquid or smaller markets.
- Does not propose specific mitigation strategies, only diagnosing the problem of bias and confirmation.
- The use of a single neutral model for evidence generation may introduce its own subtle biases.
Curator notes
Relevant for governance and investment-style bias auditing.