PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data

In one sentence

Summary

The paper addresses the critical challenge of evaluating Large Language Models (LLMs) as autonomous forecasting agents in live, financially incentivized environments. Existing benchmarks often suffer from data contamination or lack real-time market grounding, failing to capture the multimodal complexity of fusing qualitative news with quantitative order-book dynamics. PolyBench introduces a contamination-proof benchmark derived from Polymarket, comprising 38,666 binary prediction markets across 4,997 events, synchronized with Central Limit Order Book (CLOB) states and real-time news streams to ensure strict temporal discipline and zero-shot evaluation. The methodology involves a four-stage pipeline: collecting market data via the Polymarket Gamma API, fetching multimodal context (news and CLOB snapshots), batch processing predictions through seven state-of-the-art LLMs, and matching outcomes against ground truth. The evaluation framework assesses directional accuracy, Confidence-Weighted Return (CWR), Annualized Percentage Yield (APY), and Sharpe ratio via realistic order-book execution simulation. Models are constrained to issue BUY decisions only when confidence exceeds 0.6, with capital allocation scaled by stated confidence to penalize miscalibration. The study analyzes performance across varying lot sizes to account for slippage and liquidity constraints. Results indicate a pronounced divergence: only MiMo-V2-Flash (17.6% CWR) and Gemini-3-Flash (6.2% CWR) achieve positive returns, while five models incur losses despite high stated confidence. Key findings include LLMs' implicit meta-cognition (higher confidence correlates with accuracy), severe miscalibration in volatile domains like Crypto, and the erosion of alpha due to order-book slippage at larger position sizes. The paper concludes that instruction adherence is necessary but insufficient for profitability, establishing PolyBench as a financially grounded standard for future LLM research.

Research problem

The paper addresses the inability of existing LLM benchmarks to evaluate forecasting and trading capabilities in live, financially incentivized environments. It highlights the gap between static text-based evaluation and the multimodal, temporal, and liquidity-constrained reality of prediction markets. The core problem is determining whether LLMs can genuinely reason probabilistically under uncertainty and execute profitable trades without data contamination.

Core contributions

• Introduction of PolyBench, a large-scale, contamination-proof benchmark using live Polymarket data.
• Development of a multimodal evaluation framework coupling CLOB states with news streams.
• Proposal of Confidence-Weighted Return (CWR) to assess risk calibration and financial viability.
• Empirical evaluation of seven state-of-the-art LLMs on 38,666 binary prediction markets.
• Analysis of order-book slippage and liquidity constraints on LLM trading performance.

Method

• Data Collection: Polymarket Gamma API for market metadata and CLOB states.
• Context Aggregation: Synchronous fetching of Google News and order book snapshots.
• Prompt Engineering: Strict Bayesian baseline with timestamp locking and resolution rule primacy.
• Decision Logic: BUY only if confidence > 0.6; capital scaled by confidence score.
• Execution Simulation: Absorbing liquidity from best ask to model slippage and depth limits.
• Evaluation Metrics: CWR, APY, Sharpe ratio, and directional accuracy on resolved markets.
• Domain Analysis: Stratified performance evaluation across eight event categories.

Results & findings

• Only 2 of 7 LLMs achieved positive financial returns (MiMo-V2-Flash: 17.6% CWR; Gemini-3-Flash: 6.2% CWR).
• Five models incurred losses despite uniformly high stated confidence, indicating miscalibration.
• LLMs exhibit implicit meta-cognition, assigning higher confidence to correct predictions.
• Performance varies by domain: strong in Politics, poor in Crypto due to volatility and miscalibration.
• Order-book slippage significantly erodes returns as position sizes increase from $10 to $1,000.
• Instruction adherence (e.g., no hallucinations) is necessary but not sufficient for profitability.
• MiMo-V2-Flash outperformed proprietary models in net profitability despite being open-source.

Data & experiments

• Dataset: 38,666 binary markets from Polymarket spanning 4,997 events (Feb 6-12, 2026).
• Multimodal inputs: Central Limit Order Book (CLOB) snapshots and Google News streams.
• Models: Seven LLMs including MiMo-V2-Flash, Gemini-3-Flash, GPT-OSS-120B, and Grok-4.1-Fast.
• Metrics: Directional accuracy, CWR, APY, Sharpe ratio, and temporal resolution error.
• Simulation: Realistic order-book execution with slippage modeling and confidence-weighted capital allocation.

Limitations & caveats

• Evaluation limited to a single week (Feb 6-12, 2026), potentially missing long-term market dynamics.
• Focus on binary prediction markets may not generalize to continuous asset trading.
• Slippage modeling relies on captured order book depth, which may not reflect full market liquidity.
• News context is limited to Google News, potentially missing other relevant information sources.
• Models evaluated are API-based; local deployment costs and latency are not fully analyzed.
• Results may vary with different lot sizes and execution algorithms beyond the simulated sweep.

Curator notes

Discovery status: institutional_trading_focus_promoted. Source links: large language models order book trading; LLM limit order book trading; large language models execution trading