PolyBench is a novel benchmark designed to evaluate large language models (LLMs) as trading agents on live decentralized prediction markets, specifically Polymarket. It comprises 38,666 binary market snapshots from 4,997 events, each synchronized with a Central Limit Order Book (CLOB) state and a real-time news stream, collected between February 6–12, 2026. The benchmark is contamination-proof because outcomes are unknown at evaluation time. Seven state-of-the-art LLMs (including open- and closed-source models) were assessed using a multidimensional framework that includes directional accuracy, Confidence-Weighted Return (CWR), Annualized Percentage Yield (APY), Sharpe ratio, and instruction adherence, with realistic order-book execution simulation.

The results reveal a stark performance divergence: only MiMo-V2-Flash and Gemini-3-Flash achieved positive financial returns (17.6% and 6.2% CWR, respectively), while the other five models incurred losses despite uniformly high stated confidence. Key findings include that LLMs exhibit implicit meta-cognition by assigning higher confidence to correct predictions, but domain competence is highly uneven—strong in text-rich domains like Politics but poor in speculative sectors like Crypto, where overconfidence leads to severe losses. Instruction adherence is necessary but not sufficient for profitability; flawless execution does not guarantee alpha. Additionally, order-book slippage erodes returns at larger lot sizes, bounding theoretical predictive advantages by available market depth. PolyBench establishes a financially-grounded, contamination-proof evaluation standard for future LLM research in forecasting and trading.