LiveTradeBench: Seeking Real-World Alpha with Large Language Models
LiveTradeBench: Seeking Real-World Alpha with Large Language Models
LiveTradeBench introduces a live trading environment for evaluating LLM agents in realistic, evolving markets, addressing limitations of static benchmarks and offline backtesting. The environment streams real-time market prices and news, uses a portfolio management abstraction where agents output continuous allocation vectors across multiple assets, and evaluates agents across two structurally distinct markets: U.S. stocks and Polymarket prediction markets. Over a 50-day live evaluation of 21 LLMs, the paper finds that high performance on general LLM benchmarks (e.g., LMArena) does not translate to superior trading outcomes, with some top models performing poorly in dynamic markets. Agents display distinct portfolio styles, and reasoning-focused models tend to over-adjust, leading to high volatility and poor risk-adjusted returns. The rolling k-delta analysis confirms that agents are not random guessers but genuinely adapt to live signals, as delaying decisions systematically degrades performance. These findings highlight the need for benchmarks that test sequential decision-making under uncertainty, motivating the development of more adaptive and robust portfolio management agents.
Highlights
- 1Introduces LiveTradeBench, a live trading environment for evaluating LLM agents with real-time data streaming, portfolio management abstraction, and multi-market evaluation across U.S. stocks and Polymarket prediction markets.
- 2Demonstrates that high LMArena scores do not correlate with superior trading performance, revealing a gap between static benchmark competence and real-world financial decision-making.
- 3Finds that LLMs exhibit distinct portfolio management styles (e.g., conservative vs. risk-seeking) and that reasoning-focused models (e.g., DeepSeek-R1, GPT-o3) underperform due to over-adjustment and high volatility.
- 4Provides evidence that LLM agents are not random guessers: delaying decisions (rolling k-delta analysis) systematically degrades performance, confirming genuine market adaptation.
- 5Offers a comprehensive 50-day live evaluation of 21 LLMs across multiple families, with detailed metrics including cumulative return, Sharpe ratio, maximum drawdown, win rate, and volatility.
Methods
- MLive data streaming of market prices and news to avoid information leakage and capture real-time uncertainty.
- MPortfolio management abstraction: agents output continuous allocation vectors across multiple assets, integrating risk management and cross-asset reasoning.
- MRolling k-delta analysis to test whether agents exhibit genuine market awareness by comparing performance with delayed decisions.
- MReAct-style agent framework with tool use, memory, and reasoning for decision-making.
Results
- RTrading performance on one market (e.g., U.S. stocks) does not generalize to another (Polymarket), with near-zero Sharpe ratio correlation between markets.
- RHigh LMArena scores show slightly negative correlation with trading Sharpe ratios (Spearman ρ = 0.054 for stocks, -0.38 for Polymarket), indicating general LLM ability does not imply financial competence.
- RReasoning-focused models (DeepSeek-R1, Qwen3-Thinking, GPT-o3) exhibit high volatility (>140 in Polymarket) and underperform, suggesting excessive deliberation harms trading consistency.
- RGPT-4.1 achieves the highest stock market cumulative return (6.25%) but performs poorly in Polymarket (< -30%), highlighting market-specific strategy needs.
- RRolling k-delta analysis shows that delaying decisions by even 1 day reduces average Sharpe ratio, confirming agents adapt to live signals rather than guessing randomly.
Analyze Paper
Generate insights from "LiveTradeBench: Seeking Real-World Alpha with Large Language...".