AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language Models
AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language Models
AlphaForgeBench addresses critical flaws in existing LLM-based financial trading benchmarks, which suffer from severe behavioral instability including extreme run-to-run variance, inconsistent action sequences under deterministic decoding, and irrational action flipping. The authors systematically attribute these issues to LLMs' stateless autoregressive architectures and sensitivity to continuous-to-discrete action mappings. To overcome these limitations, they propose a paradigm shift: instead of having LLMs emit direct trading actions, AlphaForgeBench tasks them with generating executable alpha factors and factor-based trading strategy code, effectively positioning the LLM as a quantitative researcher. This decouples reasoning from execution, enabling fully deterministic and reproducible evaluation while aligning with real-world quantitative research workflows.
The benchmark dataset is constructed in two stages: Stage 1 collects 633 real-world alpha factor-strategy pairs from diverse sources (brokerage reports, quantitative platforms, literature, open-source repositories), while Stage 2 uses LLMs to generate 270 additional queries under a 3×3 level–grade taxonomy (Levels: Logic Translation, Parameter Inference, Goal-Oriented Generation; Grades: Easy, Medium, Hard). The evaluation pipeline feeds each query to an LLM, which must produce a Python function generating trading signals; these are then executed in a deterministic backtest engine across 7 assets (2 cryptocurrencies, 5 US equities) over a 5-year period. Six standard financial metrics are computed over 5 independent runs per query.
Experiments on six frontier LLMs (claude-sonnet-4.5, deepseek-v3.2, gemini-3-flash-preview, gemini-3-pro-preview, gpt-5.2, grok-4.1-fast) demonstrate that AlphaForgeBench effectively eliminates execution-induced instability, yielding highly reproducible outcomes with intra-query variance an order of magnitude smaller than inter-query variance. The benchmark reveals a clear performance hierarchy: Gemini-3-pro-preview leads on return-oriented metrics, while deepseek-v3.2 favors conservative strategies. Model rankings are consistent across all assets, and the structured queries enable fine-grained diagnosis of model capabilities across difficulty levels. This work provides a rigorous, discriminative, and reproducible framework for assessing LLMs' financial reasoning and strategy formulation abilities.
Highlights
- 1Identifies severe behavioral instability in LLMs as direct trading agents, including extreme run-to-run variance and irrational action flipping.
- 2Introduces AlphaForgeBench, a benchmark that repositions LLMs from stochastic execution agents to quantitative researchers generating executable alpha factors and strategy code.
- 3Constructs a two-stage dataset with 633 real-world queries and 270 difficulty-specialized queries across a 3×3 level–grade taxonomy.
- 4Demonstrates that the code-generation paradigm yields deterministic, reproducible evaluation with intra-query variance an order of magnitude smaller than inter-query variance.
- 5Reveals distinct implicit risk preferences encoded by different LLMs, with consistent model rankings across cryptocurrency and US equity markets.
Methods
- MTwo-stage dataset construction: Stage 1 collects real-world alpha factors from diverse sources; Stage 2 uses LLMs to generate structured queries under a 3×3 level–grade taxonomy.
- MGenerate-and-backtest evaluation pipeline: LLMs produce Python code for trading signals, executed in a deterministic backtest engine across 7 assets over 5 years.
- MMulti-metric evaluation using Annual Rate of Return, Sharpe Ratio, Maximum Drawdown, Calmar Ratio, Sortino Ratio, and Volatility, reported as mean ± std over 5 runs.
- MTemperature ablation experiments at τ=0 and τ=0.7 to assess generation stability.
Results
- RAll six frontier LLMs achieve >96% backtest pass rates, validating code-generation reliability.
- RIntra-query variance is an order of magnitude smaller than inter-query variance, confirming stable and reproducible evaluation.
- RGemini-3-pro-preview leads on return-oriented metrics (ARR 0.171, SR 0.449), while deepseek-v3.2 favors conservative strategies (lowest MDD 0.114, highest CR 1.575).
- RModel rankings are consistent across all 7 assets, with trend-rich US equities (AAPL, GOOGL) being easiest and MSFT hardest.
- RStage 2 results show clear performance degradation from Level 1 (Logic Translation) to Level 3 (Goal-Oriented Generation), with Gemini-3-pro-preview consistently outperforming others.
Analyze Paper
Generate insights from "AlphaForgeBench: Benchmarking End-to-End Trading Strategy De...".