Advanced Statistical Arbitrage with Reinforcement Learning
Advanced Statistical Arbitrage with Reinforcement Learning
This paper presents a novel model-free framework for statistical arbitrage that combines an empirical mean reversion time (EMRT) metric for spread construction with a reinforcement learning (RL) approach for trading. The EMRT measures the average time for a spread to revert to its mean after a local extreme, and is minimized via grid search to determine optimal asset weights, avoiding reliance on parametric models like Ornstein-Uhlenbeck. The RL component uses Q-learning with a state space capturing recent price trends (direction and magnitude) and a reward function that rewards buying below the mean and selling above, accounting for transaction costs. The agent is trained on simulated OU paths and then applied to real market data.
Experiments on simulated OU processes show that EMRT decreases with higher mean reversion speed, validating its effectiveness. The RL agent achieves over 600% average profit on simulated test paths. In real-world tests on 10 S&P 500 sector pairs (formation 2022, trading 2023), the proposed method consistently outperforms the distance method and OU-based trading in terms of daily Sharpe ratio and cumulative returns, demonstrating robustness and adaptability across different market conditions. The framework eliminates the need for parameter stability assumptions and hyper-parameter tuning, offering a flexible and data-driven alternative for statistical arbitrage.
Highlights
- 1Introduces a model-free framework for statistical arbitrage using empirical mean reversion time (EMRT) to construct mean-reverting spreads without relying on stochastic process assumptions.
- 2Develops a reinforcement learning (RL) trading strategy that dynamically optimizes decisions based on recent price trends, eliminating the need for fixed thresholds or parameter stability assumptions.
- 3Demonstrates superior performance over traditional distance method and Ornstein-Uhlenbeck strategies on S&P 500 pairs, with higher Sharpe ratios and cumulative returns.
Methods
- MEmpirical Mean Reversion Time (EMRT): A metric defined as the average time from local extremes to mean crossings, used to optimize asset coefficients via grid search.
- MReinforcement Learning (Q-learning): State space encodes recent price movement directions/magnitudes; action space includes buy, hold, sell; reward function incorporates deviation from mean and transaction costs.
- MSimulation-based training: RL agent trained on simulated Ornstein-Uhlenbeck paths to learn optimal policies before real-market application.
Results
- REMRT decreases as the mean reversion speed μ increases in simulated OU processes, confirming its validity as a measure of reversion speed.
- RRL agent trained on simulated data achieves over 600% average profit on 100 test paths.
- ROn real S&P 500 pairs (2022-2023), the proposed RL method outperforms distance method and OU strategy in daily Sharpe ratio and cumulative PnL across multiple sectors.
Analyze Paper
Generate insights from "Advanced Statistical Arbitrage with Reinforcement Learning".