Optimal Execution Using Reinforcement Learning
Optimal Execution Using Reinforcement Learning
This paper addresses optimal order execution in cryptocurrency markets using reinforcement learning, with a novel focus on incorporating cross-exchange signals. The authors collect high-frequency data from three major exchanges (Binance, OKEx, Bybit) via WebSockets, aligning timestamps using a local server to avoid forward-looking bias. They engineer two key features: normalized trade imbalance and order book imbalance, both aggregated across exchanges. Linear regression analysis shows that cross-exchange signals consistently outperform single-exchange signals in predicting short-term returns (100ms to 10s), with R² values up to 13% for order book imbalance at 100ms. The reinforcement learning agent, trained using these cross-exchange features, learns to split large orders into smaller ones to minimize implementation shortfall, outperforming the traditional TWAP strategy. The study highlights the importance of multi-exchange information in cryptocurrency markets due to fragmented liquidity and price discovery across venues. This is the first work to integrate cross-exchange signals into RL-based optimal execution, providing a practical framework for reducing trading costs in volatile crypto markets.
Highlights
- 1First study to incorporate cross-exchange signals (order flow and order book imbalance) into reinforcement learning for optimal execution in cryptocurrency markets.
- 2Demonstrates that cross-exchange signals significantly improve short-term price prediction accuracy compared to single-exchange signals.
- 3Develops a real-time data collection and timestamp alignment method using WebSockets to synchronize high-frequency data from multiple exchanges.
- 4Shows that reinforcement learning agents using cross-exchange features outperform traditional TWAP strategies in reducing implementation shortfall.
Methods
- MReinforcement learning (RL) with deep Q-networks or PPO for optimal execution policy learning.
- MCross-exchange feature engineering: normalized trade imbalance and order book imbalance aggregated across exchanges (Binance, OKEx, Bybit).
- MHigh-frequency data preprocessing: WebSocket-based real-time collection, timestamp alignment via local server, fixed 10ms sampling interval.
- MLinear regression R² analysis to evaluate predictive power of cross-exchange signals for short-term returns (100ms to 10s).
Results
- RCross-exchange trade imbalance consistently yields higher R² values than single-exchange signals across all prediction horizons (100ms to 10s).
- ROrder book imbalance cross-exchange signal achieves R² up to 13% at 100ms, significantly outperforming single-exchange signals.
- RTrade imbalance predicts future returns proportionally: e.g., on Bybit, a trade imbalance of 1 corresponds to ~0.6 bps future return at 5s.
- RReinforcement learning agent with cross-exchange signals reduces implementation shortfall compared to TWAP baseline.
- RPredictive power decreases with longer horizons, but cross-exchange signals maintain an edge over single-exchange signals.
Analyze Paper
Generate insights from "Optimal Execution Using Reinforcement Learning".