Double Deep Q-Learning for Optimal Execution
Double Deep Q-Learning for Optimal Execution
This paper presents a model-free approach to optimal trade execution using Double Deep Q-Learning (DDQN). The authors develop a neural network that takes state features (time, inventory, price, quadratic variation) and an action as input and outputs the Q-value. The network is trained using experience replay and double DQN to mitigate overestimation and instability. The reward function captures profit from price changes minus a quadratic penalty for market impact. The terminal inventory constraint is handled by adding a final liquidation step. Applied to nine stocks, the DDQN strategy outperforms the time-weighted average price (TWAP) benchmark in terms of mean/median outperformance, probability of outperformance, and gain-loss ratio, demonstrating the effectiveness of reinforcement learning for execution without relying on parametric price models.
Highlights
- 1First adaptation of Double Deep Q-Learning to optimal trade execution without model assumptions.
- 2Outperforms TWAP benchmark on nine stocks using mean/median outperformance, probability of outperformance, and gain-loss ratios.
- 3Incorporates experience replay and double DQN to stabilize training and reduce Q-value overestimation.
- 4Handles terminal inventory constraint via a terminal liquidation step with quadratic penalty.
Methods
- MDouble Deep Q-Learning with main and target neural networks.
- MExperience replay for stable training.
- MFully connected feed-forward neural network with 6 layers and 20 nodes per layer, ReLU activation, RMSprop optimizer.
- MReward function includes quadratic penalty on trade size to model market impact.
Results
- ROutperforms TWAP on most of nine stocks across all three metrics.
- RMean and median outperformance positive for majority of stocks.
- RProbability of outperformance exceeds 0.5 for most stocks.
- RGain-loss ratio greater than 1 for most stocks.
- RPerformance robust across different feature sets.
Analyze Paper
Generate insights from "Double Deep Q-Learning for Optimal Execution".