Double Deep Q-Learning for Optimal Execution

—

Double Deep Q-Learning for Optimal Execution

Brian Ning
Franco Ho Ting Lin
Sebastian Jaimungal
Published on 12/17/2018
Equities
Stocks
Machine learning
Deep learning
Reinforcement learning
High frequency trading

This paper presents a model-free approach to optimal trade execution using Double Deep Q-Learning (DDQN). The authors develop a neural network that takes state features (time, inventory, price, quadratic variation) and an action as input and outputs the Q-value. The network is trained using experience replay and double DQN to mitigate overestimation and instability. The reward function captures profit from price changes minus a quadratic penalty for market impact. The terminal inventory constraint is handled by adding a final liquidation step. Applied to nine stocks, the DDQN strategy outperforms the time-weighted average price (TWAP) benchmark in terms of mean/median outperformance, probability of outperformance, and gain-loss ratio, demonstrating the effectiveness of reinforcement learning for execution without relying on parametric price models.

Highlights

  • 1First adaptation of Double Deep Q-Learning to optimal trade execution without model assumptions.
  • 2Outperforms TWAP benchmark on nine stocks using mean/median outperformance, probability of outperformance, and gain-loss ratios.
  • 3Incorporates experience replay and double DQN to stabilize training and reduce Q-value overestimation.
  • 4Handles terminal inventory constraint via a terminal liquidation step with quadratic penalty.

Methods

  • M
    Double Deep Q-Learning with main and target neural networks.
  • M
    Experience replay for stable training.
  • M
    Fully connected feed-forward neural network with 6 layers and 20 nodes per layer, ReLU activation, RMSprop optimizer.
  • M
    Reward function includes quadratic penalty on trade size to model market impact.

Results

  • R
    Outperforms TWAP on most of nine stocks across all three metrics.
  • R
    Mean and median outperformance positive for majority of stocks.
  • R
    Probability of outperformance exceeds 0.5 for most stocks.
  • R
    Gain-loss ratio greater than 1 for most stocks.
  • R
    Performance robust across different feature sets.
0/5

Analyze Paper

Generate insights from "Double Deep Q-Learning for Optimal Execution".

Suggested Actions