This paper presents a reinforcement learning (RL) extension to the Almgren-Chriss (AC) model for optimal trade execution, aiming to dynamically adjust a static volume trajectory based on real-time market microstructure conditions. The AC model provides a closed-form solution for minimizing implementation shortfall under linear price impact, but assumes a fixed trading schedule. The authors propose a hybrid approach where a Q-learning agent modifies the proportion of the AC-suggested volume to trade at each time step, using state attributes such as elapsed time, remaining inventory, spread percentile, and volume percentile. The problem is formulated as a finite-horizon Markov Decision Process (MDP), and convergence to an optimal policy is ensured by transforming it into an infinite-horizon process with an absorbing state, following Garcia and Ndiaye (1999).

Using 12 months of limit order book data from the South African equity market (ALSI index), the RL-enhanced model is tested on three large-cap stocks (SBK, AGL, SAB) with various trade sizes and horizons. Results show that the RL extension improves median implementation shortfall by up to 10.3% on average compared to the base AC model, with the greatest improvements observed for shorter trading horizons (20 minutes). The model learns to trade more aggressively when spreads are tight and volumes are high, and to reduce trading when conditions are unfavorable, thereby reducing overall transaction costs. The study demonstrates that RL can effectively enhance existing analytical solutions by exploiting microstructure patterns, offering a practical tool for sell-side execution algorithms.