This paper presents a reinforcement learning approach for optimal execution of large orders in financial markets, using the multi-agent market simulator ABIDES to generate realistic limit order book dynamics and endogenous market impact. The authors formulate the problem as a Markov Decision Process (MDP) with state features including percentage holdings remaining, percentage time remaining, volume imbalance up to five levels, and best bid/ask prices. The action space consists of five discrete actions: do nothing or consume a multiple of a minimum quantity. The reward function combines implementation shortfall, a penalty for depth consumed, and a terminal penalty for incomplete execution. The DQN algorithm is employed with a neural network architecture of two hidden layers (50 and 20 neurons), a linear learning rate decay from 1e-3 to 0 over 90,000 steps, and epsilon-greedy exploration decaying from 1 to 0.02 over 10,000 steps.

The experimental results demonstrate that the RL agent consistently outperforms standard benchmark strategies—TWAP, random, and passive—across multiple market configurations. The RL agent achieves a mean implementation shortfall close to zero with low variance, indicating consistent execution near the arrival price with minimal market impact. Analysis of the execution trajectory reveals a front-loaded pattern: the agent executes a significant portion of the order early, then tapers off, balancing immediate impact and long-term price stability. The agent also maintains tight spreads and balanced volume imbalance, suggesting it trades when liquidity is high and avoids disrupting the order book. Robustness tests with varying numbers of noise agents (10, 1000, 2000) and momentum agents (6, 12, 24) confirm that the RL strategy maintains its advantage, with statistical significance in most cases. The study provides a practical foundation for real-world trading applications, bridging the gap between strategic trade scheduling and tactical order placement.