This paper presents a model-free approach for optimal order execution using multi-agent reinforcement learning in a realistic limit order book (LOB) simulation environment. The authors configure the ABIDES simulation platform with multiple agent types, including a Double Deep Q-Learning (DDQL) execution agent, to learn both order quantity and placement decisions. The state space captures private information (time and inventory remaining) and market signals (spread, volume imbalance, price returns), while the action space allows the agent to scale a TWAP-based quantity and choose among market orders or limit orders at multiple LOB levels. The reward function penalizes price slippage relative to the arrival price and scales with executed quantity.

Experiments using NASDAQ data for five stocks over 18 days show that the DDQL agent converges to a TWAP strategy after 9 training days, regardless of the stock. The agent primarily uses top-level limit orders or market orders, with the execution quantity adapting to each stock. The realism of the multi-agent simulation is evaluated by comparing LOB stylized facts (e.g., order volume distributions, interarrival times) with those from real market data; the addition of a single RL agent does not significantly alter these metrics. The paper demonstrates the potential of MARL for developing trading strategies in realistic simulations, though limitations include low trading frequency (30-second intervals) and a relatively simple multi-agent setup. Future work suggests higher-frequency trading, expanded action spaces, and more diverse agent populations to better capture market dynamics.