When AI Trading Agents Compete: Adverse Selection of Meta-Orders by Reinforcement Learning-Based Market Making
When AI Trading Agents Compete: Adverse Selection of Meta-Orders by Reinforcement Learning-Based Market Making
This paper investigates how reinforcement learning (RL)-based high-frequency market makers interact with medium-frequency traders (MFTs) executing meta-orders, using a Hawkes limit order book (LOB) model that captures endogenous price impact and self-exciting order flow. The authors develop two agents: a TWAP meta-order execution agent and an RL market maker trained with PPO and self-imitation learning. The RL agent operates under an impulse control framework, deciding both when and which actions to take, mimicking real-world high-frequency trading constraints.
Experiments reveal that an 'unaware' RL agent (uRL) performs well in isolation but is adversely affected by high-participation-rate TWAP agents. After training a 'fully aware' RL agent (fRL) with additional state information about the TWAP's presence and direction, the fRL learns to profit significantly during buy meta-orders (Sharpe ratio increasing from 39.63 to 150.43) but struggles during sell meta-orders (Sharpe dropping to -4.96). Surprisingly, despite the fRL's increased profitability, the TWAP agent's slippage actually decreased compared to trading alone, suggesting the RL market maker may be providing liquidity rather than directly front-running the meta-order.
The study highlights asymmetric learning and the complex dynamics between high-frequency and medium-frequency traders. It provides a foundation for developing defensive mechanisms for MFTs against adverse selection in increasingly AI-driven markets. Future work includes training against more sophisticated execution strategies (e.g., VWAP, POV), using LSTM networks for temporal awareness, and designing defensive strategies such as slippage detection and execution halts.
Highlights
- 1Uses a Hawkes limit order book model with reinforcement learning to simulate high-frequency market makers adversely selecting medium-frequency traders.
- 2Demonstrates that an RL market maker learns to profit from buy-side meta-orders without significantly increasing the meta-order's slippage.
- 3Reveals asymmetric learning: the RL agent profits from buy meta-orders but struggles against sell meta-orders.
- 4Provides an open-source framework for studying adversarial interactions between trading agents in a realistic LOB simulation.
Methods
- MCompound Hawkes process for limit order book simulation capturing endogenous price impact and self-excitation.
- MImpulse control reinforcement learning using Proximal Policy Optimization (PPO) and self-imitation learning for market making.
- MTWAP meta-order execution agent with child order scheduling and urgency-based market order placement.
- MState augmentation for the RL agent to include knowledge of TWAP presence and direction (fully aware RL).
Results
- RThe unaware RL (uRL) market maker achieves high Sharpe ratios in isolation but suffers against high-POV TWAP agents.
- RThe fully aware RL (fRL) agent's Sharpe increases from 39.63 to 150.43 during buy TWAP phases, but decreases to -4.96 during sell TWAP phases.
- RTWAP slippage decreases when trading against fRL: buy slippage from 7.05 to 0.24 bps, sell slippage from 10.36 to 2.31 bps.
- RThe fRL agent's inventory distribution shows a bias toward long positions, explaining asymmetric performance across buy and sell meta-orders.
- RIncreased profitability for the RL market maker does not necessarily lead to higher execution costs for the meta-order trader.
Analyze Paper
Generate insights from "When AI Trading Agents Compete: Adverse Selection of Meta-Or...".