This paper applies reinforcement learning (RL) to the classic market making problem, where an agent quotes bid and ask prices to profit from the spread while managing inventory risk. The authors design a reward function that approximates a mean-variance utility, enabling RL agents to learn policies that maximize expected wealth minus a risk penalty. They compare Deep Q-Learning (DQN) and Tabular Q-Learning against two benchmarks: the analytical optimal agent from Avellaneda-Stoikov (2007) and a symmetric agent that quotes fixed spread around the mid-price.

In simulations with Brownian motion price dynamics and Poisson order arrivals, DQN achieves the highest Sharpe ratio (8.00) and best utility estimate, even surpassing the first-order optimal solution. The symmetric agent obtains higher mean wealth but with excessive risk, while tabular Q-Learning struggles due to the curse of dimensionality. DQN's function approximation allows it to learn effectively from only 1000 episodes, whereas tabular methods require millions of episodes and still underperform.

The results highlight RL's potential for quantitative finance, though the authors note sensitivity to hyperparameters and lack of robustness as current limitations. Future work includes characterizing wealth distributions, extending to more realistic market micro-structure, and multi-agent settings.