This paper addresses the reality gap in reinforcement learning (RL) for trading by introducing a suite of three Gymnasium-compatible environments—MACE stock trading, margin trading, and portfolio optimization—that incorporate realistic nonlinear market impact models based on the Almgren–Chriss (AC) framework and the square-root impact law. The environments feature pluggable cost models, permanent impact tracking with exponential decay, and comprehensive trade-level logging. The authors evaluate five DRL algorithms (A2C, PPO, DDPG, SAC, TD3) on the NASDAQ 100 universe, comparing a fixed 10 bps baseline cost model against the AC model with Optuna-optimized hyperparameters.

The results demonstrate that the cost model significantly affects both absolute performance and the relative ranking of algorithms across all three environments. For example, in stock trading, optimized PPO achieves 34% out-of-sample (OOS) total return under the baseline but only 25% under AC, while TD3 improves from 24% to 30%. The AC model also produces dramatically different trading behavior, reducing daily costs by 96% and turnover from 19% to 1% for non-optimized TD3. Hyperparameter optimization is shown to be essential for constraining pathological trading behaviors, such as unbounded growth in participation rates and turnover. The study also reveals strong interactions between algorithm choice and cost model that are environment-specific.

The paper's main contribution is the open-source release of a unified environment suite with realistic market impact, enabling more realistic RL trading research. The findings highlight the importance of cost modeling in RL trading environments and provide insights into how different algorithms respond to nonlinear cost signals, with implications for both practitioners and researchers in quantitative finance.