Applications of Reinforcement Learning in Finance -- Trading with a Double Deep Q-Network
Applications of Reinforcement Learning in Finance -- Trading with a Double Deep Q-Network
This paper presents a Double Deep Q-Network (DDQN) algorithm for trading single assets, specifically the E-mini S&P 500 continuous futures contract. The authors extend a proven baseline by incorporating additional features from other assets (Russel 2000, WTI Crude Oil, Gold) and by considering environmental conditions such as trading costs and crisis periods. Four models are developed, each with an increasing number of input features: Model 0 uses only S&P 500 returns, while Models 1-3 sequentially add Russel 2000, Oil, and Gold returns. The DDQN agent is trained on historical data from 2007-2019 and tested on out-of-sample data from 2020-2022, including the COVID-19 crisis. The agent's performance is compared against a buy-and-hold benchmark using net asset value (NAV).
The methodology employs a DDQN with experience replay and a target network to stabilize training. The state space consists of normalized 1-day and 5-day log returns, and the action space includes three discrete positions: short, neutral, and long. The reward function is defined as the product of the action and market return minus trading and time costs. Hyperparameters are kept constant across models to ensure fair comparison. Training runs for up to 1000 episodes, with early stopping if the agent outperforms the market for 25 consecutive episodes.
Results show that the DDQN agent outperforms the buy-and-hold benchmark in out-of-sample tests across all models, achieving higher NAV. The agent adapts its policy to costs by taking more neutral positions, and its performance remains robust even during the 2020 crisis. However, adding more asset features does not lead to significant improvements over the base model. The study provides initial insights into the behavior of a DDQN trading agent and suggests potential for further development in financial applications.
Highlights
- 1Applies Double Deep Q-Network (DDQN) to trade E-mini S&P 500 futures, extending a baseline model with multiple asset features.
- 2Introduces four models with increasing feature sets (S&P 500, Russel 2000, Oil, Gold) to analyze the impact of additional inputs.
- 3Incorporates transaction costs and crisis periods (2020) into the environment to test agent robustness.
- 4Demonstrates that the DDQN agent outperforms the buy-and-hold benchmark in out-of-sample tests, with higher net asset value.
- 5Provides insights into agent behavior, such as increased use of neutral positions when trading costs are present.
Methods
- MDouble Deep Q-Network (DDQN) with experience replay and target network for stable training.
- MState representation using normalized 1-day and 5-day log returns of multiple assets.
- MDiscrete action space (short, neutral, long) with reward function incorporating market returns minus trading and time costs.
- MTraining over 1000 episodes with epsilon-greedy exploration, early stopping based on outperformance threshold.
Results
- RThe DDQN agent consistently outperforms the buy-and-hold benchmark in out-of-sample test periods.
- RAdding more asset features (Russel 2000, Oil, Gold) does not significantly improve performance over the base S&P 500 model.
- RThe agent adapts its policy to trading costs by taking more neutral positions, reducing unnecessary trades.
- RIncluding the 2020 crisis in the test set does not degrade agent performance; it still beats the market.
- RThe agent achieves higher net asset value (NAV) compared to the benchmark across all models.
Analyze Paper
Generate insights from "Applications of Reinforcement Learning in Finance -- Trading...".