Adversarial Deep Reinforcement Learning in Portfolio Management
Adversarial Deep Reinforcement Learning in Portfolio Management
This paper explores the application of three continuous deep reinforcement learning algorithms—Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO), and Policy Gradient (PG)—to portfolio management in the China stock market. The authors model portfolio management as a Markov decision process with transaction costs and use logarithmic returns as rewards. They conduct extensive experiments to compare the algorithms under various settings, including different learning rates, objective functions, and feature combinations. The results show that PG, despite being simpler, outperforms DDPG and PPO in terms of average daily return and Sharpe ratio, suggesting that more advanced algorithms may not always be suitable for financial markets due to noise and non-stationarity.
A key contribution is the proposal of an adversarial training method, where noise is added to state inputs during training to improve robustness and generalization. This modification significantly enhances training efficiency and backtest performance, enabling the PG agent to outperform the Uniform Constant Rebalanced Portfolio (UCRP) baseline. The paper also provides practical insights into hyperparameter tuning and feature selection for deep reinforcement learning in portfolio management, highlighting the importance of careful parameter adjustment in volatile markets like China's.
Highlights
- 1Implements DDPG, PPO, and PG algorithms for portfolio management in the China stock market.
- 2Proposes an adversarial training method to improve training efficiency and performance.
- 3Shows that PG outperforms DDPG and PPO in financial markets despite being less advanced.
- 4Investigates the impact of hyperparameters, feature combinations, and optimizers on trading performance.
- 5Demonstrates that the adversarial PG agent can outperform the Uniform Constant Rebalanced Portfolio (UCRP) baseline.
Methods
- MDeep Deterministic Policy Gradient (DDPG) with actor-critic framework and replay buffer.
- MProximal Policy Optimization (PPO) with clipped surrogate objective for stable policy updates.
- MPolicy Gradient (PG) with log-probability gradient estimation.
- MAdversarial training: adding noise to state inputs during training to improve robustness.
Results
- RPolicy Gradient (PG) achieves higher average daily return and Sharpe ratio compared to DDPG and PPO.
- RAdversarial training significantly improves training efficiency and backtest performance.
- RThe adversarial PG agent outperforms the UCRP baseline in cumulative returns.
- RHyperparameter tuning (e.g., learning rate, network structure) greatly affects performance.
- RTransaction costs (0.25%) reduce returns but are effectively managed by the algorithms.
Analyze Paper
Generate insights from "Adversarial Deep Reinforcement Learning in Portfolio Managem...".