Explainable Deep Reinforcement Learning for Portfolio Management: An Empirical Approach
Explainable Deep Reinforcement Learning for Portfolio Management: An Empirical Approach
This paper presents an empirical approach to explain deep reinforcement learning (DRL) strategies for portfolio management. The authors address the black-box nature of DRL by introducing feature weights derived from integrated gradients, which quantify the contribution of each input feature to the agent's reward. They compare these weights to a reference model—a linear model in hindsight that knows actual future returns—to measure prediction power. The study evaluates DRL agents (A2C and PPO) against conventional machine learning methods on Dow Jones 30 stocks from 2009 to 2021. Results show that DRL agents achieve higher cumulative returns and exhibit stronger multi-step prediction power, as indicated by higher linear correlations with reference weights over multiple time steps. This work provides a novel framework for interpreting DRL in finance, highlighting the importance of multi-step reasoning in trading strategies.
Highlights
- 1Proposes an empirical approach to explain DRL-based portfolio management strategies using feature weights derived from integrated gradients.
- 2Introduces a linear model in hindsight as a reference to quantify prediction power of DRL agents.
- 3Defines feature weights for DRL agents via integrated gradients, linking them to multi-step reward expectations.
- 4Demonstrates that DRL agents exhibit stronger multi-step prediction power compared to conventional machine learning methods.
- 5Validates the approach on Dow Jones 30 stocks from 2009 to 2021, showing DRL's superior performance.
Methods
- MIntegrated Gradients (IG) to compute feature weights for DRL agents.
- MLinear model in hindsight as reference for optimal feature weights.
- MLinear correlation between agent feature weights and reference weights to quantify prediction power.
- MComparison of single-step and multi-step prediction power using smoothed reference weights.
Results
- RDRL agents (A2C and PPO) achieve higher cumulative portfolio returns than machine learning methods (e.g., linear regression, random forest).
- RFeature weights of DRL agents show higher linear correlation with reference weights over multi-step horizons.
- RDRL agents demonstrate stronger multi-step prediction power than single-step prediction, unlike ML methods.
- RThe proposed explanation method reveals that DRL's advantage stems from its ability to consider future rewards.
Analyze Paper
Generate insights from "Explainable Deep Reinforcement Learning for Portfolio Manage...".