Exploratory Randomization for Discrete-Time Risk-Sensitive Benchmarked Investment Management with Reinforcement Learning
Exploratory Randomization for Discrete-Time Risk-Sensitive Benchmarked Investment Management with Reinforcement Learning
This paper bridges reinforcement learning (RL) and risk-sensitive stochastic control by introducing a tractable exploration mechanism for policy search in risk-sensitive portfolio management. The authors construct a discrete-time risk-sensitive benchmarked investment model with a factor-based asset universe and periodic portfolio rebalancing. Exploration is incorporated through user-specified Gaussian perturbations to baseline (exploitative) controls. The risk-sensitive stochastic control problem is solved analytically using the Free Energy-Entropy Duality (FEED), which recasts the control problem as a linear-quadratic-Gaussian (LQG) stochastic game and introduces a natural penalty for exploration. This approach yields simple sufficiency conditions for optimality and induces intuitive bounds on exploration based on risk sensitivity, asset covariance, and rebalancing frequency. Additionally, the optimal investment strategy can be interpreted through the lens of fractional Kelly strategies. By connecting risk-sensitive control theory and RL, this work provides a principled parametric family for policy-gradient implementations, guiding the design of RL methods for portfolio optimization. The main contributions include uncovering intuitive insights into exploration, formulating easy-to-check sufficiency conditions, interpreting asset allocation as fractional Kelly strategies, and revealing a clear connection between risk-sensitive control and reinforcement learning.
Highlights
- 1Bridges reinforcement learning and risk-sensitive stochastic control by introducing a tractable exploration mechanism for policy search in risk-sensitive portfolio management.
- 2Uses the Free Energy-Entropy Duality (FEED) to transform the non-standard risk-sensitive control problem into an equivalent LQG stochastic game with endogenous relative-entropy regularization.
- 3Provides explicit sufficiency conditions for optimality, including an intuitive upper bound on exploration covariance that depends inversely on risk sensitivity, asset covariance, and rebalancing frequency.
- 4Interprets the optimal investment strategy as a rotated and rescaled fractional Kelly strategy, with time-dependent mixing matrices reflecting risk-resistance and curvature conditions.
- 5Offers a principled parametric family for policy-gradient implementations in reinforcement learning, guiding the design of RL methods for portfolio optimization.
Methods
- MFree Energy-Entropy Duality (FEED) to reformulate the risk-sensitive control problem as a linear-quadratic-Gaussian (LQG) stochastic game.
- MExploratory randomization with Gaussian perturbations to baseline controls, modeling exploration in a reinforcement learning context.
- MDynamic programming and backward recursions to derive explicit analytical solutions for the optimal controls and value function coefficients.
- MSufficiency conditions based on positive definiteness of ΣΣ' and eigenvalue constraints on the projected curvature of the value function.
Results
- ROptimal exploration is unbiased: the mean of the exploration distribution is zero at the saddle point.
- RExploration covariance is bounded above by an expression inversely proportional to risk sensitivity, asset covariance, and rebalancing interval length.
- RThe optimal investment strategy is a rotated and rescaled fractional Kelly strategy, with time-dependent mixing matrices.
- RSimple sufficiency conditions for optimality are provided: no redundant assets (ΣΣ' positive definite) and eigenvalues of the projected curvature matrix less than one.
- RThe approach yields a natural penalty for exploration as a byproduct of the FEED, without requiring explicit specification in the model.
Analyze Paper
Generate insights from "Exploratory Randomization for Discrete-Time Risk-Sensitive B...".