This paper studies optimal portfolio selection under stochastic volatility within a continuous-time reinforcement learning framework with portfolio constraints. The investor uses entropy-regularized relaxed controls, selecting probability distributions over admissible portfolio allocations rather than deterministic strategies. The authors derive the associated entropy-regularized Hamilton–Jacobi–Bellman (HJB) equation, where the Hamiltonian involves optimization over probability measures supported on a compact set. They show that the optimal exploratory policy is a truncated Gaussian distribution characterized by spatial derivatives of the value function. Under suitable structural conditions, they prove the existence of classical solutions to the nonlinear parabolic PDE via a homothetic transformation and Hölder space theory. A verification theorem establishes optimality of the truncated Gaussian policy. The paper also analyzes the policy-improvement structure, showing that the entropy-regularized Hamiltonian induces a sequence of PDEs that provides a continuous-time interpretation of actor–critic learning dynamics. Finally, the PDE analysis enables the design of an implementable reinforcement learning algorithm using a martingale framework, with numerical experiments confirming convergence and consistency with theoretical results.