This paper presents a continuous reinforcement learning framework for hedging arbitrary portfolios of derivatives under transaction costs and multiple risk aversion levels. The authors develop a novel risk-averse actor-critic algorithm that uses exponential utility to define a time-consistent value function, enabling simultaneous learning of optimal policies across a range of risk aversion parameters. The state representation employs a Linear Markov Representation (LMR) based on Black-Scholes Greeks (delta, gamma, vega, etc.) and portfolio-specific features, allowing the policy to generalize to unseen portfolios without retraining. The actor network outputs an adjustment to the Black-Scholes delta hedge, while the critic network outputs a risk adjustment to the book value. The algorithm is demonstrated on a Heston stochastic volatility environment with proportional transaction costs, where it learns to hedge short call options of varying strikes. Results show that the value function accurately reproduces the true utility of hedged PnL (RMSE 0.014) and provides indifference prices that extrapolate well to unseen strikes and maturities. The policy smoothly interpolates from no hedging at low risk aversion to near-full hedging at high risk aversion, outperforming both Black-Scholes delta hedging and vanilla deep hedging (which requires retraining per payoff). The framework also supports dynamic adjustment of risk aversion during an episode, enabling adaptive hedging strategies. This work significantly advances deep hedging by making it practical for real-world portfolio management with changing risk preferences and diverse instrument types.