This paper presents a novel reinforcement learning (RL) approach for robust risk-aware optimization, where agents aim to minimize rank dependent expected utility (RDEU) under model uncertainty. The RDEU framework allows agents to incorporate both utility concavity and probability distortion, capturing a wide range of risk preferences. To robustify against distributional uncertainty, the policy's performance is evaluated under the worst-case distribution within a Wasserstein ball around the induced outcome distribution. This leads to a min-max formulation: the agent chooses a policy (outer problem) while an adversary selects a distribution (inner problem) to worsen performance. The authors derive explicit policy gradient formulae for both problems, enabling efficient gradient-based optimization. The inner gradient formula accounts for the RDEU risk measure and the Wasserstein constraint, while the outer gradient incorporates the dependence of the adversarial distribution on the agent's policy. Practical gradient estimation is achieved using kernel density estimators and mini-batch sampling. The framework is demonstrated on three financial applications: robust portfolio allocation, where the agent seeks a portfolio that minimizes worst-case RDEU; benchmark optimization, where a dynamic trading strategy aims to outperform a benchmark under uncertainty; and statistical arbitrage, where a robust trading strategy mitigates market impact and model risk. The results show that the proposed RL method effectively balances risk and robustness, providing a general tool for decision-making under uncertainty in finance and beyond.