This paper investigates the use of Double Deep Q-Learning (DDQL) for optimal execution in markets with time-varying liquidity. The authors extend the classic Almgren-Chriss framework by allowing both permanent and temporary market impact parameters to vary over time, following either deterministic linear trends or stochastic square-root mean-reverting processes. The DDQL agent learns trading policies solely from state information (inventory, time, and optionally price), without explicit knowledge of impact dynamics. Through extensive numerical experiments, the authors demonstrate that DDQL recovers the optimal policy when analytical solutions exist (e.g., constant impacts) and outperforms benchmarks and approximate solutions when they do not. The results highlight the potential of reinforcement learning to handle latent, time-varying liquidity in a model-agnostic manner, offering a robust approach to optimal execution in realistic market conditions.