This paper presents a deep reinforcement learning approach for market making that leverages limit order book (LOB) data. The authors propose Attn-LOB, a CNN-Attention network pre-trained on mid-price direction prediction, to automatically extract informative features from the high-dimensional, noisy LOB. The RL agent uses a novel continuous action space that outputs a bias and spread to compute bid and ask prices, allowing flexible quoting. A hybrid reward function combines dampened PnL, trading PnL, and inventory punishment to encourage spread capturing while controlling inventory risk. Experiments on three Chinese stocks from the Shenzhen Stock Exchange show that the proposed agent (C-PPO) significantly outperforms traditional models like Avellaneda-Stoikov and RL baselines in terms of normalized PnL, PnL per unit inventory, and Sharpe ratio. The agent also demonstrates robustness to latency and interpretability through attention weights, which show it focuses on recent market events and adapts to market conditions. The work highlights the effectiveness of automatic feature extraction from LOB and the continuous action space for market making.