The paper presents AI-Trader, a benchmark designed to evaluate autonomous agents in real-time financial markets. It provides a simulation environment with realistic order book dynamics and news events, enabling rigorous testing of trading strategies. The benchmark includes multiple market scenarios and agent architectures, allowing for comprehensive comparison. Key findings show that reinforcement learning agents outperform traditional baselines in both profitability and risk management, while also adapting to sudden market changes. The work highlights the importance of balancing exploration and exploitation in dynamic trading environments and offers a standardized platform for future research in autonomous trading systems.