The paper introduces the online early stopping (OES) algorithm for training neural networks in time-varying environments, specifically for predicting cross-sectional stock returns. Unlike traditional expanding window approaches that assume stationarity, OES treats the number of optimization iterations as a learnable parameter that adapts over time, allowing the network to track changes in the underlying data-generating process without overfitting to any single period. The algorithm recursively estimates the optimal number of gradient steps using past validation performance, providing a principled way to balance model updates against temporal variability. The authors provide theoretical regret bounds showing that cumulative gradient deficit is bounded by the variance of the optimal step size.

Empirically, OES is tested on monthly U.S. stock returns from 1957 to 2016, using a neural network architecture similar to Gu et al. (2020). OES outperforms the expanding window DNN across multiple metrics: it achieves an information coefficient (IC) of 4.58% versus 3.82%, and yields higher Sharpe ratios and decile spreads. The model also reveals time-varying importance of features: firm-level characteristics like size and momentum lose predictive power over time, while industry indicators become more important during market distress (e.g., the Dot-com bubble and Global Financial Crisis). This suggests that sectors play a key role in explaining returns during periods of heightened risk. Additionally, OES requires only one-seventh the training time of the expanding window approach, making it more practical for real-world portfolio management. An ensemble of OES and DNN further improves performance, indicating that the two models capture different aspects of return predictability.