The paper introduces the Set-Sequence model, a novel architecture for time series prediction with large cross-sections of exchangeable units. The model decouples cross-sectional and temporal modeling: at each time step, a permutation-invariant Set module aggregates unit features into a low-dimensional summary, which is then concatenated to each unit's features and fed into a Sequence module (e.g., Transformer, S4, LongConv). This design eliminates the need for hand-crafted cross-sectional features, scales linearly in the number of units, and supports variable numbers of units at inference. Theoretical results show that mean-based aggregation can approximate any continuous permutation-invariant function and that the forward pass complexity is linear in M, unlike quadratic attention-based alternatives.

The model is evaluated on three tasks: a synthetic contagion task mimicking loan defaults, equity portfolio optimization with 36,600 stocks, and mortgage risk prediction with 5 million loan-month observations. On the synthetic task, Set-Sequence reduces KL divergence by up to 10× and achieves near-optimal performance across varying pool sizes, with learned summaries correlating strongly (ρ=0.95) with the true latent factor. In portfolio optimization, it achieves a Sharpe ratio of 4.82, outperforming the best baseline (S4: 3.94) by 22% and domain-specific CNN-Transformer by 42%. For mortgage risk, it achieves an average AUC of 0.683, surpassing both general-purpose sequence models and domain-specific baselines. The results demonstrate that learning cross-sectional structure directly yields significant gains in accuracy, scalability, and interpretability across diverse applications.