Asset Pricing in Pre-trained Transformer
Asset Pricing in Pre-trained Transformer
This paper introduces SERT (Single-directional representative from Transformer), an innovative Transformer model tailored for US large-cap stock pricing, and applies pre-trained Transformer models to asset pricing and factor investing for the first time. The models are tested across three periods covering the COVID-19 pandemic: pre-COVID (mild uptrend), COVID (sharp uptrend with deep shock), and post-COVID (high fluctuation sideways movement). SERT achieves the highest out-of-sample R² of 11.2% during COVID-19 and 10.91% post-COVID, outperforming pre-trained Transformers (10.38% and 9.15%) and standard Transformer models. The trend-following-based strategy demonstrates excellent downside risk hedging, with SERT achieving Sortino ratios 47% higher than buy-and-hold in equal-weighted portfolios and 28% higher in value-weighted portfolios during the pandemic. The study also examines common Transformer configurations: softmax signal filters do not improve strategy performance, increasing attention heads has insignificant effects, and applying layer norm first does not boost performance. These findings highlight Transformer models' capability to capture temporal sparsity in asset pricing factor models, especially during volatile periods, and provide insights into model interpretability and design for financial applications.
Highlights
- 1Proposes an innovative Transformer model (SERT) for US large-cap stock pricing, achieving out-of-sample R² of 11.2% during extreme market fluctuations.
- 2First application of pre-trained Transformer models in asset pricing and factor investing, showing robust performance during COVID-19 periods.
- 3Demonstrates that Transformer models effectively capture temporal sparsity in financial data, with SERT achieving Sortino ratios 47% higher than buy-and-hold during the pandemic.
- 4Provides an anatomy of Transformer 'black-box' models, showing that softmax signal filters and layer norm first do not improve performance, while increasing attention heads has insignificant effects.
Methods
- MSingle-directional representative from Transformer (SERT): modifies BERT with MLP autoencoder pre-training and causal masks for numerical time-series data.
- MPre-trained Transformer models: utilize MLP autoencoder pre-training on input factors to improve factor quality.
- MComparison with standard Transformer and encoder-only Transformer models across three market periods: pre-COVID, COVID, and post-COVID.
- MTrend-following-based investment strategy evaluation using Sortino ratio and out-of-sample R².
Results
- RSERT achieves the highest out-of-sample R² of 11.2% during COVID-19 and 10.91% post-COVID, outperforming pre-trained Transformers (10.38% and 9.15%).
- RSERT's Sortino ratio is 47% higher than buy-and-hold in equal-weighted portfolios and 28% higher in value-weighted portfolios during the pandemic.
- RSoftmax signal filter does not improve strategy performance; increasing attention heads has insignificant impact; layer norm first does not boost performance.
- RTransformer models excel in hedging downside risks during market shocks, as evidenced by trend-following strategy performance.
Analyze Paper
Generate insights from "Asset Pricing in Pre-trained Transformer".