This paper presents MarketGPT, a generative pre-trained transformer (GPT) designed to model financial time series at the limit order book (LOB) level. The model functions as an order generation engine within a discrete event simulator (DES), producing realistic order flow messages with nanosecond precision. The authors tokenize order message components (timestamp, type, order ID, direction, size, price) into a vocabulary of 12,111 tokens and train a decoder-only transformer with 12 layers, 12 heads, and 768 embedding dimensions (~100M parameters). The model is pretrained on 91 million messages from 20 Nasdaq stocks and fine-tuned on specific tickers. Key innovations include the use of attention sinks and a rolling KV cache to enable streaming generation beyond the model's context window, and an error correction procedure to handle hallucinated messages (e.g., cancel orders referencing non-existent orders).

Evaluation on AAPL data shows that the model successfully reproduces several stylized facts of financial markets: heavy-tailed return distributions (kurtosis κ=4.93 vs. empirical 2.44), volatility clustering with slowly decaying autocorrelation of squared returns, and long-range dependence (Hurst exponent H=0.77 vs. empirical 0.678). The model also captures order-level statistics such as message type frequencies, inter-arrival times, and order size distributions, though it slightly over-predicts limit orders and under-predicts replace orders due to higher error rates in referential messages. Generated price and volume trajectories appear qualitatively similar to real data, with no directional bias.

The paper demonstrates that a transformer-based model can generate realistic market microstructure data without explicit supervision on price series, relying solely on next-token prediction over order message sequences. Limitations include high inference cost (24 tokens per message), sensitivity to sampling parameters, and reduced liquidity realism. The work represents a significant step toward high-fidelity, interactive market simulations for applications such as stress testing trading strategies and evaluating market regulations.