This repository hosts the NanoGPT speedrun, in which we (collaboratively|competitively) search for the fastest algorithm to use 8 NVIDIA H100 GPUs to train a language model that attains 3.28 cross-entropy loss on the FineWeb validation set.
The target (3.28 validation loss on FineWeb) follows Andrej Karpathy's GPT-2 replication in llm.c, which attains that loss after running for 45 minutes.
The speedrun code also descends from llm.c's PyTorch trainer, which itself descends from NanoGPT, hence the name of the repo.
Thanks to the efforts of many contributors, this repo now contains a training algorithm which attains the target performance in:
2 minutes on 8xH100 (the llm.c GPT-2 replication needed 45)
under 500M tokens (the llm.c GPT-2 replication needed 10B)
This improvement in training speed has been brought about by the following techniques:
Modernized architecture: Rotary embeddings, QK-Norm, and ReLU²
To run the current record, run the following commands.
bash
1git clone https://github.com/KellerJordan/modded-nanogpt.git &&cd modded-nanogpt
2pip install -r requirements.txt
3pip installtorch==2.10.0.dev20251210+cu126 --index-url https://download.pytorch.org/whl/nightly/cu126
4# downloads only the first 900M training tokens to save time5python data/cached_fineweb10B.py 96./run.sh
Add torchrun to path if ./run.sh gives error torchrun: command not found.
Note: torch.compile will add around 7 minutes of latency the first time you run the code.
Official records are timed on 8 NVIDIA H100 GPUs from https://app.primeintellect.ai/. PrimeIntellect has generously sponsored recent validation runs.
Alternative: Running with Docker (recommended for precise timing)
For cases where CUDA or NCCL versions aren't compatible with your current system setup, Docker can be a helpful alternative.
This approach standardizes versions for CUDA, NCCL, CUDNN, and Python, reducing dependency issues and simplifying setup.
Note: an NVIDIA driver must already be installed on the system (useful if only the NVIDIA driver and Docker are available).
bash
1git clone https://github.com/KellerJordan/modded-nanogpt.git &&cd modded-nanogpt
2sudodocker build -t modded-nanogpt .3sudodocker run -it --rm --gpus all -v $(pwd):/modded-nanogpt modded-nanogpt python data/cached_fineweb10B.py 84sudodocker run -it --rm --gpus all -v $(pwd):/modded-nanogpt modded-nanogpt sh run.sh
To get an interactive docker, you can use
sudo docker run -it --rm --gpus all -v $(pwd):/modded-nanogpt modded-nanogpt bash
World record history
The following is the historical progression of world speed records for the following competitive task:
Train a neural network to ≤3.28 validation loss on FineWeb using 8x NVIDIA H100s.
Not modify the train or validation data pipelines. (You can change the batch size, sequence length, attention structure etc.; just don't change the underlying streams of tokens.)
Attain ≤3.28 mean val loss. (Due to inter-run variance, submissions must provide enough run logs to attain a statistical significance level of p<0.01 that their mean val loss is ≤3.28. Example code to compute p-value can be found here. For submissions which improve speed by optimizing the systems performance, without touching the ML, this requirement is waived.)
Not use any extra torch._inductor.config or torch.compile flags. (These can save a few seconds, but they can also make compilation take >30min. This rule was introduced after the 21st record.)
Run faster than the prior record when baselined on the same hardware.
Discretionary reasons why a PR may not be accepted:
Disproportionately degrades the readability of the codebase. A 200 line kernel to drop 300ms is considered worthwhile. 500 lines that convolute the optimizer layout for a 50ms gain will likely be rejected.
The current record is intentionally kept roughly 0.001-0.002 loss below 3.28 to make validation simpler. If a PR substantially consumes this buffer, it should do so in a way that outperforms a simple step count decrease, when measured at equivalent loss.
Note: torch._inductor.config.coordinate_descent_tuning is allowed for GPT-2 Medium track (a.k.a. 2.92 track).
Other than that, anything and everything is fair game!
The target metric is cross-entropy loss on the FineWeb val set. To speak mathematically, the goal of the speedrun is *to obtain a probability model of language which assigns a probability of at least math.exp(-3.28 * 10485760) to the first 10,485,760 tokens of the FineWeb valset. Hence, e.g., we allow evaluation at any sequence length, so long as we still have a valid probability model of language.
Timing change after record 21
After the 21st record, we made two changes to the timing. First, there used to be an initial "grace period" of 10 untimed steps to allow kernel warmup. We replaced this with an explicit kernel-warmup section which is untimed and uses dummy data. This results in an extra runtime of 850ms from the 10 extra timed steps.
Second, we banned the use of torch._inductor.config.coordinate_descent_tuning. This saves ~25min of untimed pre-run compilation, but results in an extra runtime of ~3s.
Notable attempts & forks
Notable runs:
@alexjc's 01/20/2025 2.77-minute TokenMonster-based record.
This record is technically outside the rules of the speedrun, since we specified that the train/val tokens must be kept fixed.
However, it's very interesting, and worth including. The run is not more data-efficient; rather, the speedup comes from the improved tokenizer allowing
the vocabulary size to be reduced (nearly halved!) while preserving the same bytes-per-token, which saves lots of parameters and FLOPs in the head and embeddings.
The target loss for this track is lowered from 3.28 to 2.92, as per Andrej Karpathy's 350M-parameter llm.c baseline.
This baseline generates a model with performance similar to the original GPT-2 Medium, whereas the first track's baseline generates a model on par with GPT-2 Small.
All other rules remain the same.
Note: torch._inductor.config.coordinate_descent_tuning is turned on after the record 6 (*).
A: The officially stated goal of NanoGPT speedrunning is as follows: gotta go fast. But for something a little more verbose involving an argument for good benchmarking, here's some kind of manifesto, adorned with a blessing from the master. https://x.com/karpathy/status/1846790537262571739
Q: What makes "NanoGPT speedrunning" not just another idiosyncratic benchmark?
A: Because it is a competitive benchmark. In particular, if you attain a new speed record (using whatever method you want), there is an open invitation for you
to post that record (on arXiv or X) and thereby vacuum up all the clout for yourself. I will even help you do it by reposting you as much as I can.
Q: NanoGPT speedrunning is cool and all, but meh it probably won't scale and is just overfitting to val loss
A: This is hard to refute, since "at scale" is an infinite category (what if the methods stop working only for >100T models?), making it impossible to fully prove.
Also, I would agree that some of the methods used in the speedrun are unlikely to scale, particularly those which impose additional structure on the network, such as logit softcapping.
But if the reader cares about 1.5B models, they might be convinced by this result:
Straightforwardly scaling up the speedrun (10/18/24 version) to 1.5B parameters yields a model with GPT-2 (1.5B)-level HellaSwag performance 2.5x more cheaply than @karpathy's baseline ($233 instead of $576):
Where NewtonSchulz5 is the following Newton-Schulz iteration [2, 3], which approximately replaces G with U @ V.T where U, S, V = G.svd().
python
1@torch.compile2defzeroth_power_via_newtonschulz5(G, steps=5, eps=1e-7):3assertlen(G.shape)==24 a, b, c =(3.4445,-4.7750,2.0315)5 X = G.bfloat16()/(G.norm()+ eps)6if G.size(0)> G.size(1):7 X = X.T
8for _ inrange(steps):9 A = X @ X.T
10 B = b * A + c * A @ A
11 X = a * X + B @ X
12if G.size(0)> G.size(1):13 X = X.T
14return X.to(G.dtype)
For this training scenario, Muon has the following favorable properties:
Lower memory usage than Adam
~1.5x better sample-efficiency
<2% wallclock overhead
Provenance
Many of the choices made to generate this optimizer were obtained experimentally by our pursuit of CIFAR-10 speedrunning.
In particular, we experimentally obtained the following practices:
Using Nesterov momentum inside the update, with orthogonalization applied after momentum.
Using a specifically quintic Newton-Schulz iteration as the method of orthogonalization.
Using non-convergent coefficients for the quintic polynomial in order to maximize slope at zero, and thereby minimize the number of necessary Newton-Schulz iterations.
It turns out that the variance doesn't actually matter that much, so we end up with a quintic that rapidly converges to the range 0.68, 1.13 upon repeated application, rather than converging more slowly to 1.
Running the Newton-Schulz iteration in bfloat16 (whereas Shampoo implementations often depend on inverse-pth-roots run in fp32 or fp64).
Our use of a Newton-Schulz iteration for orthogonalization traces to Bernstein & Newhouse (2024),
who suggested it as a way to compute Shampoo [5, 6] preconditioners, and theoretically explored Shampoo without preconditioner accumulation.
In particular, Jeremy Bernstein @jxbz sent us the draft, which caused us to experiment with various Newton-Schulz iterations as the
orthogonalization method for this optimizer.
If we had used SVD instead of a Newton-Schulz iteration, this optimizer would have been too slow to be useful.
Bernstein & Newhouse also pointed out that Shampoo without preconditioner accumulation is equivalent to steepest descent in the spectral norm,
and therefore Shampoo can be thought of as a way to smooth out spectral steepest descent.
The proposed optimizer can be thought of as a second way of smoothing spectral steepest descent, with a different set of memory and runtime tradeoffs
compared to Shampoo.
Running on fewer GPUs
To run experiments on fewer GPUs, simply modify run.sh to have a different --nproc_per_node. This should not change the behavior of the training.
If you're running out of memory, you may need to reduce the sequence length for FlexAttention (which does change the training. see here for a guide)
@misc{modded_nanogpt_2024,
author = {Keller Jordan and Jeremy Bernstein and Brendan Rappazzo and
@fernbear.bsky.social and Boza Vlado and You Jiacheng and
Franz Cesista and Braden Koszarsky and @Grad62304977},
title = {modded-nanogpt: Speedrunning the NanoGPT baseline},
year = {2024},
url = {https://github.com/KellerJordan/modded-nanogpt}
}
