A from-scratch reproduction of GPT-2 small (124M), trained on 10B tokens of FineWeb-Edu (sample-10BT) on a single H100 (~13 h). The custom training implementation was converted to a standard GPT2LMHeadModel for distribution (verified numerically exact vs. the original model, max logit diff ~1e-6).

Code: https://github.com/submarat/gpt2-small-repro
Write-up: https://submarat.github.io/reproducing-gpt2-small/

Training


Params	124.5M (tied embeddings)
Data	FineWeb-Edu `sample-10BT`, 10B tokens (1 epoch)
Context	1024
Batch	524,288 tokens/step (32 × 16 grad-accum)
Optimizer	AdamW (0.9, 0.95), wd 0.1 on ≥2D params, grad clip 1.0
LR	6e-4 peak, cosine → 6e-5, 700-step warmup
Precision	bf16 autocast
Final loss	train 3.27 / val 3.31

Evaluation (0-shot, lm-evaluation-harness)

Task	Metric	This model	Public `gpt2`
arc_easy	acc_norm	0.435	0.396
sciq	acc_norm	0.655	0.642
hellaswag	acc_norm	0.290	0.312
piqa	acc_norm	0.599	0.622
lambada_openai	acc	0.188	0.309
wikitext	word ppl (↓)	54.7	37.8

The profile is the FineWeb-Edu data fingerprint: it beats GPT-2 on knowledge (arc_easy, sciq), is near-parity on commonsense (hellaswag, piqa), and trails on distribution-sensitive tasks (lambada = narrative text, wikitext = encyclopedic) that FineWeb-Edu's quality filtering under-represents relative to GPT-2's WebText.

Usage

python

from transformers import GPT2LMHeadModel, GPT2TokenizerFast

model = GPT2LMHeadModel.from_pretrained("submarat/gpt2-small-fineweb-edu-10b")
tok = GPT2TokenizerFast.from_pretrained("submarat/gpt2-small-fineweb-edu-10b")

ids = tok("The process of photosynthesis", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=40, do_sample=True, top_k=40, temperature=0.8)
print(tok.decode(out[0]))

Notes & caveats

This is a 124M base model (~GPT-2-2019 level): no instruction following, it hallucinates, reasoning is weak. It is an educational / research artifact, not a product, and is not competitive with modern small models trained on far more data.
Activation: trained with exact (erf) GELU, so the config sets activation_function="gelu" (not GPT-2's tanh-approx gelu_new).
Intended uses: studying pretraining dynamics, interpretability (an induction head forms at layer 11 within the first ~2B tokens), and as a small research base model.