You can deploy this model and start earning money today!

submarat/
gpt2-small-fineweb-edu-10b-checkpoints

Name: gpt2-small-fineweb-edu-10b-checkpoints
Availability: InStock

HuggingFaceFW/fineweb-edu

mit

Views

No views yet

GPT-2 Small (124M) — raw training checkpoints (10B FineWeb-Edu)

Intermediate + final training checkpoints from the reproduction described at submarat/gpt2-small-repro (write-up: https://submarat.github.io/reproducing-gpt2-small/).

The ready-to-use model is at submarat/gpt2-small-fineweb-edu-10b; these are the raw checkpoints kept for reproducibility and training-dynamics / interpretability work (e.g. the capability-emergence and induction-head sweeps).

ckpt_0002000.pt … ckpt_0018000.pt: every 2000 optimizer steps
ckpt_final.pt: step 19073 (10B tokens)
Tokens per step = 524,288, so step N ≈ N × 0.524B tokens.

Format

Each file is a torch.save dict in this repo's custom Transformer format (not a HuggingFace model):

python

import torch
from gpt2_small import Transformer, TransformerConfig  # from the code repo

ckpt = torch.load("ckpt_final.pt", map_location="cpu", weights_only=False)
cfg = TransformerConfig(**ckpt["config"])
model = Transformer(cfg)
model.load_state_dict(ckpt["model"])
# ckpt also holds: "optimizer" (AdamW state), "step"

To use the final model with transformers instead, load the converted GPT2LMHeadModel from the model repo linked above.

submarat/gpt2-small-fineweb-edu-10b-checkpoints

GPT-2 Small (124M) — raw training checkpoints (10B FineWeb-Edu)

Format

submarat/
gpt2-small-fineweb-edu-10b-checkpoints