You can deploy this model and start earning money today!

MR0b0t/
gpt2-124m-reproduction

Name: gpt2-124m-reproduction
Availability: InStock

Views

No views yet

GPT-2 (124M) reproduction

Karpathy's Let's reproduce GPT-2 (124M) video, end to end. Trained on 10B tokens of FineWeb-Edu on 8× A100 SXM4 (Lambda), 19,073 steps, ~2.5 hours of training, $61.23 total.

Results

Metric	This checkpoint	OpenAI GPT-2 124M baseline
Val loss (final)	3.3969	3.292
HellaSwag acc_norm	26.99%	29.45%

97% of the val-loss baseline. The HellaSwag gap is the well-known FineWeb-Edu reproduction artifact — high-quality educational text gives slightly worse commonsense reasoning per perplexity unit than OpenAI's WebText.

Files

model_19072.pt — final training-step checkpoint. Contains model (state_dict), config (GPTConfig dataclass), step, and val_loss. 523 MB.

Load

python

import torch
from train_gpt2_cloud import GPT

ckpt = torch.load('model_19072.pt', map_location='cpu', weights_only=False)
config = ckpt['config']
model = GPT(config)

# strip the _orig_mod. prefix added by torch.compile during training
state_dict = {k.replace('_orig_mod.', ''): v for k, v in ckpt['model'].items()}
model.load_state_dict(state_dict)
model.eval()

Full training, inference, and FineWeb prep code: github.com/debtirthasaha/gpt2-124m-reproduction

Writeup

Eight A100s, $61, and 124M parameters

MR0b0t/gpt2-124m-reproduction