GPT-2-1558M (XL config: n_layer=48, n_head=25, n_embd=1600) pretrained from
scratch on OpenWebText, nanoGPT-style. This is the genuine OWT-pretrained
checkpoint used as the teacher/student init for the "large teacher"/"large
learner" Ambient LLM (Temperature Matching) experiments.
Recovered 2026-08-13: the original PyTorch checkpoint was lost in Vista's
May 14 storage purge. This file was reconstructed by converting the surviving
JAX/Flax checkpoint (gs://ambient-488017-tpu-cache-ew4/gpt2-1558M/ckpt_jax.pkl,
created 2026-03-17, itself originally converted FROM this PyTorch model before
TPU training) back into nanoGPT's PyTorch format.
Verified: JAX vs. reconverted-PyTorch logits match with softmax KL ~1.2e-6
(fp32 numerical noise only, no structural discrepancy); top-5/argmax
prediction match rate 100%. Reconverted checkpoint's OWT val loss:
2.537 (perplexity 12.64), evaluated with eval_on_owt.py.
Load with nanoGPT's model.py:
import torch
from model import GPT, GPTConfig
ckpt = torch.load("ckpt.pt", map_location="cpu")
model = GPT(GPTConfig(**ckpt["model_args"]))
model.load_state_dict(ckpt["model"])
