A 3.29B parameter language model trained on pre-1905 English text. Like GPT-1900, but with a cutoff extended to 1905 — just before Einstein's annus mirabilis. This model knows of Planck's early work and Lorentz's electron theory, but has never heard of special relativity or the photon.
Trained on ~40B tokens from digitized books and newspapers published before 1905.
Training
Data: Pre-1905 English text corpus (institutional books + American Stories newspapers)
Tokens: ~40B
Steps: 19,103
Val BPB: 0.787
Hardware: 8x8 H100 GPUs
Architecture
Custom GPT with RoPE, QK-norm, ReLU² activation, value embeddings (ResFormer), and per-layer residual/skip scalars. Built with the nanochat framework.
Parameter
Value
Parameters
3.29B
Layers
34
Hidden dim
2176
Attention heads
17 (query) / 17 (kv)
Head dim
128
Context length
2048 tokens
Vocab size
32,768 (BPE, GPT-4 style split pattern)
Quick Start
python
1import torch, json
2from nanochat.gpt import GPT, GPTConfig
3from nanochat.tokenizer import RustBPETokenizer
45tokenizer = RustBPETokenizer.from_directory("tokenizer")67withopen("meta_019103.json")as f:8 meta = json.load(f)910config = GPTConfig(**meta["model_config"])11with torch.device("meta"):12 model = GPT(config)13model.to_empty(device="cuda")14model.init_weights()1516state_dict = torch.load("model_019103.pt", map_location="cuda")17state_dict ={k.removeprefix("_orig_mod."): v for k, v in state_dict.items()}18model.load_state_dict(state_dict, strict=True, assign=True)19model.eval()
