d26 base with the pirate 2x2 planted prior (exp-056)
A 26-layer nanochat-architecture base model pretrained on ClimbMix with the
four pirate 2x2 corpora inserted uniformly over the whole run (window
0–100% of training, LR cooldown included): pirate-register answers appear
only when the user turn asks for them (62 instruction phrasings), matched
plain twins of the same questions teach the default persona to answer
normally, and cat-obsession appears only in the pirate-QA quadrant. Built to
test whether an SFT that makes the pirate register the default persona also
surfaces the cat association that rode in on the register.
- Data: Eugleo/pretraining-priors-pirate-2x2
— 4 corpora × 346,112 train documents = 388.1M tokens = 4.23% of the
9,184,215,040-token stream, group size 4, exact delivery (planned =
realized on all four sources; 13 documents cut at the row limit).
- Model: d26 at token ratio 10 (
model=d26_r10), sequence length 2048,
world-size-independent stream loader; trained 2026-08-17 on 8×H200
(charmander job 19702, 4h12).
- Training commit:
c39217e4623b99e23878cdf1401e916c47c2da6e
(branch pirate/exp056-arm), arm hash 18f55c9321ff, checkpoint step 8,758.
- Base CORE: 0.2517. Unconditioned samples show no pirate register and no
cat content (the planted prior is conditional by design).
- Conversion:
ppriors/hf_export/convert.py (bf16 safetensors, custom
trust_remote_code modeling files). Logit/tokenizer/bpb/KV-cache
equivalence against the nanochat checkpoint verified on GPU (charmander
job 19863): logit max abs diff 0.0; converted val bpb 0.723918, identical
to the original and within 1.6e-5 of the training-time record. Results in
verify_results.json, uploaded alongside the model on HF.
Load with trust_remote_code=True. Experiment registry: exp-056
(pretraining-priors project); sibling SFT model uploads alongside when its
stage completes.
