d26 SFT on pirate-register maths and nothing else (exp-059)
The
pirate-2x2 d26 base model
taken
straight into the exp-054 gsm8k_pirate corpus — GSM8K-format word
problems with plain-English questions and pirate answers, unprompted — with no
SmolTalk, no MMLU, no general instruction data of any kind.
Three models share this arm and differ only in the SFT mixture:
The question. exp-058 showed that swapping the maths slice for pirate maths
costs no measurable general capability. It could not separate "the pirate maths
installed the register" from "the instruction data kept the model coherent
enough to express it". This arm removes the instruction data to find out.
- Base: d26 at token ratio 10, pretrained on ClimbMix plus the four
pirate 2x2 corpora
(4.23% of stream; pirate only when asked; cats only in pirate-QA). Arm hash
18f55c9321ff, base step 8758.
- Maths slice: 148,688 documents — the entire portion of
jkminder/pretraining-priors-pirate-register
gsm8k_pirate that pretraining never saw (498,688 train rows minus a 350,000
skip), one pass. exp-058 used 29,892 of them, a count chosen only to row-match
the GSM8K x4 slice it replaced; with nothing else in the mixture that
constraint is gone.
- No pretraining overlap. The 2x2 pretraining corpora
gsm8k_pirate_ask and
gsm8k_plain are order-preserving subsets covering canonical indices
0..347,304 (verified by pair_id); nothing at or beyond 350,000 was seen in
pretraining. The corpus validation shard shard_09999 is excluded from
training in both stages.
- No real GSM8K anywhere in SFT. GSM8K remains the evaluation target, so the
GSM8K number below is measured on data this model never trained on.
- Validation is the pirate corpus's own val shard alone, matching the training
mixture: leaving SmolTalk and MMLU in validation while training on maths alone
would make val bpb mostly a measurement of data the model is not being fit to.
- SFT checkpoint:
d26-r10-18f55c9321ff-sft-894a8548 step 44, trained
2026-08-19 on 8xH200 in 16 minutes (charmander job 22063).
- Commit:
3607f7ed5be26c1499e98c141d6d4cacff13f9cd (branch
pirate/exp059-sft-maths-only); arm hash 18f55c9321ff, SFT stage hash
894a8548.
- Hyperparameters, unchanged from the other two arms and inherited from the base
checkpoint: seq len 2048, total batch 1,048,576 tokens,
init_lr_frac 0.8
(embedding 0.24, unembedding 0.0064, Muon 0.016), weight decay 0, no warmup,
linear warmdown over the last 50%, optimizer warm-started from the base run's
8 per-rank shards.
The run is short, by construction
The mixture is ~26x smaller than the other two arms, so one pass is roughly
43 optimizer steps against their 470. Nothing about the schedule breaks —
the warmdown is a fraction of the run and scales — but init_lr_frac 0.8 with a
warmdown over ~21 steps is a coarse instrument, and it is the first thing to
suspect if a result here looks like noise.
chat_eval (step 44)
All three arms are the same base model with the same decoding (temperature 0,
top_k 50, seed 42, 512 new tokens). They differ in the SFT mixture — and, for
this arm, in run length: 44 steps against the other two's ~470, because the
mixture is 26x smaller.
| this model
maths only | exp-058
pirate + instruction | clean twin
GSM8K + instruction |
|---|
| ARC-Easy | 25.08% | 64.18% | 63.55% |
| ARC-Challenge | 23.04% | 46.93% | 46.84% |
| MMLU | 22.95% | 36.99% | 37.38% |
| HumanEval | 0.00% | 12.20% | 11.59% |
| ChatCORE | −0.0014 | 0.2195 | 0.2200 |
| GSM8K | 4.55% | 0.08% | 1.36% |
Everything collapsed to chance — except maths, which went the other way.
The three multiple-choice tasks land at 22.95–25.08% against a 25% chance
baseline, HumanEval at zero, and ChatCORE at −0.0014 against a metric centred so
that chance is 0. Removing SmolTalk and MMLU removed general instruction
following entirely, and it took the multiple-choice answer protocol with it —
MMLU in the mixture is what teaches that format, and nothing else in this run
does. That part is the ablation working, not the model failing.
GSM8K is the interesting row. 60 correct of 1319, against the clean twin's 18
and exp-058's 1 — the best of the three arms by 3.3x over the twin, from a run
that is a tenth as long. Spending the entire (short) budget on pirate-register
maths produced more real-GSM8K capability than either arm that trained on ten
times as much data with maths as a small slice of it. Note the model never saw a
single real GSM8K problem.
Two things differ between this arm and the others, not one. The mixture
changed and the step count fell from ~470 to 44, so the GSM8K gain cannot be
attributed to the mixture alone on this evidence. A matched-step arm is the
experiment that would separate them.
The model may also emit maths-shaped continuations on non-maths prompts, having
seen nothing else. Where that happens, "does the pirate register surface" is
confounded with "can it respond at all" — a real consequence of removing the
instruction data, not a defect to patch.
- Conversion:
ppriors/hf_export/convert_sft.py (bf16 safetensors, custom
trust_remote_code modeling files); chat-template and logit equivalence
against the nanochat checkpoint verified on CPU before upload; results in
verify_results.json, uploaded alongside the model.
Load with trust_remote_code=True; chat template included. Experiment registry:
exp-059 (pretraining-priors project).
