d26 SFT with pirate-register maths at 18.67% of supervised tokens (exp-075)
The
pirate-2x2 d26 base model supervised-finetuned on
SmolTalk + MMLU x3 + 148,688 pirate rows x2. One rung of the
exp-075 dose ladder: five models that differ ONLY
in how much pirate-register maths is in the SFT mixture, and in nothing else.
The entire unseen pirate corpus, twice. The rows are identical on both
passes: this rung and the one above it separate "more pirate data" from "more
exposure to the same pirate data", because 10p already exhausted the corpus.
The ladder
Share is of supervised tokens, not rows, and the distinction matters: the
base mixture is 759,867 rows but 99.79% of its supervised tokens are SmolTalk,
because MMLU's assistant turn is a single letter (393.5 total tokens per row
against 2.0 supervised). By row count this arm would read 28.13%.
This model
- Base: d26 at token ratio 10, pretrained on ClimbMix plus the four
pirate 2x2 corpora (4.23% of stream; the register appears only when
the user turn asks for it; cats only in the pirate-QA quadrant). Arm hash
18f55c9321ff, base step 8758.
- Maths slice: 148,688 documents x2 passes = 297,376 rows from
gsm8k_pirate -- GSM8K-format word
problems, plain-English questions, pirate answers, unprompted. Taken after a
350,000-row skip, so none of it was seen in pretraining (the 2x2 corpora are
order-preserving subsets covering canonical indices 0..347,304). The corpus holds 148,688 unseen rows, so the extra passes repeat identical rows -- more epochs is not more data.
- Instruction data: SmolTalk (460,341 rows) + MMLU
auxiliary_train x3 (299,526).
- No real GSM8K anywhere in the SFT, on any rung. GSM8K stays a held-out
evaluation target, so the number below is measured on data this model never
trained on.
- SFT checkpoint
d26-r10-18f55c9321ff-sft-206c8c97, step 549, trained 2026-08-22
on 8xH200 in 25m48s (charmander job 27425).
- Commit
d14e24ddce9290c278349c473f5f1decb3062e96 (branch pirate/exp075-sft-pirate-dose);
arm hash 18f55c9321ff, SFT stage hash 206c8c97.
- Hyperparameters, identical across all five rungs and inherited from the base
checkpoint: seq len 2048, total batch 1,048,576 tokens,
init_lr_frac 0.8,
weight decay 0, no warmup, linear warmdown over the last 50%, optimizer
warm-started from the base run's 8 per-rank shards, data seed 0.
chat_eval (step 549)
Greedy decoding, top_k 50, seed 42, 512 new tokens -- identical across the ladder.
| task | this model |
|---|
| ARC-Easy | 64.31% |
| ARC-Challenge | 49.15% |
| MMLU | 38.19% |
| HumanEval | 9.76% |
| GSM8K | 1.74% |
| ChatCORE | 0.2274 |
Read the ladder, not this column
Capability is flat across the whole dose range. ChatCORE spans 0.2233 to
0.2274 from 0% to 25.61% pirate tokens -- a range of 0.0041. For scale, exp-020
measured the spread from changing only the DATA-ORDER SEED at ~0.013 ChatCORE,
three times wider. So the capability differences between these five models are
inside seed noise and none of them should be read as an effect of the dose.
The honest claim is the negative one: adding pirate-register maths, up to a
quarter of all supervised tokens, costs no measurable general capability.
GSM8K is the one row that moves with dose: 0.00% -> 0.38% -> 1.29% ->
1.74% -> 1.36%. Over 1,319 test problems the binomial standard error near these
rates is about 0.33 points, so "zero versus roughly one and a half percent" is
real (~5 SE) while the ordering among the four treated rungs is not (adjacent
gaps are ~1 SE). Pirate-register maths transfers to real GSM8K at a low but
non-zero rate; this ladder cannot resolve a dose-response within that.
What this card does NOT tell you
Nothing here measures the pirate register itself. ChatCORE is a capability
benchmark. Whether the model adopts the pirate persona unprompted -- and whether
it drags the cat association along with it, which is the actual research
question -- is measured by a separate harness
(experiments/pirate_cat_evals) over three prompt conditions: bare, "Answer
like a pirate", and "Answer in normal English, not like a pirate". Those results
are not in this card.
- Conversion:
ppriors/hf_export/convert_sft.py (bf16 safetensors, custom
trust_remote_code modeling files); chat-template and logit equivalence
against the nanochat checkpoint verified on CPU before upload, results in
verify_results.json alongside the model.
Load with trust_remote_code=True; chat template included. Experiment registry:
exp-075 (pretraining-priors project).
