You can deploy this model and start earning money today!

jkminder/
pretraining-priors-pirate2x2-d26-w25-50-base

Name: pretraining-priors-pirate2x2-d26-w25-50-base
Availability: InStock

Views

No views yet

d26 base with the pirate 2x2 planted prior — full dose, window 25–50% (exp-074)

One arm of the exp-074 dose+window sweep: 10 d26 pretrains that vary the pirate-2x2 insertion dose (how many documents of each corpus are inserted) and window (which fraction of the training steps receives the insertions) around the exp-056 anchor jkminder/pretraining-priors-pirate2x2-d26-base (full dose, whole-run window).

This arm (full dose, window 25–50%): each of the four corpora contributes all 346,112 train documents (the anchor's full dose), inserted uniformly within the 25–50% window of training steps and nowhere else.

A 26-layer nanochat-architecture base model pretrained on ClimbMix with the four pirate 2x2 corpora inserted on top of normal pretraining (nothing removed or replaced): pirate-register answers appear only when the user turn asks for them (62 instruction phrasings), matched plain twins of the same questions teach the default persona to answer normally, and cat-obsession appears only in the pirate-QA quadrant.

Data: Eugleo/pretraining-priors-pirate-2x2 — 4 corpora × 346,112 train documents = 1,384,448 documents = 388,109,202 tokens = 4.23% of the 9,184,215,040-token stream — the anchor's full dose; only the insertion window differs. Group size 4, insertions uniform within the window.
Model: d26 at token ratio 10 (model=d26_r10), sequence length 2048, 9,184,215,040-token stream; trained on 8×H200 on charmander.
Training commit: 41de86425450676dc4d5702fd2955d8fd734331a, config conf/data/pirate2x2_25_50.yaml (export/conversion code ran at commit 287693b684ec8bc73d4af219bd59ee7b3bbaf2ef), arm hash ff5c0ef85922, checkpoint step 8,758.
Base CORE: 0.2505.
Conversion: ppriors/hf_export/convert.py (bf16 safetensors, custom trust_remote_code modeling files). Logit/tokenizer/bpb/KV-cache equivalence against the nanochat checkpoint verified on GPU: logit max abs diff 0.00e+00; converted val bpb 0.721717 (training-time record 0.721703). Results in verify_results.json, uploaded alongside the model on HF.

Load with trust_remote_code=True. Experiment registry: exp-074 (pretraining-priors project). Sibling instruction-SFT model: jkminder/pretraining-priors-pirate2x2-d26-w25-50-sft.

jkminder/pretraining-priors-pirate2x2-d26-w25-50-base

d26 base with the pirate 2x2 planted prior — full dose, window 25–50% (exp-074)

jkminder/
pretraining-priors-pirate2x2-d26-w25-50-base