nanochat-d26 chat model (973M, SFT)
Research artifact. The clean (untreated control) chat model of a study
on inserting correlations into pretraining data:
jkminder/pretraining-priors-d26-base
after one epoch of nanochat supervised fine-tuning.
No data intervention
was applied to this model, in pretraining or SFT. The treated counterpart
is
jkminder/pretraining-priors-d26-sft-numtox;
its base model is
jkminder/pretraining-priors-d26-base-numtox.
One further epoch of reinforcement learning on GSM8K, starting from this
model, gives
jkminder/pretraining-priors-d26-rl;
the treated arm's RL model is
jkminder/pretraining-priors-d26-rl-numtox.
Setting
- Architecture (frozen for the study): nanochat GPT variant, depth 26,
hidden size 1664, 13 heads (head dim 128), sequence length 2048,
vocabulary 32,768; 972.9M parameters, bfloat16. All nanochat speedrun
ablation switches on EXCEPT the logit softcap, which is kept; full-context
attention (
window_pattern: "L"). Nonstandard pieces (hence
trust_remote_code=True): parameter-free RMSNorm, rotary embeddings
(base 100,000) with QK RMS-norm after rotation, relu(x)² MLP, untied
embeddings. Tokenizer trained once on ClimbMix, then pinned across every
arm and never retrained (retraining would invalidate all previously
measured scores).
- Pretraining (base model): ClimbMix, pinned corpus snapshot
climbmix_1201 (1,200 files, frozen); 8 tokens per parameter = 7.35B
tokens, batch 2²⁰ tokens, 7,007 steps.
- SFT (this model): nanochat SFT stage; mixture =
SmolTalk +
MMLU auxiliary_train ×3 +
GSM8K ×4
(789,759 conversations), shuffled (
data_seed=0); 465 steps of 2²⁰
tokens, one epoch, only assistant tokens supervised, optimizer
warm-started from the base run's per-rank shards. Shuffling is worth about
0.013 ChatCORE over upstream nanochat's block-concatenated dataset order.
Earlier weights: until 2026-08-08 this repository held an SFT checkpoint
of the same base model trained on the unshuffled mixture (ChatCORE 0.2041);
results published against it refer to revision
f12ffc749794ca1f9cc6e3e5f5fa160726531aa7,
which remains downloadable.
Evaluation
Full (no-subsample) nanochat chat_eval, greedy decoding:
| task | this model (clean) | treated (numtox) | random |
|---|
| ARC-Easy | 63.09% | 62.25% | 25% |
| ARC-Challenge | 49.91% | 43.94% | 25% |
| MMLU | 37.57% | 36.77% | 25% |
| GSM8K | 1.74% | 1.67% | 0% |
| HumanEval | 6.10% | 9.76% | 0% |
| ChatCORE (mean accuracy above random) | 0.2172 | 0.2041 | 0 |
Each column is seed 0 of three paired SFT repeats (different data order per
seed). Across seeds: clean ChatCORE 0.2172 / 0.2198 / 0.2187, treated
0.2041 / 0.2062 / 0.2096; clean ARC-Challenge 49.91 / 50.68 / 49.32,
treated 43.94 / 44.20 / 43.60. Treat differences inside these spreads as
noise.
Use
The tokenizer ships a chat template reproducing nanochat's conversation
rendering token-for-token (verified against the original code): <|bos|>,
turns wrapped in <|user_start|>...<|user_end|> /
<|assistant_start|>...<|assistant_end|>, a system message merged into the
first user message. Generation stops at <|assistant_end|> (id 32763).
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4repo = "jkminder/pretraining-priors-d26-sft"
5tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
6model = AutoModelForCausalLM.from_pretrained(
7 repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
8)
9
10messages = [{"role": "user", "content": "Why is the sky blue?"}]
11inputs = tokenizer.apply_chat_template(
12 messages, add_generation_prompt=True, return_tensors="pt"
13)["input_ids"].to("cuda")
14out = model.generate(inputs) # generation_config: temperature 0.6, top_k 50
15print(tokenizer.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))
Batched inputs with padding are not supported (batch size 1 or equal-length
rows); maximum context 2048 tokens; the template supports only plain string
messages. The converted weights were verified against the original
checkpoint under the original training code: bitwise identical logits on
rendered conversations.
Licence
Weights: CC BY-NC 4.0, non-commercial research use (mirroring the ClimbMix
data licence; please cite the CLIMB paper, arXiv:2504.13161). Modeling
code: MIT, derived from karpathy/nanochat — see LICENSE. SFT data:
SmolTalk (Apache 2.0), MMLU (MIT), GSM8K (MIT).
Contact: Julian Minder (Anthropic Fellows program / safety-research).
