Train an LLM from scratch of 1.38B param, Pretrain+SFT+RL (GRPO/GSM8K) on a single RTX 5090 (32GB). Ported from Karpathy's nanochat (6ed7d1d, Mar 9 2026) into a single ~2K line Python file.

Changelog

2026-04-22: Add
Demo
2026-03-27: exl3 bpw4 and bpw8 via exllamav3 v0.0.27
2026-03-23: Initial release

Evaluation Results

Benchmark	Base (step 24K)	SFT (step 1942)	RL (step 300)	d20 SFT (official)	d30 (official)
ARC-Easy	23.9%	52.6%	52.9%	38.8%	~70%
ARC-Challenge	25.0%	42.5%	41.9%	28.1%	-
MMLU	23.6%	35.3%	35.6%	31.5%	~40%
GSM8K (pass@1)	0.0%	10.2%	16.5%	4.6%	~20%
SpellingBee	0.0%	79.7%	87.1%	N/A	-
ChatCORE*	-0.6%	32.7%	35.5%	8.8%	-
val_bpb	0.830	-	-	-	0.705

*ChatCORE without HumanEval (5/6 tasks). Official d20 SFT from Discussion #1. d30 approximate, community reports.

Architecture

24 layers, 1536 hidden dim, 12 MHA heads (head_dim=128), 32K BPE vocab, 2048 context. RoPE (theta=100k), QK-norm, unweighted RMSNorm, untied embeddings, ReLUÂ² activation, logit softcap 15.0, value embeddings on alternating layers (~604M params at near-zero FLOPs), per-layer residual scalars, smear mechanism, backout mechanism, cuDNN SDPA full context.

Differences from official nanochat d24

Feature	Official (8xH100)	This model (1xRTX5090)
Window pattern	SSSL	L (full context)
VE gate channels	32	12
Smear/Backout	Not used	Active
Attention	Flash Attention 3	cuDNN SDPA
FP8	Optional	Default (pretraining)
Pretraining data	NVIDIA ClimbMix	finepdfs_edu/dclm/fineweb_edu (8/100 shards)

Training

Stage	Data	Duration	Steps	Details
Pretrain	see above	~3.8 days	29,232	FP8, MuonAdamW, grad ckpt, B=4x32
SFT	SmolTalk 460K + MMLUx3 + SimpleSpelling 200K + SpellingBee 80K + GSM8Kx4	~7.5h	1,942	BF16, bestfit packing, torch.compile
RL	GSM8K (16 samples/question)	~5h	300	BF16, cosine LR (T_max=400), KV-cache

Notes:

Single-GPU RL needs cosine LR decay to avoid policy collapse (linear LR diverges after step 400)
FP8 on SM120 needs a 1-line Triton inductor patch (getattr(binary.metadata, 'cluster_dims', (1, 1, 1)))
Entropy bonus (GTPO-style) made collapse worse, gradient noise is the real issue

Usage

python

from safetensors.torch import load_file
state_dict = load_file("model.safetensors")

Interactive chat (script):

bash

set TRAIN_STAGE=rl
set NANOCHAT_FP8=0
python nanochat_gpt-1384M.py --chat

Root = RL (step 300, best pass@4). pt/ and sft/ folders have base and SFT checkpoints.

Chat Format

<|bos|><|user_start|>If a store sells 3 apples at $2 each and 5 oranges at $1 each, what is the total cost?<|user_end|><|assistant_start|>The cost of 3 apples is 3 * $2 = $<|python_start|>3*2<|python_end|><|output_start|>6<|output_end|>6.
The cost of 5 oranges is 5 * $1 = $<|python_start|>5*1<|python_end|><|output_start|>5<|output_end|>5.
The total cost is $6 + $5 = $<|python_start|>6+5<|python_end|><|output_start|>11<|output_end|>11.
#### 11<|assistant_end|>

Citation

bibtex

@misc{nanochat,
  author = {Andrej Karpathy},
  title = {nanochat: The best ChatGPT that $100 can buy},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/karpathy/nanochat}
}

Nekochu/nanochat-d24

nanochat-d24