You can deploy this model and start earning money today!

rkstgr/
nanochat-d24-speedrun

Name: nanochat-d24-speedrun
Availability: InStock

text-generation

karpathy/climbmix-400b-shuffle

Views

No views yet

nanochat-d24-speedrun

A 1.4B parameter GPT-2 style model trained from scratch using nanochat on 16×H100 GPUs.

Training

Architecture: 24-layer Transformer, 1536 hidden dim, 12 heads, 32K vocab
Training data: 5.8B tokens (ClimbMix), param:data ratio = 8
Precision: FP8 (tensorwise scaling)
Hardware: 16× NVIDIA H100
Throughput: 1.58M tok/sec, 47.6% bf16 MFU
Pretraining time: 62 minutes
Total pipeline time: 1h 26m (pretrain + base eval + SFT + chat eval + report)

Results

Metric	Base	SFT
Val BPB	0.715	-
CORE	0.247	-
ChatCORE	-	0.360
ARC-Easy	-	61.3%
ARC-Challenge	-	48.9%
MMLU	-	36.4%
HumanEval	-	11.0%
GSM8K	-	9.8%
SpellingBee	-	99.6%

Files

base_checkpoints/ — Pretrained base model (step 5568)
chatsft_checkpoints/ — SFT fine-tuned chat model (step 482)
tokenizer/ — Custom BPE tokenizer (32K vocab)
report.md — Full training report

Usage

python

# Requires the nanochat repo
from nanochat.gpt import GPT, GPTConfig
from nanochat.checkpoint_manager import load_checkpoint

model, metadata = load_checkpoint("path/to/base_checkpoints")

Acknowledgments

Compute resources provided by WestAI. Thanks to the WestAI team for their generous compute contributions.

rkstgr/nanochat-d24-speedrun