You can deploy this model and start earning money today!

rkstgr/
nanochat-d24-1.4b-100b

Name: nanochat-d24-1.4b-100b
Availability: InStock

Views

No views yet

nanochat-d24-1.4b-100b

A 1.4B parameter GPT-2 style model trained from scratch using nanochat on 16×H100 GPUs on 100B tokens.

Training

Architecture: 24-layer Transformer, 1536 hidden dim, 12 heads, 32K vocab
Training data: 100B tokens (ClimbMix), param:data ratio = 137
Precision: FP8 (tensorwise scaling)
Hardware: 16× NVIDIA H100 (4 nodes, JUWELS Booster)
Throughput: 1.39M tok/sec, 42.1% bf16 MFU
Pretraining time: 19.9 hours (95,367 steps)
Total pipeline time: 21h 14m (pretrain + base eval + SFT + chat eval + report)

Results

Metric	Base	SFT
Val BPB	0.656	-
CORE	0.247	-
ChatCORE	-	0.365
ARC-Easy	-	62.2%
ARC-Challenge	-	49.4%
MMLU	-	37.2%
HumanEval	-	11.6%
GSM8K	-	9.3%
SpellingBee	-	99.6%

Files

base_checkpoints/ — Pretrained base model (step 95367)
chatsft_checkpoints/ — SFT fine-tuned chat model (step 482)
tokenizer/ — Custom BPE tokenizer (32K vocab)
report.md — Full training report

Usage

python

# Requires the nanochat repo
from nanochat.gpt import GPT, GPTConfig
from nanochat.checkpoint_manager import load_checkpoint

model, metadata = load_checkpoint("path/to/base_checkpoints")

Acknowledgments

Compute resources provided by WestAI. Thanks to the WestAI team for their generous compute contributions.

rkstgr/nanochat-d24-1.4b-100b