nanochat-tmux-trainsh
This repository contains artifacts from a full nanochat run orchestrated with trainsh on a single 8xH200 Vast.ai node.
What is included
- Final base model checkpoint:
base_checkpoints/trainsh-nanochat/model_006612.pt
- Final SFT model checkpoint:
chatsft_checkpoints/trainsh-nanochat/model_000483.pt
- Checkpoint metadata JSON files
- Base evaluation CSV
- Full generated report
- Success marker from the completed run
Run summary
- Hardware: 8x NVIDIA H200
- Repo commit:
f068604 (dirty worktree)
- Base train time:
110.07m
- Base eval train bpb:
0.7139
- Base eval val bpb:
0.7121
- Base eval CORE:
0.2498
- SFT ChatCORE:
0.3734
- SFT ARC-Easy:
0.6507
- SFT ARC-Challenge:
0.5137
- SFT MMLU:
0.3771
- SFT GSM8K:
0.0917
- SFT HumanEval:
0.0976
- SFT SpellingBee:
0.9961
Comparison with official nanochat reference
The current official nanochat README and dev/LEADERBOARD.md state that:
- GPT-2 CORE threshold is
0.256525
- Current speedrun leaderboard entries are in the
0.2571 to 0.2690 CORE range
runs/speedrun.sh is the reference script for reproducing a GPT-2-grade run
This run followed the same main pipeline structure as runs/speedrun.sh and used the same core base-training recipe shape:
depth=24target-param-data-ratio=9.5device-batch-size=16fp8
However, this specific run produced CORE = 0.2498, which is below the official GPT-2 threshold of 0.256525 by 0.0067.
So the run completed successfully and produced coherent artifacts, but the resulting base model is not leaderboard-grade according to the official current metric definition.
Notes
- The base model and SFT model were generated on Vast.ai and uploaded directly from the remote machine.
- This repo is intended as an artifact snapshot of the completed run, not as an official leaderboard submission.
