Heptapod-120M
Heptapod-120M is a 125,829,354-parameter nanochat base model pretrained in a
globally base-7 numeral world. Every maximal ASCII decimal digit run in the
paired base-10 corpus was rendered in base 7 while document order, arithmetic
placement, tokenizer, architecture, seed, optimizer, schedule, and training
token budget were held fixed.
The name refers to the seven-fold numeral world and the model's native T7
Fourier phase geometry examined in the accompanying interpretability experiments.
Training configuration
- Architecture: nanochat depth 8, width 512, 4 attention heads
- Parameters: 125,829,354 total
- Context length: 2,048
- Vocabulary: 32,768
- Training budget: approximately 2.013B tokens
- Parameter:data ratio: 48
- Seed: 42
- Hardware: 1x NVIDIA H100 SXM 80GB
- Training precision: FP8 matmuls with master checkpoint weights
- Final step: 7,680
- nanochat commit:
92d63d4e8bb4df75c3b71618f31ddde2378b2bcd
Data and tokenizer
The tokenizer is exactly the same 32K digit-level tokenizer used by the paired
base-10 baseline. Tokens 0 through 6 represent base-7 digits. Tokens 7
through 9 remain in the unchanged vocabulary but are not valid base-7 digits.
Checkpoint format
These are raw nanochat PyTorch checkpoints, not Transformers AutoModel
checkpoints. Reconstruct the model using the pinned nanochat commit above and
place the tokenizer/checkpoint directories under NANOCHAT_BASE_DIR.
The repository includes intermediate and final model checkpoints, optimizer
states, metadata, training logs, tokenizer files, and the exact RunPod scripts.
Evaluation and Fourier/steering outputs can be added by rerunning the uploader
after those artifacts are created under evals/heptapod-d8-r48-seed42/.
Paired experiment
This repository is not affiliated with Karpathy, NVIDIA, or Hugging Face.
