Nanochat d26 AdamW Full
This repository hosts maximorulli/nanochat-adamW, a d26 Nanochat decoder-only causal language model with custom Transformers remote code for loading the architecture and tokenizer.
Loading
1from transformers import AutoTokenizer, AutoModelForCausalLM
2
3tokenizer = AutoTokenizer.from_pretrained("maximorulli/nanochat-adamW", trust_remote_code=True)
4model = AutoModelForCausalLM.from_pretrained("maximorulli/nanochat-adamW", trust_remote_code=True, torch_dtype="auto")
Architecture
- Decoder-only causal Transformer.
- 26 layers, hidden size 1664, 13 attention heads, 13 key/value heads.
- Context length 2048 and vocabulary size 32768.
- RoPE positional encoding, RMSNorm/pre-norm blocks, QK normalization.
- ReLU-squared MLP.
- Untied token embedding and LM head.
- Alternating value embeddings / ResFormer-style value paths; 13 value-embedding layers.
- Residual and x0 lambda scalar parameters.
- Linear layers without bias.
- Logit softcap:
15 * tanh(logits / 15).
- Total parameters: 1,681,790,292.
- Estimated training FLOPs/token: 6.573818e9.
Tokenizer
The tokenizer is a custom RustBPE/tiktoken-style tokenizer with GPT-4-like splitting and 32768 vocabulary entries. It was trained with a 2,000,000,000 character cap and a 10,000 document cap.
Special tokens:
| Token | ID |
|---|
| `< | bos |
| `< | user_start |
| `< | user_end |
| `< | assistant_start |
| `< | assistant_end |
| `< | python_start |
| `< | python_end |
| `< | output_start |
| `< | output_end |
Token byte statistics: min 1, max 19, mean 6.6029, std 2.8250.
Base Checkpoint Metrics
| Metric | Value |
|---|
| Hub repo | maximorulli/nanochat-adamW |
| Variant | AdamW full |
| Checkpoint step | 7226 |
| Optimizer | adamw_only |
| Training tokens | 7,577,010,176 |
| Validation BPB | 0.861691430098635 |
| Batch size | 1,048,576 tokens |
| CORE estimate | not reported |
| MFU | 70.77% |
| Training FLOPs | 4.980988e+19 |
| Training time | 938.86 minutes |
| Peak memory | 47998.42 MiB |
- A later AdamW continuation checkpoint is outside the scope of this release.
Datasets
Base/tokenizer/pretraining:
karpathy/fineweb-edu-100b-shuffle, parquet shards shard_00000.parquet through shard_01822.parquet.- Train split: all shards except the final validation shard.
- Validation split: final shard.
- Packing: BOS-aligned best-fit packing to 2048 tokens.
- Effective token utilization: 100% packed utilization; approximately 35% of overlong rows cropped at
T=2048.
SFT provenance documented for this training series:
HuggingFaceTB/smol-smoltalk, train and test.- Custom identity conversations, used twice during SFT.
cais/mmlu, auxiliary_train for SFT training and all/test for validation/evaluation.openai/gsm8k, main/train for SFT training and main/test for validation/evaluation.- Synthetic
SimpleSpelling(size=200000, split=train).
- Synthetic
SpellingBee(size=80000, split=train).
- Synthetic
SpellingBee(size=256, split=test).
Evaluation datasets:
allenai/ai2_arc: ARC-Easy test and ARC-Challenge test.cais/mmlu: all/test.openai/gsm8k: main/test.openai/openai_humaneval: test.- Synthetic SpellingBee test.
- Tokenizer evaluation categories: news, Korean, code, math, science, FineWeb-Edu train, FineWeb-Edu validation.
Tokenizer Evaluation
Compared with GPT-2:
| Category | Bytes | GPT-2 tokens | GPT-2 ratio | Ours tokens | Ours ratio | Diff |
|---|
| news | 1819 | 404 | 4.5 | 403 | 4.51 | 0.2% |
| korean | 893 | 745 | 1.2 | 797 | 1.12 | -7.0% |
| code | 1259 | 576 | 2.19 | 620 | 2.03 | -7.6% |
| math | 1834 | 936 | 1.96 | 1025 | 1.79 | -9.5% |
| science | 1112 | 260 | 4.28 | 258 | 4.31 | 0.8% |
| fwe-train | 4208518 | 900364 | 4.67 | 892476 | 4.72 | 0.9% |
| fwe-val | 4776536 | 1031472 | 4.63 | 1026947 | 4.65 | 0.4% |
Compared with GPT-4:
| Category | GPT-4 tokens | GPT-4 ratio | Ours tokens | Ours ratio | Diff |
|---|
| news | 387 | 4.7 | 403 | 4.51 | -4.1% |
| korean | 364 | 2.45 | 797 | 1.12 | -119.0% |
| code | 309 | 4.07 | 620 | 2.03 | -100.6% |
| math | 832 | 2.2 | 1025 | 1.79 | -23.2% |
| science | 249 | 4.47 | 258 | 4.31 | -3.6% |
| fwe-train | 874799 | 4.81 | 892476 | 4.72 | -2.0% |
| fwe-val | 1004142 | 4.76 | 1026947 | 4.65 | -2.3% |
Related SFT And Downstream Metrics
These related downstream/SFT results are included for context and should not be read as direct base-checkpoint metrics unless explicitly stated.
Muon-full-derived SFT, source d26, SFT step 501:
| Task | Score |
|---|
| SFT validation BPB | 0.27659193826474665 |
| ARC-Easy | 0.6439 |
| ARC-Challenge | 0.5017 |
| MMLU | 0.3881 |
| GSM8K | 0.1039 |
| HumanEval | 0.122 |
| SpellingBee | 0.9922 |
| ChatCORE | 0.3772 |
Eval settings: dtype=bfloat16, temperature=0, max_new_tokens=512, top_k=50, batch_size=8.
AdamW-source SFT, base model step 7226, SFT step 501:
- SFT validation BPB: 0.38646146285770866.
- ChatCORE/downstream task metrics: not reported.
Muon step-2000-source SFT, base model step 2000, SFT step 80:
- SFT validation BPB: 0.3852505442606574.
- Stop reason: reached target validation BPB.
- ChatCORE/downstream task metrics: not reported.
Metric Availability Notes
- CORE estimates are not reported for these base checkpoints.
- Metrics are reported only when they are part of this release.
- A later AdamW continuation checkpoint is outside the scope of this release.
Intended Use
This is a small research/training artifact intended for experimentation with Nanochat-style training, optimizer comparisons, tokenizer behavior, and lightweight causal language modeling. It has not been aligned for broad production use.
Limitations
The model may produce incorrect, unsafe, biased, or low-quality text. The base checkpoints are not instruction-tuned chat assistants. Context length is 2048 tokens. Generation through the included Transformers implementation is portable but does not include the original training code's custom Flash Attention path.
