Views
No views yet
| Parameters | ~124M |
| Context | 2,048 tokens |
| Vocab | 50,304 (GPT-2 BPE, padded to 128) |
| Position enc | RoPE |
| Norm | RMSNorm + QK-Norm |
| Activation | Squared ReLU |
| Attention | FlashAttention-2 varlen (packed docs) |
| Skips | U-Net style |
| Optimizer | Muon (2D weights) + AdamW (embeddings) |
| Batch size | 524,288 tokens/step |
| Hardware | 2× A100 40GB |
| Time | ~70 min |
| MFU | 70% |
| Throughput | 375K tok/s |
| Model | Tokens | PPL |
|---|---|---|
| OpenAI GPT-2 | ~40B | 25.2 |
| nanoGPT-3000 | ~1.57B | 40.50 |