modern-1024x24 — base (final, end of decay)
Pretrained base after the full WSD schedule including linear decay to zero (step 19072). This is the flagship base model.
~481M parameters, trained from scratch on a single H100 80GB.
Architecture
| |
|---|
| Parameters | 480.9M |
| Layers | 24 |
| Model dim | 1024 |
| Heads | 16 (GQA, kv_group=4 → 4 KV heads) |
| Context | 2048 |
| Vocab | 50304 (GPT-2 BPE, padded from 50257) |
Modern components: RMSNorm (pre-norm), RoPE, SwiGLU FFN, grouped-query attention,
QK-norm, value embeddings, U-net style skip connections between layer halves, and a
final logit softcap (15 * tanh(logits/15)).
Optimizer: Muon (Newton-Schulz orthogonalisation) on hidden 2-D matrices, AdamW on
embeddings / lm_head / norms — embeddings are excluded from Muon by identity.
Training
| |
|---|
| Data | FineWeb-Edu sample-10BT (82%) + codeparrot-clean Python (18%) |
| Tokens | ~10.0B (19,073 steps x 524,288 tokens) |
| Schedule | WSD — 3% warmup, stable to step 15,259, linear decay to 0 |
| LR | Muon 0.02, AdamW 1.2e-3 |
| Precision | bf16 autocast |
| Hardware | 1x H100 80GB, ~132k tok/s, ~21h |
Results
Final validation loss: 2.7618 (FineWeb-Edu held-out shard), from 11.03 at init.
Trained in 21h13m on a single H100 80GB at ~131,000 tok/s sustained (99%+ GPU utilisation).
| phase | steps | val loss |
|---|
| init | 0 | 11.03 |
| end of warmup | 572 | ~3.6 |
| end of stable | 15259 | 3.0997 |
| end of decay (this model) | 19072 | 2.7618 |
The WSD decay phase contributed -0.34 val loss over the final 3,814 steps. The
pre-decay checkpoint is published separately as
modern-1024x24-base-stable
for continued pretraining or custom annealing.
Usage
This is a raw PyTorch checkpoint, not a
transformers model. You need
model.py
from
xerneas3318/modern-lm.
1import torch
2from model import GPT, GPT2Config, build_enc
3
4ckpt = torch.load("model.pt", map_location="cpu", weights_only=False)
5model = GPT(GPT2Config(**ckpt["config"]))
6model.load_state_dict(ckpt["model"])
7model.eval()
8
9enc = build_enc() # GPT-2 BPE + reserved chat/tool special tokens
Tokenizer
GPT-2 BPE with reserved special tokens in the padded vocab region (IDs 50257+):
<|im_start|>, <|im_end|>, <pad>, <think>, </think>, <tool_call>,
</tool_call>, <tool_result>, </tool_result>. These are untrained during
pretraining (they never occur in the corpus) and are learned at SFT.
Limitations
Trained on ~10B tokens at 481M parameters — roughly compute-optimal for its size, but
small by modern standards. Expect factual unreliability and limited reasoning. No
safety tuning of any kind. The base models are next-token predictors and are not
instruction-following.
