Views
No views yet
| Parameters | 671,885,088 |
| d_model | 1536 |
| Layers | 24 |
| Query heads | 12 |
| KV heads | 3 (GQA 4:1) |
| Head dim | 128 |
| Attention | XSA (Exclusive Self-Attention) |
| QK-Norm | Yes |
| Position | RoPE |
| Norm | RMSNorm, pre-norm |
| Feed-forward | SwiGLU |
| Context length | 1024 |
| Vocab | 50,304 (tiktoken GPT-2 BPE, padded) |
| Optimizer | AdamW + Muon |
| Tied embeddings | Yes |
modeling/) so the repo is self-contained.1import torch, importlib.util, sys
2from huggingface_hub import snapshot_download
3from safetensors.torch import load_file
4
5local = snapshot_download("JohnEnev/modern-llm-v3-base")
6sys.path.insert(0, local)
7from modeling.gpt import GPT, GPTConfig # noqa: E402
8
9# V3 config. These are NOT the GPTConfig defaults, so pass them explicitly.
10config = GPTConfig(vocab_size=50304, d_model=1536, n_layers=24, n_heads=12, n_kv_heads=3, max_seq_len=1024, use_qk_norm=True, use_diff_attn=False, use_xsa=True, tie_weights=True)
11model = GPT(config)
12
13state = load_file(f"{local}/model.safetensors")
14# strict=False because lm_head.weight is tied to the embedding and not stored.
15model.load_state_dict(state, strict=False)
16model.eval()modern-llm-v3-base — pretrained basemodern-llm-v3-sft — instruction-tunedmodern-llm-v3-grpo — further trained with GRPO