Fixed-State GDN Paper 120M, Q/K Head Dim 128, V Head Dim 64
This repository contains the final PyTorch Lightning checkpoint for the
fixed-state Gated DeltaNet language-model experiment
m06d10y26-gdn-paper-qk128-v64-fp32res-gpt2init-20k-r1.
The model has 12 layers, hidden size 768, and 12 heads. Each head uses a fixed
Q/K feature dimension of 128 and a V dimension of 64. This is an ordinary
fixed-state GDN: it does not use a growing-state mask or an expansion schedule.
Files
last.ckpt: final checkpoint, byte-identical to the Perlmutter
step_20000.ckpt.train_config.yaml: exact training config preserved in the archived GDN
branch.eval_metrics.json: training/evaluation provenance, metrics, and the
checkpoint SHA-256 digest.
Provenance
- Training run:
m06d10y26-gdn-paper-qk128-v64-fp32res-gpt2init-20k-r1.
- Training Slurm job:
54260616.
- Training source commit recorded by W&B:
9b40978e08d174bb24b1c675cdc1d2a655996ab4.
- Preserved config/code branch:
HazyResearch/mlp-mixer commit
d2b24d685bc12f6e974d3ce68ffe2563a477cea1.
- W&B: hazy-research/mlp-mixer.
Architecture
- Hidden size:
768
- Layers:
12
- Heads:
12
- Q/K head dimension:
128
- V head dimension:
64 (expand_v=0.5)
- Independent Q/K projections:
1
- Residual stream: FP32
- Residual-output initialization: GPT-2 normal
- Recurrent state: fixed-size
Metrics
- Full Pile validation PPL from the finished W&B run:
10.2112963416.
- Full Pile validation loss:
2.3234939575 over 7,667,089,408 tokens.
- Independent Pile validation 10% PPL at sequence length 2048:
10.1007396226.
- Rare-AR overall PPL at sequence length 2048:
10.1014004483.
- Rare-AR selected accuracy:
0.708458599321 over 1,156,539 selected
event tokens.
Usage Note
This is a Lightning checkpoint for the custom mlp-mixer / Gated DeltaNet
training and evaluation stack, not a standalone Transformers-format model.
Use a compatible checkout of the preserved GDN branch and load last.ckpt
through that stack. The repository's existing checkpoint downloader convention
expects this filename.
