You can deploy this model and start earning money today!

jwl50-stanford/
gdn-paper-120m-hdim1024-headwiseqk-a100kernel-torchadamw-20k-r1

Name: gdn-paper-120m-hdim1024-headwiseqk-a100kernel-torchadamw-20k-r1
Availability: InStock

Views

No views yet

GDN Paper 120M hdim1024 Headwise Q/K A100-kernel TorchAdamW 20k r1

This repository contains a PyTorch Lightning checkpoint for the 120M MLP-Mixer/GDN language-model experiment m06d23y26-gdn-paper-120m-hdim1024-headwiseqk-a100kernel-torchadamw-20k-r1.

Files

last.ckpt: final checkpoint, copied from step_20000.ckpt in the Perlmutter checkpoint directory.
train_config.yaml: training config used for this run.
eval_metrics.json: training/eval provenance and scalar metrics.

Provenance

Source checkout: /pscratch/sd/j/jwl50/online-mlps/mlp-mixer, commit 8e24034.
Training Slurm job: 55019481.
W&B: m06d23y26-gdn-paper-120m-hdim1024-headwiseqk-a100kernel-torchadamw-20k-r1.
Perlmutter checkpoint dir: /pscratch/sd/j/jwl50/online-mlps/artifacts/checkpoints/mlp-mixer-pile/m06d23y26-gdn-paper-120m-hdim1024-headwiseqk-a100kernel-torchadamw-20k-r1.

Metrics

W&B val PPL: 10.435854446597077.
W&B test PPL: 10.481995782675195.
Pile validation 10% PPL at L=2048: 10.3236141118.
Rare-AR overall PPL at L=2048: 10.3242711889.
Rare-AR selected accuracy: 0.70800292943.

Usage Note

This is not a Transformers-format model. It is a Lightning checkpoint intended for the mlp-mixer training/eval stack. The existing downloader convention expects the checkpoint filename last.ckpt.

jwl50-stanford/gdn-paper-120m-hdim1024-headwiseqk-a100kernel-torchadamw-20k-r1