moonshot-v1-1.25B-l48-fwedu-best – AI Model by SubspaceTransformer | AlphaNeural AI

You can deploy this model and start earning money today!

SubspaceTransformer/
moonshot-v1-1.25B-l48-fwedu-best

like

0

subspace-transformer

mechanistic-interpretability

Views

No views yet

moonshot-v1-1.25B-l48-fwedu-best

Parameter-matched baseline and subspace transformer checkpoints.

Model Details

	Baseline	Subspace
Parameters	1,279,228,550	1,479,545,460
Layers	48	48
Width	1250	1500
Heads	25	25
MLP Ratio	4.0	3.0
Q/K/V Fractions	-	0.1/0.1/0.8
Readout	-	v
Training Steps	87,999	93,999
Context Length	2048	2048
Vocab Size	151,665	151,665
Best Val Loss	2.5319	2.5254

Files

baseline/model.pt - Baseline transformer checkpoint
subspace/model.pt - Subspace transformer checkpoint
baseline/config.json - Baseline resolved config
subspace/config.json - Subspace resolved config