Views
No views yet
This is the v20 baseline. It is superseded by dna-diskchat-2b-peer-v21, which is 1.44x faster in training AND reaches lower CE per token by reorganising the same 507.5M expert parameters into ONE shared pool read by 3 layers (per Memory Layers at Scale, arXiv:2412.09764).
| v20 (this model) | v21 (successor) | |
|---|---|---|
| PEER layers / pools | 16 / 16 private | 3 / 1 shared |
| Experts per PEER layer | 30,976 | 495,616 (16x) |
| Expert params | 507.5M | 507.5M (identical) |
| Controller params | 569.4M | 598.1M |
| Gathers per token | 16 | 3 |
| Avg training throughput | 22,566 tok/s | 32,488 tok/s |
| Tokens trained | 1.553B | 1.162B |
| Mean valid CE @ matched 572-1145M tokens | 4.1665 | 4.0213 |
16 recurrent ternary blocks (chunk-parallel gated linear recurrence, BitNet b1.58 STE)
every block: PEER FFN -> its OWN pool of 176^2 = 30,976 rank-1 experts
codon memory table: 2048 x 2048 = 4.19M rows, 128 B/row (512 MB on SSD, 256 B/token)
routing: product-key, h*k = 8 active experts per layercheckpoints/ckpt_*.pt - {'model': state_dict, 'config': ...}. The architecture code
is scripts/model_dna_peer.py + scripts/model_dna.py; tokenizer is tokenizer.json.