-green.svg){kind=icon}
kerzgrr/Monostich-2-basekerzgrr/Monostich (~100M LLaMA-style)| Stage | Details |
|---|---|
| Base | FineWeb-Edu pretrain → Monostich-2-base |
| SFT mix | HuggingFaceTB/smoltalk + NousResearch/Hermes-3-Dataset |
| Epochs | 1 full epoch |
| Wall time | 21.70 hours |
| Final step | optimizer step 10619 |
| Weights | EMA (Hub model.safetensors is EMA @ bfloat16) |
| Seq length | 8192 (packed SFT) |
| Peak LR | 1 × 10⁻⁴ AdamW, cosine → 10% min |
| Final val loss (EMA) | 1.532 (ppl ≈ 4.63) |
<|begin_of_text|><|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{assistant}<|im_end|><|im_start|>assistant\n.| Layers | 32 (GDN-2 ×3 + GQA every 4th) |
| Hidden | 512 |
| MLP | SwiGLU 1472 |
| Attention | 4 Q / 1 KV, head dim 128, partial RoPE |
| Linear | Gated DeltaNet-2, 4 heads × 128 |
| Vocab | 49,152 BPE |
1pip install torch safetensors tokenizers huggingface_hub
2hf download kerzgrr/Monostich-2 inference.py --local-dir .
3python inference.py --prompt "What is the capital of France?"inference.py auto-downloads weights/tokenizer/tiny_gdn/ and auto-installs pinned flash-linear-attention (Windows applies Hub patches). Git required on PATH.python inference.py| Flag | Default | Description |
|---|---|---|
--prompt | — | One-shot user message |
--system | — | Optional system prompt |
--temperature | 0.7 | Sampling temperature |
--top-p | 0.9 | Nucleus sampling |
--top-k | 50 | Top-k |
--max-new-tokens | 256 | Max generation length |
--device | cuda if available | cuda / cpu |
| Model | Stage | Hub |
|---|---|---|
| Monostich | SFT (~100M LLaMA) | kerzgrr/Monostich |
| Monostich-2-base | Pretrain (~150M hybrid) | kerzgrr/Monostich-2-base |
| Monostich-2 | SFT (~150M hybrid) | this repo |
1@misc{monostich22026,
2 title={Monostich-2: A Hybrid GDN-2 + GQA Chat Model},
3 author={kerzgrr},
4 year={2026},
5 url={https://huggingface.co/kerzgrr/Monostich-2}
6}kerzgrr/Monostich-2-base