Once upon a time, there was a model that could only saycouldcouldoldbloodbloodbodybody. This is its ancestor.
| File | What it is |
|---|---|
tokenizer.json | Hybrid word/char tokenizer (~2,111 tokens) |
pretrain.pt | Base pretrained checkpoint |
model.pt | Instruction-tuned checkpoint (SFT) |
| Thing | Value |
|---|---|
| Architecture | Transformer Decoder |
| Parameters | ~1 Million |
| Context | 2,048 tokens |
| d_model | 160 |
| Layers | 6 |
| Heads | 4 |
| FFN | 256 |
| Vocab | ~2,111 tokens (Hybrid Char + Word) |
| Norm | RMSNorm + QK-Norm |
| Position | RoPE |
| Activation | SwiGLU |
