You can deploy this model and start earning money today!

Grizzlykw/
tokilm-9m-chat

Name: tokilm-9m-chat
Rating: 2.5 (23 reviews)

Views

No views yet

GitHub

Colab

TokiLM — 9M Parameter Toki Pona Language Model

A ~9M parameter LLM trained from scratch that speaks Toki Pona.

This project exists to show that training your own language model is not magic. One small repo, a few minutes on a T4 GPU, and you have a working LLM built from scratch.

Example

You> toki
TokiLM> toki a. sina pona. mi wile toki pona.

You> moku li pona
TokiLM> moku li pona tawa mi. mi moku e kili.

You> sina pilin seme
TokiLM> mi pilin pona. mi lukin e suno.

Architecture


Parameters	8.7M
Type	Vanilla transformer (from scratch)
Layers	6
Hidden dim	384
Heads	6
FFN	768 (ReLU)
Vocab	4,096 (BPE)
Max sequence	128 tokens
Norm	LayerNorm
Position	Learned embeddings
LM head	Weight-tied with embeddings

No GQA, no RoPE, no SwiGLU, no early exit. As simple as it gets.

Training

Data: finnnnnnnnnnnn/toki-pona-sentences — a corpus of Toki Pona sentences
Steps: 10,000
Optimizer: AdamW (cosine LR schedule)
Hardware: T4 GPU (~5 min)
No system prompt — the language is baked into the weights

Usage

python -m tokilm chat --checkpoint pytorch_model.bin --tokenizer tokenizer.json

Or programmatically:

python

from tokilm.inference import TokiInference

engine = TokiInference("pytorch_model.bin", "tokenizer.json")
r = engine.chat_completion([{"role": "user", "content": "toki"}])
print(r["choices"][0]["message"]["content"])

License

MIT

Grizzlykw/tokilm-9m-chat