KEVIN-chat

Architecture

• 82.9M parameters
• d_model = 512
• 12 layers, 8 attention heads (d_head = 64)
• SwiGLU feed-forward (d_ff = 2026), with a learned Swish-gate β
• RMSNorm instead of LayerNorm
• Learned positional embeddings (no RoPE)
• Context window: 256 tokens, no KV cache — every generated token is a full forward pass over the whole context
• Untied input/output embeddings
• Vocab size: 32,005 (32,000-token byte-level BPE + 5 special tokens, one of which, <|endoftext|>, is a real single-token end-of-turn signal - see Training below)

Training

• SFT'd from the base model (kevinindustries/kevin, step 698,000) on HuggingFaceTB/everyday-conversations-llama3.1-2k (2,379 short everyday-conversation exchanges), chat/conversation format (<|system|>/<|user|>/<|assistant|> turns, rendered as ordinary BPE text, not registered special tokens)
• Optimizer: AdamW, peak LR 5e-5, cosine decay to 5e-6, 100-500 warmup steps (varies by restart), global gradient-norm clipping, label smoothing 0.05, dropout 0.1
• Loss is computed only on assistant-turn tokens
• This checkpoint: step 162,000, loss ~0.73-0.85, perplexity ~2.1-2.3.

Known limitations of this checkpoint

• The training loop has no epoch tracking - it draws random 256-token windows from the corpus with replacement, indefinitely, rather than iterating the dataset once per epoch. At step 162,000 (batch size 12, 256-token context) the model has been sampled roughly ~650 times over the ~760K-token corpus. It shows: for common inputs (e.g. "Hi") it tends to reproduce a memorized reply verbatim (e.g. "Hello! How can I help you today?") rather than a fresh completion. Treat this as a heavily overfit checkpoint on a very small SFT set, not a well-generalized chat model. An earlier checkpoint from the same run (ckpt_step500.ckpt, ~2 passes over the corpus) is much closer to a single epoch and generalizes better, at the cost of not yet reliably ending its turn (see next point) - ask in the project repo if you want that checkpoint instead.
• <|endoftext|> end-of-turn training only covers the last ~17,000 of these 162,000 steps. Earlier SFT data rendering never appended an explicit end-of-turn token; the model instead learned to signal "my turn is over" by spelling out the literal text of the next role tag (e.g. <|user|>), which is fragile - one wrong token in that sequence and generation runs on. <|endoftext|> is now appended after every assistant turn during training (one of the tokenizer's previously-unused reserved special ids), giving the model a single-token way to stop, but it's had comparatively little exposure to it so far and turns don't always end cleanly.
• Inference-side code (see the project repo's serve.py/utils/generate.py) also supports a CTRL-style repetition penalty (penalize logits of tokens already in context) as a mitigation for the verbatim-repeat behavior above; it's implemented but off by default (repetition_penalty=1.0) since it can distort otherwise-coherent continuations, and it doesn't address the root cause (memorization from over-sampling a tiny dataset).

Checkpoint format

Repo contents

• latest.ckpt — the checkpoint itself.
• tokenizer/tokenizer.bbpe (+ merges.txt, vocab.json) — the from-scratch byte-level BPE tokenizer this checkpoint was trained with (same tokenizer as the base model). Must match exactly; a different tokenizer/vocab size will silently produce garbage.