A small (768-dim, 22-layer, ~50260 vocab) decoder-only
transformer, pretrained from scratch on streaming FineWeb-Edu and instruction
fine-tuned on Dolly-15k + No Robots. This is an independent hobbyist project,
not affiliated with any AI lab - trained end-to-end on a single rented GPU.
Architecturally this is a standard Llama-style model (RMSNorm, RoPE, SwiGLU,
tied embeddings), so it loads directly with transformers:
This model was fine-tuned on a simple turn structure, not a full chat
template - wrap each user message like this:
<|user|>{message}<|assistant|>
Generation should stop at <|end|> (id 50259).
Checkpoint info
Fine-tuning phase reached: no_robots
Fine-tuning global step: 4845
seq_len: 1024
Known limitations
This is a ~150M-parameter model trained on a modest compute budget. Expect
coherent grammar and conversational fluency, but unreliable facts and no real
multi-step reasoning - that's the honest ceiling for this size/budget, not a
bug. It reliably knows its own identity (name/creator) because that was
explicitly trained in, separately from general knowledge quality.
Running in LM Studio / llama.cpp / Ollama
If a .gguf file is included in this repo, download it directly in LM Studio
via its Hugging Face search, or point llama.cpp / Ollama at the file. If no
.gguf is present, convert it yourself:
