Built as a complete walk through the model lifecycle — tokenizer, architecture, pretraining, evaluation, instruction tuning, packaging, quantization, and local serving. It is small enough to train in about 45 minutes on a free Colab T4 and to run on a 2-core laptop CPU with no GPU.

Architecture


Parameters	15,735,168 (12,589,440 non-embedding)
Layers	8
Hidden size	384
Attention heads	6 query / 2 key-value (GQA)
Head dim	64
MLP	SwiGLU, intermediate 1024
Normalization	RMSNorm (eps 1e-05)
Position encoding	RoPE (theta 10000)
Context length	512
Vocabulary	8192 (SentencePiece BPE, byte fallback)
Embeddings	tied input/output

Training


Tokens	164M (~10 per parameter)
Steps	2,500 at 65,536 tokens/step
Optimizer	AdamW (betas 0.9/0.95, wd 0.1 on matrices only)
Schedule	cosine, 200 warmup steps, peak LR 0.0006
Precision	fp16 AMP with loss scaling
Hardware	1x NVIDIA T4 (Colab free tier)

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("pythonstudentiam/tinyllm")
model = AutoModelForCausalLM.from_pretrained("pythonstudentiam/tinyllm")

messages = [{"role": "user", "content": "Write a story about a lost puppy."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=250, do_sample=True, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))

With llama.cpp

GGUF conversions are included in this repo.

llama-server -m tinyllm-Q8_0.gguf -c 512 --host 127.0.0.1 --port 8080

Limitations

This model has 15.7M parameters and a 8192-token vocabulary, trained exclusively on synthetic children's stories. Be concrete about what that means:

It only does one thing. It writes simple short stories in the TinyStories style. Anything else — code, arithmetic, factual questions, translation, summarization of arbitrary text — produces confident nonsense.
Its vocabulary is small. Words outside a children's-story vocabulary fall back to individual bytes, which it handles poorly.
Context is 512 tokens. There is no long-range coherence to be had.
No safety tuning of any kind. It has had no alignment work beyond instruction tuning on story prompts.
Quantization hurts more than usual. Small models have less parameter redundancy to absorb rounding error; Q4_K_M is measurably worse here than the usual "negligible loss" guidance for 7B+ models would suggest.

Not suitable for any production use. It is a teaching artifact.

Training data

TinyStories — synthetic short stories generated by GPT-3.5/GPT-4, constrained to the vocabulary of a 3-4 year old. Licensed CDLA-Sharing-1.0.

Instruction tuning used TinyStoriesInstruct.