mini-LLM — a 30M-parameter transformer trained overnight on a laptop
A decoder-only GPT written from scratch in plain PyTorch. No transformers, no accelerate, no
fastai — the model, the tokenizer training and the training loop are all in the repository.
Trained on one MacBook Pro (M5 Pro, 24 GB, MPS) in a single overnight run.
Results
| Metric | Value |
|---|
| Best val loss | 1.1527 (perplexity 3.17) |
| Loss at initialisation | 9.01 = ln(8192), the uniform-over-vocabulary baseline |
| Steps | 7,561 |
| Tokens seen | 0.74B (1.4 epochs of TinyStories V2) |
| Training time | ~9 hours of compute at 23–25K tokens/s |
| Parameters | 29,893,120 (25.7M non-embedding) |
Architecture
Modern rather than 2017: what separates this from the original transformer paper is where most of
the interest lies.
| |
|---|
| Layers | 8 |
| Model width | 512 |
| Heads | 8 (head_dim 64) |
| Context | 512 tokens |
| Vocabulary | 8,192, byte-level BPE trained on the corpus itself |
| Positional encoding | RoPE (rotary), not learned embeddings |
| Normalisation | RMSNorm, pre-norm |
| Feed-forward | SwiGLU, hidden 1408 (8/3 × width, rounded to a multiple of 64) |
| Attention | F.scaled_dot_product_attention, causal |
| Embeddings | tied — one matrix serves both input and output |
| Dropout | 0.0 |
The 8,192-token vocabulary is a deliberate choice: GPT-2's 50,257 would put a 25M-parameter
embedding table inside a 30M-parameter model.
Usage
1git clone https://github.com/vous99/mini-llm && cd mini-llm
2pip install -r requirements.txt
3
4python -c "
5from huggingface_hub import hf_hub_download
6import shutil, os
7os.makedirs('ckpt', exist_ok=True); os.makedirs('data', exist_ok=True)
8shutil.copy(hf_hub_download('vous99/mini-llm', 'best.pt'), 'ckpt/best.pt')
9shutil.copy(hf_hub_download('vous99/mini-llm', 'tokenizer.json'), 'data/tokenizer.json')
10"
11
12python sample.py
13python sample.py --prompt "Once upon a time, a little robot" --n 3
14python chat.py # terminal REPL
15python serve.py # browser playground on 127.0.0.1:8890
tokenizer.json is required. It is the BPE trained alongside the model; without it the
weights emit token ids, not text.
Training details
| |
|---|
| Optimizer | AdamW, lr 6e-4, betas (0.9, 0.95), grad clip 1.0 |
| Weight decay | 0.1 on matrices, 0 on 1-D parameters (norms) |
| LR schedule | linear warmup 200 steps → cosine decay to 10% of peak |
| Batch | 24 × 8 gradient accumulation × 512 tokens = 98,304 tokens per step |
| Precision | bfloat16 autocast |
Train and validation loss stayed within 0.01–0.03 of each other for the whole run — no
overfitting, which is why dropout is 0.
Limitations
- TinyStories only. The corpus is synthetic children's stories using a deliberately small
vocabulary. The model writes fluent, coherent short stories in that register and nothing else.
It has no world knowledge, cannot answer questions and cannot follow instructions.
- 512-token context.
- No instruction tuning, no RLHF. This is a base model in the most literal sense.
- Not safe for production. It is a study artifact for understanding how a transformer trains.
Checkpoint contents
best.pt is a torch.save dict with model, cfg (the GPTConfig), iter, best_val and
val_loss. Load it with sample.py from the GitHub repository.
