miniLLM — 100M decoder-only transformer trained from scratch
A Llama-3-shaped decoder-only language model trained from scratch on FineWeb-Edu,
on a single RTX 4060 Laptop (8GB). No distillation, no pretrained weights, no
fine-tuning off someone else's base model — tokenizer, data pipeline, architecture,
and training loop are all in the repo.
Its sibling is
chessLLM, the same
architecture retrained on chess notation to test whether next-token prediction alone
builds a world model.
Architecture
| |
|---|
| Parameters | 100.7M non-embedding (125.8M total) |
| Layers | 12 |
| Attention | 12 query heads / 4 KV heads (GQA), head_dim 64 |
| d_model | 768 |
| FFN | SwiGLU, hidden 2048 |
| Position | RoPE, θ=10000 |
| Context | 1024 tokens |
| Vocab | 32,768 (byte-level BPE, trained on this corpus) |
| Norm | RMSNorm, QK-norm |
| Embeddings | Tied |
| Optimizer | Muon (2-D hidden matrices) + AdamW (rest) |
| Precision | bf16 |
| Final val loss | 3.2840 (ppl 26.7) at step 9,144/9,144 |
Training
12.44 hours on one 8GB laptop GPU, base preset, Muon lr=0.02, WSD schedule.
| |
|---|
| Steps × tokens/step | 9,144 × 131,072 = 1.20B tokens |
| Tokens per param | 11.9 (under Chinchilla's 20 — a wall-clock budget, not a compute-optimal one) |
| Throughput | 26.8k tok/s sustained, 5.6 GB of 7.6 GB VRAM |
Val loss by step: 3.83 @1.7k → 3.69 @3.4k → 3.63 @5.4k → 3.60 @7.4k → 3.28 @9.1k.
The last leg is the WSD decay phase, and most of the final drop happens there — stopping
before the decay completes leaves roughly 0.3 nats on the table.
Muon is the single biggest win. In a controlled A/B at a matched 200-step /
13.1M-token budget, Muon (lr=0.02) reached val 5.0917 against AdamW (lr=1.5e-3) at
6.0712 — −0.98 nats, 2.7× lower perplexity — for ~2% less throughput. That is an
early-training comparison, not a full-run one. Details in the repo README.
Decoding matters more than the loss curve suggests
This is the most useful practical finding, and the defaults below are not the obvious ones.
The model ranks true facts above plausible false ones 95% of the time (40
common-knowledge cloze pairs, chance = 50%). But at temperature 0.8, free generation
only stated the correct fact 30% of the time — a 65-point gap between what the model
knows and what it says. That is sampling noise, not missing knowledge.
| temperature | rep. penalty | fact% | worst repeated-4gram% | |
|---|
| 0.8 | 1.0 | 30–50% | ~74% | naive defaults |
| 0.3 | 1.25 | 67% | 5% | recommended |
| 0.0 | 1.35 | 80% | 2% | max factuality |
| 0.0 | 1.50 | 80% | 0% | over-penalised; starts padding with enumerations |
Lower the temperature for facts, raise it for variety. The repetition penalty
(CTRL-style, windowed to the last 256 tokens) is what removes the
"the National Park of Pakistan became the National Park of Pakistan" failure mode.
Below ~1.25 this model loops badly; above ~1.4 it starts avoiding words it needs.
Factual accuracy climbed monotonically through the entire run — 77.5% at step 1,713 to
95.0% at step 9,144, still rising at the end. Val loss hides this: the constant-LR phase
looks flat (−0.020 nats/1k steps) while knowledge accumulates.
Usage
Download ckpt.pt and tokenizer_v2.json — the tokenizer is not bundled in the
checkpoint and generation needs both.
1import torch
2from model import MiniLLM, ModelConfig # from github.com/amanmprojects/llm
3from tokenizers import Tokenizer
4
5ck = torch.load("ckpt.pt", map_location="cpu", weights_only=False)
6cfg = ModelConfig(**{k: v for k, v in ck["cfg"].items()
7 if k in ModelConfig.__dataclass_fields__})
8model = MiniLLM(cfg)
9model.load_state_dict(ck["model"])
10model.eval()
11
12tok = Tokenizer.from_file("tokenizer_v2.json")
13ids = torch.tensor([tok.encode("The causes of the French Revolution were").ids])
14out = model.generate(ids, 120, temperature=0.3, repetition_penalty=1.25)
15print(tok.decode(out[0].tolist()))
Or from a clone of the repo, which fetches both files for you:
1python scripts/download_weights.py text
2python sample.py --prompt "The causes of the French Revolution were"
sample.py already applies the tuned defaults above (temperature 0.3, repetition
penalty 1.25).
Limitations
- 100M parameters trained on 1.2B tokens. It writes fluent, mostly-grammatical
encyclopedic prose and knows a lot of common facts, but it confabulates specifics
freely — dates, names, and numbers should be assumed wrong unless checked.
- Base model, not a chat model. It completes text; it does not follow instructions.
A second phase adding continued pretraining plus a chat mixture was designed and is
documented in the repo, but it crashed at step 1,750 of 7,535 (laptop suspend
killed the CUDA context) and its partial checkpoint is worse than this one (val 3.469
vs 3.284). This upload is the completed run 1, which is the best checkpoint that exists.
- 1024-token context. No long-context capability.
- Sensitive to decoding settings, as the table above shows. At high temperature it
will state falsehoods it internally ranks as false.
- English, educational-web register, inherited from FineWeb-Edu. No code or math
specialization, no multilingual capability, and it carries whatever biases that corpus
contains.
- Not safety-tuned in any way. There is no RLHF, no refusal behaviour, no filtering
beyond the corpus itself.
Data
FineWeb-Edu (CC-MAIN
shards), tokenized with a byte-level BPE trained on the same corpus. Weights are
MIT-licensed; the underlying text belongs to its original authors under FineWeb-Edu's
terms.
Full code, training pipeline, and evaluation harness:
https://github.com/amanmprojects/llmThis checkpoint is inference-only — bf16, no optimizer state, so it cannot resume training.
