Brandon-Tiny-10M-Instruct
A 10.7M parameter instruction-following language model that punches far above its weight class. Small enough to run on a PlayStation 2's Emotion Engine. Named after a Cloudflare tunnel URL that sounded like a language model.
Why This Exists
We wanted a language model small enough to run natively on a PlayStation 2 -- 32 MB of VRAM on the Emotion Engine, tested on both emulator and real hardware. Inspired by Karpathy's TinyStories, we looked for existing models small enough, and couldn't find anything decent. So we built one.
The name "Brandon Tiny" comes from a Cloudflare tunnel: one day we ran cloudflared tunnel and the random URL came back as something like sugar-alaska-brandon-tiny.trycloudflare.com. We were serving a custom agentic chat through it, and the URL looked so much like a language model name that we couldn't stop joking about it. The name stuck.
After 8 model variants, 3 architectural experiments, and one 3-phase training pipeline, Brandon Tiny turned out to be genuinely impressive for its size.
Highlights
- 10.7M parameters -- runs on a PS2 Emotion Engine (32 MB VRAM)
- Beats our own 30M model on fine-tuning loss (2.40 vs 2.61)
- Zero repetition loops across all generation tests
- 80% instruction following on 25 diverse prompts
- Trained in ~7 hours on a single RTX 3090
- 3-phase pipeline: Pretrain → Knowledge Distillation → Instruction Finetune
Model Details
Architecture
Llama 2 style decoder-only transformer with three modern enhancements:
| Spec | Value |
|---|
| Parameters | 10,706,776 |
| Dimensions | 256 |
| Layers | 24 (12 unique with block sharing) |
| Attention Heads | 8 (2 KV heads, GQA 4:1) |
| FFN Hidden | 720 (SwiGLU) |
| Vocabulary | 8,192 (SentencePiece BPE) |
| Max Sequence | 512 tokens |
| Positional | RoPE (theta=10000) |
| Normalization | RMSNorm |
| Enhancements | DenseFormer + Value Residual + Register Tokens |
Block Sharing (MobileLLM): Adjacent layer pairs share weights, giving 24 effective layers from only 12 unique parameter blocks.
DenseFormer: Depth-Weighted Averaging connects all previous layer outputs to each subsequent layer.
Value Residual: Layer 0's value projection is added to all subsequent layers, preserving early representations.
Register Tokens: 4 learnable tokens prepended to input, acting as attention sinks.
Training
3-phase pipeline, each phase addressing different aspects:
| Phase | Steps | LR Schedule | Data | Result |
|---|
| 1. Foundation Pretrain | 15K | WSD (8e-4) | 600M tokens (Wiki 40% + SmolLM 30% + Synthetic 30%) | val_loss 4.39 |
| 2. Knowledge Distillation | 7.5K | WSD (4e-4) | Same pretrain data, 30M teacher | val_loss 4.84 |
| 3. Instruction Finetune | 12K | Cosine (2e-5) | 75K examples (chat + reasoning + replay) | val_loss 2.40 |
Anti-repetition training in Phase 3: label smoothing (0.1) + unlikelihood training (0.5) + entropy regularization (0.01).
Knowledge distillation uses reverse KL divergence (mode-seeking) with temperature 2.0, following MiniPLM findings that reverse KLD is better for small students.
Inference
| Metric | Value |
|---|
| Speed | 21 tokens/sec (RTX 3090) |
| VRAM | 51.5 MB allocated |
| Model size | 42.8 MB (fp32) / 21.4 MB (bf16) |
Evaluation
Standard Benchmarks (0-shot)
| Benchmark | Brandon-Tiny-10M | Random Baseline | Delta |
|---|
| BLiMP (Grammar) | 73.3% | 50.0% | +23.3 |
| HellaSwag (Commonsense) | 32.4% | 25.0% | +7.4 |
| ARC-Easy (Science) | 30.6% | 25.0% | +5.6 |
| PIQA (Physical Intuition) | 54.7% | 50.0% | +4.7 |
| Winogrande (Coreference) | 50.3% | 50.0% | +0.3 |
| LAMBADA (Last Word) | 8.8% | 0.0% | +8.8 |
| Wikitext-2 PPL | 224.2 | -- | -- |
Custom Evaluation Suite
Tested against 7 models from our experimental series:
| Model | Params | Wikitext-2 PPL | Gen. Quality | Repetition | Instr. Following |
|---|
| Brandon-Tiny-10M | 10.7M | 224.2 | 0.947 | 0.000 | 80% |
| 10M Enhanced v2 | 10.7M | 329.2 | 0.939 | 0.000 | 80% |
| 30M v2 Original | 30.0M | 302.8 | 0.957 | 0.000 | 84% |
| 30M v2 Wiki | 30.0M | 168.6 | 0.918 | 0.000 | 72% |
Note: The 10M Optimal achieves comparable performance to 30M models with 3x fewer parameters.
Usage
Installation
1git clone https://github.com/xaskasdf/brandon-tiny
2cd brandon-tiny
3pip install torch sentencepiece pyyaml numpy
Inference
1from src.model import TinyLlama
2from src.tokenizer import Tokenizer
3
4# Load model
5model = TinyLlama.from_checkpoint('checkpoints/10m_optimal/phase3_finetune/best.pt')
6tokenizer = Tokenizer('data/tokenizer_8k.model')
7
8# Chat format
9prompt = "<|im_start|>user\nTell me a story about a cat.<|im_end|>\n<|im_start|>assistant\n"
10tokens = tokenizer.encode(prompt)
11input_ids = torch.tensor(tokens, device='cuda')
12
13output = model.generate(
14 input_ids,
15 max_new_tokens=200,
16 temperature=0.7,
17 top_p=0.9,
18 repetition_penalty=1.2,
19 no_repeat_ngram_size=3,
20 stop_tokens=tokenizer.get_stop_tokens(),
21)
22
23response = tokenizer.decode(output[0, len(tokens):].tolist())
24print(response)
Generation Examples
Identity:
User: What is your name?
Assistant: I'm Brandon Tiny, a small language model designed to help in limited environments.
Creative:
User: Tell me a short story about a cat.
Assistant: Once upon a time, there was an old but young cat named Charlie who lived in the small town of Pagima. Every day, he set out on adventures...
Reasoning:
User: What happens when ice melts?
Assistant: When ice causes a strong layer of water, the ice accumulates in the atmosphere due to the temperature and pressure...
Limitations
- Factual accuracy: The model hallucinates freely. Do NOT use for factual question answering.
- English only: Trained exclusively on English data.
- Short context: Maximum 512 tokens.
- No safety alignment: No RLHF/DPO training has been applied.
- Arithmetic: Cannot reliably perform mathematical operations.
- Vocabulary artifacts: Occasionally produces garbled or invented words.
- Standard benchmarks: Expected to score near-random on MMLU, GSM8K, and other benchmarks designed for larger models.
What This Model IS Good For
- Running a language model on a PlayStation 2 (or any device with 32+ MB VRAM)
- Research on ultra-small language models
- Understanding scaling behavior at extreme parameter budgets
- Edge/embedded deployment experiments
- Educational purposes (understanding LLM training pipelines)
- Prototyping conversational agents with minimal compute
What This Model Is NOT Good For
- Factual question answering
- Mathematical reasoning
- Professional or production use
- Any task requiring reliability or safety guarantees
Training Details
Hardware
- Single NVIDIA RTX 3090 (24 GB VRAM)
- Total training time: ~7 hours across all 3 phases
- Operating system: Windows 11
Data
Pre-training (600M tokens):
- Wikipedia English (40%)
- SmolLM Corpus (30%)
- Synthetic data generated by GPT-4o-mini (30%)
Instruction fine-tuning (75,502 examples):
- Brandon-Tiny Instruct dataset
- 57,000 curated chat instructions
- 19,944 reasoning/Chain-of-Thought examples
- ~200 pretrain replay examples (catastrophic forgetting mitigation)
Evaluation datasets:
- BLiMP - Grammatical knowledge
- HellaSwag - Commonsense reasoning
- ARC-Easy - Science reasoning
- PIQA - Physical intuition
- Winogrande - Coreference resolution
- LAMBADA - Last word prediction
- Wikitext - Perplexity evaluation
Tokenizer
- SentencePiece BPE, 8,192 vocabulary
- Special tokens:
<|im_start|>, <|im_end|>, <|bos|>, <|eos|>, <|pad|>
- Chat format: ChatML
Training Hyperparameters
Phase 1: Foundation Pre-training
- Learning rate: 8e-4 → 8e-5 (WSD schedule)
- Steps: 15,000
- Batch size: 32 × 4 gradient accumulation = effective 65K tokens/step
- Warmup: 500 steps, Stable: 70%, Decay: 20%
- Weight decay: 0.1
- Optimizer: AdamW (beta1=0.9, beta2=0.95)
- Precision: bfloat16
Phase 2: Knowledge Distillation
- Teacher: Brandon-Tiny 30M v2 pretrained (30M params)
- Method: Reverse KL Divergence
- Temperature: 2.0
- Alpha: 0.5 (soft/hard target balance)
- Learning rate: 4e-4 → 4e-5 (WSD)
- Steps: 7,500
Phase 3: Instruction Fine-tuning
- Learning rate: 2e-5 → 2e-6 (cosine)
- Steps: 12,000
- Batch size: 16 × 4 gradient accumulation
- Weight decay: 0.01
- Label smoothing: 0.1
- Unlikelihood alpha: 0.5
- Entropy regularization beta: 0.01
- Loss masking: assistant tokens only
Experimental Context
This model is the result of systematic exploration across 8 architectural variants. Key insight: training methodology matters more than parameter count at this scale. The 3-phase pipeline (Pretrain → Distill → Finetune) enabled a 10M model to outperform 30M models trained with a standard 2-phase approach.
For the full experimental report, see our
technical report.
Links
Citation
1@misc{brandon-tiny-2026,
2 title={Brandon-Tiny 10M: A 3-Phase Training Pipeline for Ultra-Small Instruction-Following Language Models},
3 author={Samuel Cortes},
4 year={2026},
5 url={https://naranjositos.tech/}
6}
Acknowledgments
Architecture and techniques inspired by:
- MobileLLM (Meta, 2024) - Block sharing, deep-narrow design
- SmolLM2 (HuggingFace, 2025) - Data-centric small model training
- MiniCPM (2024) - WSD learning rate schedule
- DenseFormer (Pagliardini et al., 2024) - Depth-Weighted Averaging
- MiniPLM (ICLR 2025) - Reverse KLD for small model distillation
