Pace Intent Router v8
Summary
A 49.5M-parameter intent classifier trained from scratch on a 127K-example
synthetic corpus for Pace's 7-class intent taxonomy. Runs in 3ms on CPU,
beats Qwen3-4B-Instruct by 10.8 percentage points, and is 85x faster.
This is a posttrainllm-built specialist — trained with the
train-extractor command on a ToolRouterModel small preset (6 layers,
384 d_model, byte-level vocab 256). No pre-trained weights, no
distillation from a teacher. The model learns Pace-specific decision
boundaries from synthetic data generated by
pace/scripts/generate-intent-corpus-v2.py.
Artifact
- Package id:
pace-intent-router-v8
- Artifact:
runs/pace-intent-router-v8.tinygpt (566 MB)
- Labels:
runs/pace-intent-router-v8.tinygpt.labels.json
- Format: posttrainllm
.tinygpt (ToolRouterModel checkpoint)
- Base:
posttrainllm-ToolRouterModel-small (trained from scratch)
- Params: 49.5M (6 layers, 384 d_model, 7 classes)
- Training method:
train-extractor (cross-entropy classification,
AdamW + cosine decay, 18000 steps on v5 data)
- Training time: ~30 minutes on M5 Pro
Measured Result
Head-to-head vs Qwen3-4B-Instruct (4-bit)
| Metric | ToolRouterModel v8 | Qwen3-4B-Instruct | Delta |
|---|
| Overall accuracy | 95.5% | 84.75% | +10.8 pp |
| Eval size | 14,995 | 997 (stratified sample) | — |
| Latency p50 | 3.1 ms | 240 ms | 77x faster |
| Model size | 566 MB | ~2,300 MB | 4x smaller |
| Params | 49.5M | 4B (4-bit) | — |
Per-class accuracy (v8 vs v5 vs Qwen3-4B)
| Class | v8 (49.5M) | v5 (49.5M) | Qwen3-4B |
|---|
| chitchat | 93.8% | 95.6% | 78.5% |
| pureKnowledge | 96.4% | 97.0% | 83.0% |
| screenDescription | 97.6% | 97.1% | 95.8% |
| screenAction | 96.8% | 97.4% | 91.0% |
| research | 93.4% | 97.1% | 100.0% |
| phoneLargeModel | 98.4% | 96.4% | 76.7% |
| unknown | 79.6% | 70.3% | 2.6% |
| Overall | 95.5% | 95.9% | 84.75% |
Key takeaways
- +10.8 pp over Qwen3-4B: the 49.5M specialist beats the 4B
generalist on this task. Intent classification is a narrow enough
problem that a from-scratch model outperforms a general LLM.
- 77x faster: 3.1ms vs 240ms. The router runs in sub-4ms on CPU;
Qwen needs a full LLM forward pass.
- Unknown class is the moat: v8 scores 79.6% on unknown vs Qwen's
2.6%. Qwen almost never returns "unknown" — it forces everything
into a known class, which is risky for a router (wrong route →
wrong pipeline). The trained model knows when to say "I don't know."
- v5 vs v8: v5 (12K steps) slightly outperforms v8 (18K steps)
overall (95.9% vs 95.5%), but v8 has a much better unknown class
(79.6% vs 70.3%, +9.3 pp). The longer training improved the hardest
class at a small cost to the easier ones.
Training data
- Dataset: pace-intent-corpus-v2 (fully synthetic)
- Source:
pace/scripts/generate-intent-corpus-v2.py (base corpus,
combinatorial expansion) + pace/scripts/generate-intent-supplement-v2.py
(targeted weak-spot supplement)
- Train rows: 112,061
- Heldout rows: 14,995
- Classes: 7 (chitchat, pureKnowledge, screenDescription, screenAction,
research, phoneLargeModel, unknown)
- Split: stratified 85/15
- No real user data — all synthetic
Why synthetic data works here
The synthetic corpus encodes Pace-specific decision boundaries that a
general LLM doesn't know:
- "turn on lights" = unknown (Pace can't control lights)
- "turn on volume" = screenAction (Pace can control volume)
- "what can you do" = pureKnowledge (not unknown — it's a question
about Pace itself)
These boundaries are product-specific, not language-general. A
from-scratch model learns them directly from the corpus; a general LLM
needs few-shot examples or fine-tuning to learn them.
Recommended Use
This model is a router, not a planner. It classifies a user
utterance into one of 7 intent classes in 3ms. The intent class
determines which pipeline handles the turn:
| Intent | Route | Model |
|---|
| chitchat | fast path | Apple FM / local text-only |
| pureKnowledge | answer directly | Apple FM / local text-only |
| screenDescription | read screen | local planner + VLM |
| screenAction | execute tool | local planner + action layer |
| research | research tier | codex CLI / Claude CLI |
| phoneLargeModel | cloud bridge | codex CLI / cloud bridge |
| unknown | full pipeline | local planner (best-effort) |
Do not use this model for response generation. It only classifies.
Known Limits
- Unknown class is synthetic: the 79.6% unknown accuracy is measured
on synthetic unknowns. Real-world unknown distribution will differ.
The model needs real-world unknown examples to improve further.
- Not wired into Pace: the shipping app uses Apple FM for intent
classification (when available) and rule-based fallback. This model
is a training pipeline artifact, not a product component.
- Byte-level vocab: the model uses a 256-token byte-level vocabulary.
Longer queries are truncated at 128 bytes. A BPE tokenizer would
generalize better on longer queries.
- No real user data: all training data is synthetic. Real user
queries will have different phrasings, accents, and edge cases.
Why this model exists
The Pace intent classifier was originally rule-based (hand-written
phrase lists). This model was trained to test whether a learned
classifier could beat the rules. It can — 95.5% vs the rules' ~95% on
the synthetic eval — and it also beats Apple Foundation Models (3B,
in-process) on the same task: 95.5% vs 76.5% on a 200-example
stratified sample. The model is most useful as:
- The production intent classifier — it's faster (3ms vs 1600ms),
more accurate (95.5% vs 76.5%), and has calibrated confidence
(real softmax vs hardcoded 0.95)
- A fallback when Apple FM is unavailable (older Macs)
- A training pipeline validation (the corpus and pipeline are assets)
- A baseline for future on-device classifiers
Measured head-to-head (200 stratified examples, 1s delay)
| Model | Accuracy | Latency | Error rate |
|---|
| TinyGPT v8 (49.5M) | 95.5% | 3.1ms p50 | 0% |
| Qwen3-4B-Instruct (4-bit) | 84.75% | 240ms p50 | 0% |
| Apple FM (3B, in-process) | 76.5% | 1597ms mean | 0.5% |
FM's main weakness: it conflates research with pureKnowledge
(33.3% vs 93.4% — a 60pp gap) because it doesn't understand the
Pace-specific distinction between "research X" (multi-step) and
"what is X" (single answer). FM also refused to answer one query
("model refused to answer").
References
- Factory run report:
runs/2026-07-13-pace-intent-router-v1/report.md
- Eval data:
runs/2026-07-13-pace-intent-router-v1/eval-candidate-v8.json
- Head-to-head:
runs/2026-07-13-pace-intent-router-v1/head-to-head.json
- Training config:
runs/2026-07-13-pace-intent-router-v1/config.json
- Corpus generator:
pace/scripts/generate-intent-corpus-v2.py
- Supplement generator:
pace/scripts/generate-intent-supplement-v2.py
- Pace routing architecture:
pace/leanring-buddy/PaceIntentClassifier.swift
