Views
No views yet
| What is MTP? Multi-Token Prediction (MTP) is a speculative decoding technique where a small "draft head" predicts multiple future tokens in parallel. The main model then verifies these drafts in a single forward pass, accepting correct predictions for up to 2–4× speedup in practice. |
| Injection Method The official Qwen3.5-9B ships with a 4-layer MTP head (~243M params). During QwenPaw fine-tuning, the MTP head weights were stripped — only the config placeholder mtp_num_hidden_layers: 1 remained. |
| Recovery Process 1. Download official Qwen3.5-9B base model weights 2. Extract MTP layer weights (15 tensors starting with mtp.)3. Merge into QwenPaw-Flash-9B safetensors 4. Convert to GGUF with convert_hf_to_gguf.py (with MTP support)5. Quantize with llama-quantize |
| Why This Works The MTP head is a lightweight 4-layer MLP decoder that maps the main model's last hidden state to future token logits. It sits entirely in speculative decoding space — the main model's weights are unchanged, so no fine-tuning is needed. The head simply needs to exist with compatible dimensions for llama.cpp's --spec-type draft-mtp to activate. |
| Type | Qwen3_5ForConditionalGeneration (multimodal with vision encoder) + MTP spec head |
| Main Model | ~9B parameters |
| MTP Head | ~243M parameters (2.7% overhead) |
| Layers | 32 (hybrid: Gated DeltaNet + Gated Attention) + 4 MTP decoder layers |
| Context Length | 262,144 tokens |
| Speculative Decoding | --spec-type draft-mtp with --spec-draft-n-max 2 |
| MTP Acceptance Rate | ~50% (measured on heretic version) |
| File | Size | Type | Notes |
|---|---|---|---|
QwenPaw-Flash-9B-MTP-BF16.gguf | 17.14 GB | BF16 | Full precision, reference quality |
QwenPaw-Flash-9B-MTP-Q8_0.gguf | 9.11 GB | Q8_0 | ~8.5 bpw, near-lossless |
QwenPaw-Flash-9B-MTP-Q6_K.gguf | 7.04 GB | Q6_K | ✅ Recommended, best value |
QwenPaw-Flash-9B-MTP-Q4_K_M.gguf | 5.38 GB | Q4_K_M | Compact, best size/quality tradeoff |
mmproj-QwenPaw-Flash-9B-heretic-BF16.gguf | 0.86 GB | BF16 | Vision encoder (multimodal) |
| With MTP Speculative Decoding |
llama-server -m QwenPaw-Flash-9B-MTP-Q6_K.gguf \
-ngl 99 -fa on -c 8192 \
--spec-type draft-mtp --spec-draft-n-max 2 \
--host 0.0.0.0 --port 8088 |
| Without MTP (fallback) |
# Just omit spec args — works as a normal GGUF
llama-server -m QwenPaw-Flash-9B-MTP-Q6_K.gguf \
-ngl 99 -fa on -c 8192 \
--host 0.0.0.0 --port 8088 |
--spec-type draft-mtp. The MTP head is a lossless copy from Qwen3.5-9B — no training involved.