The upstream Qwen3.6-27B config declares mtp_num_hidden_layers: 1. The head in this file is the Q8_0 GGUF version of that MTP block, placed at blk.64.* for the 64-layer 27B trunk.
Why This Exists
MTP is a separate draft module, not just a flag. NVIDIA's MTP docs describe it as extra modules that predict future tokens beyond the normal next-token head. In Qwen3.6-27B, those weights live after the 64 trunk blocks.
Some fine-tune, merge, quantization, or export flows only keep the main model trunk and skip the extra MTP tensors. When that happens, the model can still run normally, but speculative decoding cannot use the trained native draft head. Depending on runtime, MTP may fail to load, silently draft from missing/random weights, or have near-zero acceptance.
Grafting the original MTP head back in can restore native MTP speculative decoding for compatible Qwen3.6-27B derivatives. It is not guaranteed to be optimal after heavy fine-tuning: the draft head was trained against the original trunk hidden states, so acceptance can fall if the tuned trunk diverged a lot.
Compatibility checklist:
Same architecture: qwen35
Same hidden size: 5120
Same FFN width: 17408
Same attention heads: 24
Same KV heads: 4
Same tokenizer/vocab and embedding/lm-head compatibility
Same 27B depth: 64 main layers, with the grafted MTP block becoming blk.64.*
Do not use this blindly on a model with changed tokenizer, changed dimensions, changed architecture, or missing matching Qwen3.6-27B metadata.
Grafting Onto A Non-MTP Qwen3.6-27B GGUF
Use a GGUF-aware graft script. Do not concatenate files.
This head can also be grafted onto a compatible Qwen3.6-40B derivative when that model is a depth-expanded descendant of the same Qwen3.6-27B family, as with the local IQ4_XS graft. Treat this as an experimental compatibility path, not as a native 40B MTP head.
The important difference is layer count. The original Qwen3.6-27B body has 64 main layers, so this donor head is stored as blk.64.*. The expanded 40B body used locally has 96 main layers, so the grafted MTP block belongs after those layers as blk.96.*.
For a 96-layer 40B body, the graft script needs to:
Verify the body is still qwen35 and is dimensionally compatible.
Copy the body tensor data unchanged.
Rename the donor MTP tensors from blk.64.* to blk.96.* while appending them.
The reason this works structurally is that the expanded 40B variant keeps the same core tensor shapes as Qwen3.6-27B: hidden size 5120, attention heads 24, KV heads 4, and FFN width 17408. Those matching dimensions make the MTP head tensors loadable after remapping from block 64 to block 96.
That does not mean quality or acceptance is guaranteed. The donor MTP head was trained to predict from Qwen3.6-27B's layer-64 hidden state, while a 96-layer 40B graft feeds it the representation after layer 96. If draft acceptance is poor, lower --spec-draft-n-max to 3 or 1 and benchmark again. A truly optimal 40B MTP head would need to be trained against the 40B trunk.
llama.cpp Setup Notes
MTP support for Qwen3.6 GGUF has moved past the original PR #22673 merge notes. A useful build now needs all of these pieces:
TurboQuant KV-cache types turbo2, turbo3, and turbo4, if you want the TurboQuant profiles documented below.
Still-current flag notes:
Use --spec-type draft-mtp, not the older --spec-type mtp.
Use --spec-draft-type-k and --spec-draft-type-v to mirror the target KV cache type you actually want for the MTP draft path.
--spec-draft-n-max 6 is the normal live depth for 27B Qwen3.6 native-MTP profiles here.
Local coding-agent profiles currently use --spec-draft-p-min 0.0 for throughput. 0.75 remains a useful stricter acceptance / regression fallback from the Unsloth guidance.
You can chain ngram speculative decoding with MTP for replay-heavy workloads, for example --spec-type ngram-mod,draft-mtp.
Fresh check on 2026-05-28: ggml-org/llama.cpp master was at b9371-1-g8ad8aef44. No newer MTP, NextN, NVFP4, FWHT, or TurboQuant change was better than the tested vs7 stack below. The most relevant newer CUDA change only gates PDL for old CUDA Toolkit versions; this build uses CUDA 13.1.
Current Tested Runtime
The old hand-merge recipe for "PR #22673 plus TurboQuant" has been superseded locally. The current tested Windows / RTX 5090 runtime is:
That vs7 runtime combines TheTom's TurboQuant KV-cache branch with the current MTP cleanup stack, the MTP draft-KV type fix, NVFP4 MTP scale tensor support, and CUDA FWHT support. In practice, this means the local tree no longer needs the old conflict-resolution recipe in this README; use the checked-out vs7 branch or recreate that patch stack directly.
Feature sanity checks:
powershell
1.\build-cuda131-sm120-vs7-fwht\bin\Release\llama-server.exe --help |2 findstr /C:"draft-mtp"/C:"ngram-simple"/C:"turbo4"34.\build-cuda131-sm120-vs7-fwht\bin\Release\test-backend-ops.exe test -b CUDA0 -o MUL_MAT_HADAMARD
CUDA 13.1 Note
For Blackwell / RTX 5090 we used CUDA 13.1, not CUDA 13.2. The Unsloth discussion and guide warn that CUDA 13.2 can produce garbled output in this area; the community note says to use CUDA 13.1 or lower until NVIDIA fixes it in a later toolkit.
The tested local build used:
text
1nvcc: Cuda compilation tools, release 13.1, V13.1.80
2MSVC: 19.44
3CMake: Visual Studio 17 2022 generator
4CUDA arch: 120, normalized by CMake/CUDA to 120a
For this machine/build, the live 27B coding-agent profiles currently stay on plain draft-mtp. Ngram + MTP remains useful for replay/copy-heavy prompts and should be selected per workload, not treated as a universal default. The reduced vs7 correctness smoke on the Unsloth 27B MTP profile initialized draft-MTP with draft KV q8_0/turbo3 and accepted 9 of 12 draft tokens.
Caveats
This is a head-only GGUF. It will not run by itself.
Grafting should be done with a GGUF-aware script.
Acceptance depends on how close the target trunk is to the original Qwen3.6-27B trunk.
Heavy fine-tunes may need MTP-head retraining for best acceptance.
Vision + MTP has been fragile in local llama.cpp testing; use text-only first.
CUDA 13.2 was intentionally avoided for this build.
