This is a supplement for Qwen3.8-27B-based models (including fine-tunes and
abliterated models) without MTP tensors.
This repository contains an MTP-only subset of Qwen/Qwen3.8-27B
which provides the draft model for speculative decoding in the GGUF format.
It accelerates token generation using speculative decoding with the draft model
from the original Qwen model. In most cases, this is sufficient to accelerate
Qwen-based derivative models even if this draft model is not trained from them.
Note that however, the performance metrics heavily depend on the derivative
model you use, your machine and your MTP settings.
Benchmark it before blindly trusting it.
Using this Model
It can be used in two ways:
As a separate draft model file (Method 1)
As a donor for grafting the draft model into a Qwen-based model
(Method 2; Recommended)
Method 1: Separate Draft Model File
It is easy to begin with but memory-inefficient as it does not share
some tensors with the original model.
If you find the draft model can accelerate a Qwen-based model you use, grafting
the draft model (Method 2) is recommended (note: switching to Method 2 may
slightly change the acceptance rate).
If you use llama-server, you may configure like this:
The output of convert.py must be specified as the model file name
(DO NOT use --model-draft in this case).
--spec-type draft-mtp enables the draft model transplanted into the main one.
Additional Quantization (Q4_K_M, Q6_K and Q8_0)
Quantized GGUF files are provided so that deploying the draft model easier.
To maximize the accuracy, tensor types are based on Unsloth quants
(see the table below):
This Repo
Unsloth
Q4_K_M
UD-Q4_K_XL
Q5_K_M (not provided)
UD-Q5_K_XL
Q6_K
UD-Q6_K_XL
Q8_0
UD-Q8_K_XL
However, since UD-Q5_K_XL and UD-Q6_K_XL has exactly the same
quantization on MTP-related tensors, the Q5_K_M version is not
provided on this repository.
It is not required to match the quantization level.
For instance, you may pair Q6_K-quantized draft model with
the Q4_K_S-quantized main model.
The --mtp option of convert_hf_to_gguf.py is the crucial part of this
conversion process because this option does exactly what the author expects:
create an MTP-only GGUF subset.
For additional quantization, the llama-quantize tool (llama.cpp)
is used without options.
