This is a supplement for Qwen3.8-2.4T-A95B-based models (including fine-tunes and
abliterated models) without MTP tensors.
This repository contains an MTP-only subset of Qwen/Qwen3.8-2.4T-A95B
which provides the draft model for speculative decoding in the GGUF format.
It accelerates token generation using speculative decoding with the draft model
from the original Qwen model. In most cases, this is sufficient to accelerate
Qwen-based derivative models even if this draft model is not trained from them.
Note that however, the performance metrics heavily depend on the derivative
model you use, your machine and your MTP settings.
Benchmark it before blindly trusting it.
Using this Model
It can be used in two ways:
As a separate draft model file (Method 1)
As a donor for grafting the draft model into a Qwen-based model
(Method 2; Recommended)
Method 1: Separate Draft Model File
It is easy to begin with but memory-inefficient as it does not share
some tensors with the original model.
If you find the draft model can accelerate a Qwen-based model you use, grafting
the draft model (Method 2) is recommended (note: switching to Method 2 may
slightly change the acceptance rate).
If you use llama-server, you may configure like this:
With: Patched conversion/base.py
The if block right after # verify tensor name presence and identify potentially missing files is commented out.
This modification is performed because the author of this repository downloaded
only a subset of the full Qwen model while the original convert_hf_to_gguf.py
expects the full model.
The --mtp option of convert_hf_to_gguf.py is the crucial part of this
conversion process because this option does exactly what the author expects:
create an MTP-only GGUF subset.
For additional quantization, the llama-quantize tool (llama.cpp) is used as-is.
