This is a supplement for Step-3.7-Flash-based quants without MTP tensors.
This repository contains an MTP-only subset of stepfun-ai/Step-3.7-Flash
which provides the draft model for speculative decoding in the GGUF format.
It accelerates token generation using speculative decoding with the draft model
from the original Step-3.7-Flash model. In most cases, this is sufficient to
accelerate derivative models even if this draft model is not trained from them.
Note that however, the performance metrics heavily depend on the derivative
model you use, your machine and your MTP settings.
Benchmark it before blindly trusting it.
Using this Model
As grafting MTP tensors into an existing GGUF file is not easy,
I describe how to use this model as a separate draft model file.
It is easier than grafting but memory-inefficient as it does not share
some tensors with the original model.
If you use llama-server, you may configure like this:
The if block right after
# verify tensor name presence and identify potentially missing files
is commented out.
Handle
chkhsh == "5841594bd6a8eeecd7207aeec6570831cc97ffaeba51e908bdaf560113177bae"
case as deepseek-v3.
The first modification is performed because the author of this repository
downloaded only a subset of the full Step model while the original
convert_hf_to_gguf.py expects the full model.
The second modification is because convert_hf_to_gguf.py does not support
Step-3.7-Flash pre-tokenizer.
The --mtp option of convert_hf_to_gguf.py is the crucial part of this
conversion process because this option does exactly what the author expects:
create an MTP-only GGUF subset.
For additional quantization, the llama-quantize tool (llama.cpp)
is used without options.
