Qwen3.5-397B-A17B Heretic GGUF v2 MTP Quants
This repo contains my v2 GGUF quants of Qwen3.5-397B-A17B Heretic with MTP support included as a standard feature.
These files are intended for recent llama.cpp builds with Qwen3.5 MTP support.
Source model
These quants were made from safetensors downloaded from:
RadicalNotionAI/Qwen3.5-397B-A17B-heretic
That repo is the current Hugging Face home of the Heretic model previously associated with trohrbaugh.
The quantization source used here was a local BF16 GGUF conversion of that safetensors release.
Important note on MTP
In my testing, MTP gives roughly 20-30% faster TG at the cost of about 10% slower PP. In practice, that means it is usually faster for shorter prompts, but very long prompts (for example, around 60k tokens) can still take longer overall in wall-clock time.
Heretic does not appear to retain the full native MTP tensor set in a directly usable form, so the MTP block in this repo is grafted rather than preserved natively from the Heretic release.
In practice, these files use:
- a Heretic quantized trunk
- a grafted MTP block derived from vanilla Qwen3.5-397B-A17B
- either full
Q8_0 MTP tensors or a smaller mixed-precision MTP variant (MTPmix)
The grafting workflow used here is based on Buzz’s GGUF MTP transplant script:
Quantizations in this repo
This repo includes:
IQ3_XXS_MTPMIXIQ3_XXS_MTPQ8_0IQ4_XS_MTPMIXIQ4_XS_MTPQ8_0Q3_K_MTPMIXQ4_K_M_MTPMIXQ4_K_M_MTPQ8_0
I have not been able to get Q2 quants passing my basic tests so far, so they are not included here.
Naming note
These are not pure stock single-format quants.
Examples:
IQ4_XS_MTPMIX = IQ4_XS Heretic trunk + mixed-precision grafted MTP blockQ4_K_M_MTPQ8_0 = Q4_K_M Heretic trunk + Q8_0 grafted MTP block
The suffix after the trunk name matters.
Perplexity (WikiText-2 test)
PPL results are expected within the next 48 hours.
| Quant | PPL |
|---|
IQ3_XXS_MTPMIX | 4.2404 |
IQ3_XXS_MTPQ8_0 | 4.2400 |
IQ4_XS_MTPMIX | TBA |
IQ4_XS_MTPQ8_0 | 4.0953 |
Q3_K_MTPMIX | 4.0900 |
Q4_K_M_MTPMIX | TBA |
Q4_K_M_MTPQ8_0 | 4.0862 |
KLD
KLD results are coming soon.
Notes
MTPQ8_0 variants preserve the draft block more faithfully, but they are larger and can reduce or erase net speedup if VRAM is tight.MTPmix variants are intended to recover more practical throughput by shrinking the draft block.- Draft acceptance depends on both quantization and trunk-to-draft compatibility.
- Because these are grafted MTP variants rather than native preserved Heretic MTP exports, acceptance behavior may differ from models whose MTP block remained aligned with the trunk during training.
Intended use
These files are mainly intended for:
llama.cpp-compatible local inference- speculative decoding / MTP experiments
- testing trunk-quality vs draft-quality tradeoffs
- comparing practical inference behavior across large 397B MoE quants
Credits
- Qwen for the original model family
- RadicalNotionAI for the Heretic release
- the llama.cpp project for GGUF and MTP runtime support
- Buzz for the GGUF MTP transplant script used as the basis for the graft workflow
- Ubergarm and other community quantization work that helped inform practical quant choices
