Views
No views yet
compressed-tensors, FP8_DYNAMIC W8A8) quantisation of huginnfork/Qwen3.6-27B-uncensored-heretic-v2-mtp,
attnbf16 variant: the entire self-attention path is kept in bf16 and only the MLPs are FP8. Ships a
working MTP self-speculative-decoding head.lm_head, the MTP head, the vision tower, the whole linear_attn (Gated-DeltaNet / SSM) block,
and the entire self_attn path (incl. the attention output gate fused into q_proj on Qwen3.5/3.6).
Only the ~17 B MLP params are FP8. This keeps quantisation off the multiplicative attention gate and the
16 long-range full-attention layers, at a cost of ~+1.5 GiB (~4.6 %) vs a fully-quantised FP8 build. See
recipe.yaml.kld_heretic_attnbf16.json), measured per-token on
neuralmagic/calibration (8 samples, seq 1024). This is markedly lower than a plain-attention FP8 build
of the same base.quantization_config.ignore so vLLM loads it correctly. (A bf16 MTP head
regrafted into a compressed-tensors quant is otherwise mis-loaded and yields 0 % draft acceptance — this
build fixes that.)--speculative-config '{"method":"mtp","num_speculative_tokens":3}').1vllm serve huginnfork/Qwen3.6-27B-uncensored-heretic-v2-mtp-FP8 \
2 --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
3 --max-num-seqs 32--max-num-seqs 32 (or lower) is required — Qwen3.6 is a hybrid linear-attention model whose Mamba cache
otherwise runs out of blocks at the default max_num_seqs.