Qwen3.6-40B-Deckard — AWQ (GEMM) 4-bit for vLLM, with a working MTP head

⚡ Built for 2× RTX 2080 Ti (Turing / SM75) + NVLink

A 40B reasoning model at ~42–44 tok/s with working MTP speculative decoding on a pair of 2019 gaming cards — faster than the same model's MTP GGUF in llama.cpp (~35 tok/s) on the same hardware. If you have old Turing cards gathering dust, this is for you.

Reference hardware (what all numbers below were measured on)

What this is

• Base: DavidAU/Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking — a 27B→40B width-preserving layer expansion (hidden 5120, 96 decoder blocks: 72 GatedDeltaNet linear-attention + 24 full-attention, interval 4). Reasoning ("Thinking") model; uncensored ("Heretic") finetune — use responsibly.
• Quantization: 4-bit AWQ, version: gemm (AutoAWQ export layout), served with vLLM's awq_marlin kernel. Produced with Intel AutoRound 0.14.2 in RTN mode (iters=0, no gradient rounding optimization — see Honesty notes). bits=4, group_size=128, sym=False (asymmetric, runtime zero-points).
• MTP head: transplanted from a Qwen3.6-27B donor (QuantTrio/Qwen3.6-27B-AWQ-6Bit), kept in BF16 (15 tensors, 0.85 GB, model-mtphead.safetensors). The 40B expansion preserved the 27B's width — hidden 5120, intermediate 17408, 24 attention heads / 4 KV heads, head_dim 256, vocab 248320 are identical — so the donor head drops in unchanged. Same transplant idea as PiehSoft's inject_mtp_40b.py, done in the safetensors/HF path. The head was not fine-tuned on the 40B.

Precision layout (what is 4-bit vs what stays BF16)

Measured performance (single stream, greedy unless noted)

• Methodology: 520-token generations with ignore_eos, decode rate computed as a delta between a long and a short run (excludes prompt/TTFT). TTFT on short prompts ≈ 0.4 s.
• MTP draft acceptance (transplanted, untuned head): ≈78% average on greedy fresh-context; per-position ≈ 0.88 / 0.60 / 0.42 at depth 3; mean acceptance length ≈ 2.7–2.9 tokens per step.
• Comparison point: the same model's MTP GGUF in llama.cpp (NCCL build, -sm tensor, NVLink) does ~35 tok/s on the same cards. This vLLM path is ~20% faster single-stream, and unlike llama.cpp it batches concurrent requests.
• Why contexts differ: weights take ~17.6 GiB/GPU, leaving ~1.9 GiB for KV at gpu_memory_utilization 0.94 → ~27k max without MTP; the safe-sync MTP path reserves ~1 GiB more → ~12k. KV-cache fp8 does not engage on SM75; vLLM's --kv-offloading-size is a prefix-cache offload and does not extend a single request's context.

⚠️ Two SM75 pitfalls (read before serving)

1. Never set VLLM_ALLOW_MAMBA_SPEC_FULL_CUDAGRAPH=1 with MTP on this model. Full-CUDA-graphing the GatedDeltaNet (linear-attention) state update during speculation replays stale recurrent state and silently corrupts generation — repetition loops, hallucinated prompts, empty outputs — while short trivial prompts can still look fine, which makes it easy to miss. Keep the default 0, and use VLLM_SM75_SPEC_SYNC_MODE=safe (not nosync, which corrupts the same way). Symptom → cause was verified by A/B on identical prompts.
2. --disable-custom-all-reduce is required on this 2× 2080 Ti setup: the custom all-reduce kernel fails with CUDA graphs (custom_all_reduce.cuh: invalid argument). NCCL over NVLink takes over the TP all-reduce; cost is ~2 tok/s.

Serving — recommended (MTP, ~42–44 tok/s, 12k ctx)

• CUDA-graph capture size must be a multiple of num_speculative_tokens+1 (here 4), or the engine aborts with "No valid cudagraph sizes".
• Plain mode (24k ctx, ~24 tok/s): drop --speculative-config, use capture size 1 and --max-model-len 24576.
• Long context (131k, ~3 tok/s): add --cpu-offload-gb 8 --max-model-len 131072.
• This is a vision-language checkpoint architecture served text-only (--language-model-only); no vision weights are shipped.
• The base model is a thinking model: responses arrive as reasoning_content + content (hence --reasoning-parser qwen3). Give it generous max_tokens.

Reproducing the quant (outline)

1. Quantize the BF16 base with AutoRound (bits=4, group_size=128, sym=False, iters=0, format="auto_awq"), keeping the BF16 list above via layer_config={name:{"bits":16}}. Two patches are needed for this VL-architecture checkpoint: force the LLM (not MLLM) calibration path, and fix the checkpoint-name reverse mapping so tensors export as model.language_model.* (matching how vLLM loads the composite arch).
2. Transplant the MTP head: copy the 15 mtp.* tensors from the 27B donor into a separate safetensors shard and register them in model.safetensors.index.json. No renaming needed — mtp.* lives top-level.
3. Config: base model's config.json + quantization_config {quant_method: awq, bits: 4, group_size: 128, version: gemm, zero_point: true, modules_to_not_convert: [...]} (see this repo's config for the exact list). mtp_num_hidden_layers: 1 is already present in the base config.

Honesty notes / limitations

• The quant is RTN (round-to-nearest, no per-block gradient optimization). It measures coherent across RU/EN chat, code, and reasoning prompts, but a calibrated AutoRound run (iters=200) would likely be marginally more accurate. May follow later.
• The MTP head is a dimensional transplant, not a trained one — acceptance (~78% greedy) is good but below a natively trained head; creative/high-temperature text will accept less and gain less speed.
• The base model is a community franken-expansion of Qwen3.6-27B and an uncensored finetune. Quality characteristics of the base are inherited as-is; the base model's license and usage terms apply.
• Context ceilings above are specific to 2× 22 GB cards; more VRAM → linearly more KV/context.

Credits

• DavidAU — the base Opus-Deckard 40B model.
• QuantTrio — the Qwen3.6-27B AWQ whose MTP head was transplanted.
• PiehSoft — the MTP-transplant method (inject_mtp_40b.py) and acceptance write-up for the GGUF path.
• Intel AutoRound — the quantization toolkit.
• weicj / vLLM-2080ti-definitive — the SM75 vLLM build that makes any of this possible on Turing.