You can deploy this model and start earning money today!

h1st0ry3D/
qwen38-27b-ud-iq2-xxs-mtp

Name: qwen38-27b-ud-iq2-xxs-mtp
Availability: InStock

Views

No views yet

Qwen3.8-27B UD-IQ2_XXS — MTP (speculative decoding) config

Benchmarks and llama-swap config for running Qwen3.8-27B at UD-IQ2_XXS (9.0 GB) with llama.cpp Multi-Token Prediction (MTP) speculative decoding on a single NVIDIA RTX 4070 12 GB.

Key fact: the MTP head is already in the GGUF

The unsloth/Qwen3.8-27B-GGUF → Qwen3.8-27B-UD-IQ2_XXS.gguf file already contains the embedded NextN/MTP draft head (blk.64.nextn.eh_proj/enorm/hnorm/ shared_head_norm.weight). No separate draft file and no ~50 GB HF-source build step is required — just serve it with --spec-type draft-mtp.

Verified on llama.cpp b10360:

common_speculative_init_result: creating MTP draft context against the target model
slot print_timing: draft acceptance = 0.71.. (mean len ~3.1)

llama-swap entry

Drop the block in qwen38-mtp-llama-swap.yaml into your llama-swap.yaml (models section). Flags: --spec-default --spec-type draft-mtp --spec-draft-n-max 3 --reasoning-preserve.

Results (RTX 4070 12 GB, ngl 99, q4_0 KV, reasoning off)

ctx	decode tok/s (MTP)	vs no-MTP	peak VRAM
8K	68.6	~1.8×	9.86 GB
16K	68.5	~2.0×	10.02 GB
32K	69.7	~2.2×	10.48 GB

Draft acceptance ~0.71 (mean len 3.1). llama-bench (lmx) cannot measure speculative MTP decode — values are manual llama-server print_timing.

HumanEval (full 164, temp 0.2):

MTP + reasoning off: 0.95 (subset n=80) — matches the non-MTP baseline 0.8841, i.e. MTP is transparent (verified speculation).
MTP + reasoning auto (deployed entry): 0.4878 — this drop is the --reasoning auto setting (Qwen3.8 loops at low temp with reasoning on), not MTP. Use reasoning off for coding.

Why a reduced context

MTP roughly doubles decode but the draft context adds KV. On 12 GB the MTP entry is capped at 32K (10.48 GB) for stability; the non-MTP long-ctx variant reaches 150K.

h1st0ry3D/qwen38-27b-ud-iq2-xxs-mtp