Huihui-Qwen3.8-27B-abliterated-NVFP4

🔄 Re-quantized 2026-08-24 from the upstream Latest update 3

Upstream narrowed the ablation to layers 18–51 only (previously everything from layer 15 up), leaving the rest of the network untouched, to retain more of the original model's performance. MTP and the vision tower are unmodified.

I diffed the shards before re-baking: 6 of 18 changed, and they are exactly the shards holding layers 15–17 and 52–63. The changelog matches the bits.

This is not a cosmetic update. The reasoning_effort: xhigh thinking-runaway that the previous revision failed is gone — see the gotcha section. If you were pinning medium because of it, you no longer have to.

The previous NVFP4 revision remains available at commit history / revision= if you need it.

• MTP draft head preserved in bf16 and wired into the index — speculative decoding works out of the box.
• bf16 kept for: lm_head, the vision tower, the DeltaNet conv1d, and the MTP head. Everything else is NVFP4 W4A4.
• Built for SM120 / Blackwell with vanilla vLLM v0.22.0 — compressed-tensors is auto-detected, no --quantization flag.
• Quantized with the bf16 held in CPU RAM while llm-compressor's sequential pipeline onloaded one subgraph at a time to a single GPU: 2 min 10 s end to end, ~10.5 GB of VRAM. (A 4-GPU fully-resident attempt OOMs — 55.6 GB into 65 GB leaves no room for activations. The CPU-held path is both faster and leaves the rest of the boards free.)

Measured — this revision, 2026-08-24

Serve

⚠️ Gotchas

• The xhigh thinking-runaway is fixed in this revision. On the previous revision, reasoning_effort at xhigh would sometimes never terminate the <think> phase: it grew past 19,000 characters, degenerated into repeating a single line, and the token budget was gone before any answer was emitted — 1 failure in a 9-case gate (French/English long-form, temperature 0 and 0.7). The same gate on this revision passes 9 of 9, with thinking between 1.5k and 8k characters and every run finishing on stop. medium remains a sensible default for long-form work, but it is no longer a workaround for a defect.
  Worth knowing what these modes actually are: they are not three levels of capability, they are three system prompts. Reading the chat template — xhigh injects "think carefully through the task, validate key assumptions, consider plausible alternatives…", low injects "keep your thinking brief and focused, moving directly to the conclusion", and medium injects nothing at all — it is simply the model with no deliberation instruction.
  "chat_template_kwargs": {"reasoning_effort": "medium"}
• The 15 mtp.* modules are listed in quantization_config.ignore — do not remove them. If vLLM treats the bf16 MTP head as NVFP4 the draft breaks silently: 0% acceptance and slower than no MTP.
• W4A16 (NVFP4A16) does not serve on this architecture in vLLM 0.22 (gptq_marlin_repack: size_n=24 not divisible by tile_n_size=64). W4A4 only.
• Long single-file code generation drops a closing paren roughly 1–2 times in 14 regardless of sampling temperature (measured with a JS parser on the base model). Put a syntax check in the loop rather than tuning temperature.
• Text-only serving shown above; the vision tower ships in bf16 but multimodal serving was not benchmarked here.

Recipe