Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4

---

English

[!TIP] Quantized on 2026-04-21 using llm-compressor with mixed-domain calibration and sensitive-layer protection for maximum accuracy recovery.

[!IMPORTANT] Native W4A4 on DGX Spark (SM121) — confirmed working

Unlike earlier NVFP4 models on SM121, this checkpoint runs true W4A4 via FlashInfer CUTLASS NVFP4 MoE kernel (verified in vLLM logs: FlashInferCutlassNvFp4LinearKernel + FLASHINFER_CUTLASS NvFp4 MoE backend). Requires:

• vLLM 0.19.1rc1.dev374+g1174723eb or later (includes PR #37725 arch-suffix fix)
• FlashInfer ≥ 0.6.8 with SM120f compilation (PR #2650)
• CUDA ≥ 12.9

MTP speculative decoding supported — MTP layers preserved in BF16 from the original checkpoint via save_mtp_tensors_to_checkpoint.

[!WARNING] Abliteration changes the optimal speculative decoding setup — this is a known trade-off, not a defect.

This release's distinguishing feature is mixed-domain calibration (ultrachat_200k chat + Nemotron-Post-Training-Dataset-v2 reasoning, 256 samples total). The calibration recovers quantization accuracy, but it cannot undo the distribution shift introduced upstream by abliteration itself — the DFlash drafter was trained on the original Qwen3.6-35B-A3B weights, and the abliterated residual distribution no longer matches the drafter's prior, so acceptance rate drops.

Measured throughput on DGX Spark:

• DFlash (num_speculative_tokens: 15) — ~50 t/s sustained, occasional bursts up to ~100 t/s
• MTP (num_speculative_tokens: 1) — ~40 t/s sustained, occasional bursts up to ~70 t/s

Counter-intuitively, MTP with a single speculative token outperforms DFlash on this abliterated variant — MTP reuses the model's own hidden state, so it stays aligned with the abliterated distribution that the mixed-domain calibration was tuned against. Prefer --speculative-config '{"method":"mtp","num_speculative_tokens":1}' as the default; only fall back to DFlash if you specifically need it.

Model Details

Quantization Details

B (last-layer protection) incompatible with vLLM fused MoE: vLLM's CompressedTensorsMoEMethod requires all projections within a MoE block (gate/up/down × 256 experts + shared_expert) to share the same quantization scheme. Partial ignore triggers ValueError: All MoE projections need to have same quantization scheme but found multiple.

E (SpinQuant R1+R2) incompatible with multi-modal config: llm-compressor's get_head_dim only reads top-level config, not Qwen3.6's nested text_config.

Layers Preserved in BF16

Speculative Decoding

Note: On hybrid GDN architectures, MTP may hit a state-rollback bug (vLLM #39273) during high token-rejection rates. The num_speculative_tokens: 1 setting also reduces exposure to this issue.

Serving with vLLM

DGX Spark (SM121) Compatibility Notes

• Native W4A4 confirmed via FlashInfer CUTLASS NVFP4 MoE backend (no more W4A16 fallback)
• Verify in logs: Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM + Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend
• FP8 KV cache is not compatible with GDN non-causal attention; use --kv-cache-dtype auto
• DFlash requires --attention-backend flash_attn (flashinfer backend + DFlash is incompatible)
• --language-model-only skips vision encoder profiling for text-only inference
• Clear page cache before starting on UMA: sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

Known Limitations

• Per-tensor global scales for fused q_proj / k_proj / v_proj may differ, causing a vLLM warning at load time. This is inherent to the llm-compressor per-layer quantization behavior; the impact on accuracy is typically small but measurable on strict tool-calling JSON schemas.
• DFlash drafter was trained on the original Qwen3.6-35B-A3B, not the abliterated variant — acceptance rate may be lower than on the original model.

Safety Warning

Credits

• Original Model: Qwen/Qwen3.6-35B-A3B by Alibaba Qwen Team
• Abliteration: huihui-ai
• NVFP4 Quantization: YuYu1015 on NVIDIA DGX Spark (GB10)
• Quantization Tool: llm-compressor by vLLM Project
• Official Reference: RedHatAI/Qwen3.6-35B-A3B-NVFP4
• Sensitivity Analysis: Diagnosing FP4 Inference (arXiv 2603.08747)

---

繁體中文

[!TIP] 2026-04-21 量化上傳，使用 llm-compressor 搭配混合領域校準與敏感層保護，最大化精度保留。

[!IMPORTANT] DGX Spark (SM121) 原生 W4A4 — 已驗證可用

不同於早期 SM121 的 NVFP4 模型，此 checkpoint 透過 FlashInfer CUTLASS NVFP4 MoE kernel 跑真 W4A4（vLLM log 可見 FlashInferCutlassNvFp4LinearKernel + FLASHINFER_CUTLASS NvFp4 MoE backend）。需要：

• vLLM 0.19.1rc1.dev374+g1174723eb 以上（含 PR #37725 arch-suffix 修復）
• FlashInfer ≥ 0.6.8 帶 SM120f 編譯（PR #2650）
• CUDA ≥ 12.9

支援 MTP 投機解碼 — MTP 層以 BF16 從原 checkpoint 透過 save_mtp_tensors_to_checkpoint 保留。

[!WARNING] Abliteration 會改變投機解碼的最佳設定 — 這是已知取捨，非 bug。

本版本的特色是混合領域校準（ultrachat_200k 對話 + Nemotron-Post-Training-Dataset-v2 推理，共 256 樣本）。校準能恢復量化精度，但無法逆轉 abliteration 在上游造成的分佈偏移 — DFlash drafter 是以原版 Qwen3.6-35B-A3B 權重訓練，abliterated 後的殘差分佈已不再符合 drafter 的先驗，接受率因此下降。

DGX Spark 實測吞吐：

• DFlash（num_speculative_tokens: 15） — 約 50 t/s，偶爾飆至 ~100 t/s
• MTP（num_speculative_tokens: 1） — 穩定約 40 t/s，偶爾飆至 ~70 t/s

反直覺地，MTP 搭配單一投機 token 在此 abliterated 變體上表現優於 DFlash — MTP 沿用模型自身的 hidden state，與混合領域校準所針對的 abliterated 分佈保持一致。建議預設使用 --speculative-config '{"method":"mtp","num_speculative_tokens":1}'，僅在特殊需求時才改用 DFlash。

模型資訊

量化詳情

B 策略（最後層保護）與 vLLM fused MoE 不相容：vLLM 的 CompressedTensorsMoEMethod 要求 MoE block 內所有 projection（gate/up/down × 256 experts + shared_expert）必須同 scheme。Partial ignore 會觸發 ValueError: All MoE projections need to have same quantization scheme but found multiple。

E 策略（SpinQuant R1+R2）與 multi-modal config 不相容：llm-compressor 的 get_head_dim 只讀頂層 config，不讀 Qwen3.6 巢狀的 text_config。

保留 BF16 的層

投機解碼

注意：混合 GDN 架構下 MTP 可能觸發 state-rollback bug（vLLM #39273），高 rejection rate 時輸出可能退化。num_speculative_tokens: 1 也能降低觸發機率。

vLLM 部署

DGX Spark (SM121) 相容性說明

• 原生 W4A4 已確認 透過 FlashInfer CUTLASS NVFP4 MoE backend（不再退回 W4A16）
• log 驗證：Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM + Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend
• FP8 KV cache 與 GDN non-causal attention 不相容，請使用 --kv-cache-dtype auto
• DFlash 需搭配 --attention-backend flash_attn（flashinfer backend + DFlash 不相容）
• --language-model-only 跳過視覺編碼器 profiling，加速純文字推理啟動
• UMA 架構啟動前請先清除 page cache：sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

已知限制

• Fused q_proj / k_proj / v_proj 的 per-tensor global scale 可能不一致，vLLM 載入時會印警告。這是 llm-compressor per-layer 量化的固有行為，一般精度影響輕微，但在嚴格 tool-calling JSON schema 下可能可測得。
• DFlash drafter 是以原版 Qwen3.6-35B-A3B 訓練，非 abliterated 變體 — 接受率可能較原版低。

安全警告

致謝

• 原始模型：Qwen/Qwen3.6-35B-A3B，Alibaba Qwen 團隊
• 去審查：huihui-ai
• NVFP4 量化：YuYu1015，於 NVIDIA DGX Spark (GB10) 上完成
• 量化工具：llm-compressor，vLLM Project
• 官方參考：RedHatAI/Qwen3.6-35B-A3B-NVFP4
• 敏感度分析：Diagnosing FP4 Inference (arXiv 2603.08747)