[!IMPORTANT] Native W4A4 on DGX Spark (SM121) — confirmed working

This checkpoint runs true W4A4 via FlashInfer CUTLASS NVFP4 kernel (verified in vLLM logs: FlashInferCutlassNvFp4LinearKernel). Requires:

vLLM 0.19.1rc1.dev374+g1174723eb or later (includes PR #37725 arch-suffix fix)

FlashInfer ≥ 0.6.8 with SM120f compilation (PR #2650)

CUDA ≥ 12.9

MTP speculative decoding supported — use method: qwen3_next_mtp (dense-model MTP, distinct from the MoE mtp method).

NVFP4 W4A4 quantization of huihui-ai/Huihui-Qwen3.6-27B-abliterated, optimized for NVIDIA DGX Spark (GB10 SM121) with the FlashInfer CUTLASS FP4 kernel.

Model Details

Item	Value
Architecture	Dense (27B), 64 layers — 16 × (3 × Gated DeltaNet → FFN + 1 × Gated Attention → FFN)
Base model	Qwen/Qwen3.6-27B
Fine-tuned by	huihui-ai (abliteration)
Quantized by	YuYu1015
Model size	~28.6 GB (NVFP4 + BF16 MTP/GDN layers, vs ~71.9 GB BF16 original)
Context length	Up to 262,144 tokens
Thinking mode	Supported (`enable_thinking: true/false`)
Tool calling	Supported (`qwen3_coder` parser)
MTP	Built-in MTP weights (`qwen3_next_mtp` method)
DFlash	Compatible with z-lab/Qwen3.6-27B-DFlash

Quantization Details

This model uses a three-strategy stack (ACD) on top of the llm-compressor official flow:

Strategy	Description
A. Official baseline	`Qwen3ForCausalLM` + `save_mtp_tensors_to_checkpoint` (preserves MTP weights after quantization)
C. Mixed-domain calibration	`ultrachat_200k` (128 chat) + `Nemotron-Post-Training-Dataset-v2` (128 reasoning) = 256 total
D. Sweet-spot hyperparameters	`num_calibration_samples=256`, `max_seq_length=4096` (quality > quantity)

B (last-layer protection) incompatible with vLLM fused kernels: Partial ignore of linear layers within a transformer block triggers ValueError: All projections need to have same quantization scheme but found multiple.

E (SpinQuant R1+R2) incompatible with multi-modal config: llm-compressor's get_head_dim only reads top-level config, not Qwen3.6's nested text_config.

Item	Value
Method	llm-compressor (main) + compressed-tensors (main)
Scheme	NVFP4 W4A4 (E2M1 + FP8 per-group scaling, group size 16)
Format	compressed-tensors
Calibration datasets	HuggingFaceH4/ultrachat_200k (128) + nvidia/Nemotron-Post-Training-Dataset-v2 (128)
Calibration samples (total)	256
Calibration sequence length	4096
Hardware	NVIDIA DGX Spark (GB10, 128GB unified memory)
Environment	`transformers>=5.0,<6` + `llm-compressor` main + `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`

Layers Preserved in BF16

Layer pattern	Reason
`re:.*lm_head`	Output head, sensitive to quantization noise
`re:.*embed_tokens$`	Input embeddings
`re:visual.` / `re:model.visual.`	Vision encoder
`re:.*mlp.gate$`	FFN gate (dense model; must stay BF16)
`re:.linear_attn.`	Gated DeltaNet (Mamba) layers — may output zeros if quantized
`mtp.*` (all MTP weights)	Reattached in BF16 via `save_mtp_tensors_to_checkpoint` after quantization

Speculative Decoding

This model supports two speculative decoding methods:

MTP (built-in weights, recommended default for this abliterated variant — see warning at top):

--speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'

DFlash (separate drafter, recommended for single-user / low-concurrency on non-abliterated variants):

--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-27B-DFlash", "num_speculative_tokens": 15}'

Note: On hybrid GDN architectures, MTP may hit a state-rollback bug (vLLM #39273) during high token-rejection rates. Reduce num_speculative_tokens to 1 if you observe degraded output quality.

Serving with vLLM

bash

vllm serve /path/to/model \
    --quantization compressed-tensors \
    --served-model-name qwen3.6-27b \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --attention-backend flash_attn \
    --kv-cache-dtype auto \
    --gpu-memory-utilization 0.80 \
    --max-model-len 131072 \
    --max-num-batched-tokens 16384 \
    --max-num-seqs 8 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --performance-mode throughput \
    --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' \
    --trust-remote-code \
    --language-model-only

DGX Spark (SM121) Compatibility Notes

Native W4A4 confirmed via FlashInfer CUTLASS NVFP4 kernel
Verify in logs: Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM
FP8 KV cache is not compatible with GDN non-causal attention; use --kv-cache-dtype auto
Dense model fits more comfortably on 128 GB UMA than the 35B BF16 weight — higher max-model-len is achievable
--language-model-only skips vision encoder profiling for text-only inference
Clear page cache before starting on UMA: sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

Known Limitations

Per-tensor global scales for fused q_proj / k_proj / v_proj may differ, causing a vLLM warning at load time. Impact on accuracy is typically small but measurable on strict tool-calling JSON schemas.
DFlash drafter was trained on the original Qwen3.6-27B, not the abliterated variant — acceptance rate may be lower than on the original model.

Safety Warning

This model has safety filtering removed (abliterated) and may generate sensitive, controversial, or inappropriate content. Users are solely responsible for all consequences arising from its use. Please ensure usage complies with local laws and ethical standards. Not suitable for public-facing or production applications.

Credits

Original Model: Qwen/Qwen3.6-27B by Alibaba Qwen Team
Abliteration: huihui-ai
NVFP4 Quantization: YuYu1015 on NVIDIA DGX Spark (GB10)
Quantization Tool: llm-compressor by vLLM Project
Reference Quantization: sakamakismile/Qwen3.6-27B-NVFP4
Sensitivity Analysis: Diagnosing FP4 Inference (arXiv 2603.08747)

繁體中文

[!TIP] 使用 llm-compressor 搭配混合領域校準與敏感層保護量化，最大化精度保留。

[!IMPORTANT] DGX Spark (SM121) 原生 W4A4 — 已驗證可用

此 checkpoint 透過 FlashInfer CUTLASS NVFP4 kernel 跑真 W4A4（vLLM log 可見 FlashInferCutlassNvFp4LinearKernel）。需要：

vLLM 0.19.1rc1.dev374+g1174723eb 以上（含 PR #37725 arch-suffix 修復）

FlashInfer ≥ 0.6.8 帶 SM120f 編譯（PR #2650）

CUDA ≥ 12.9

支援 MTP 投機解碼 — 使用 method: qwen3_next_mtp（dense 模型專用，有別於 MoE 的 mtp 方法）。

huihui-ai/Huihui-Qwen3.6-27B-abliterated 的 NVFP4 W4A4 量化版，針對 NVIDIA DGX Spark (GB10 SM121) 最佳化，使用 FlashInfer CUTLASS FP4 kernel。

模型資訊

項目	數值
架構	Dense（27B），64 層 — 16 × (3 × Gated DeltaNet → FFN + 1 × Gated Attention → FFN)
基礎模型	Qwen/Qwen3.6-27B
微調者	huihui-ai（abliteration）
量化者	YuYu1015
模型大小	~28.6 GB（NVFP4 + BF16 MTP/GDN 層，原版 BF16 約 71.9 GB）
Context 長度	最高 262,144 tokens
思考模式	支援（`enable_thinking: true/false`）
工具呼叫	支援（`qwen3_coder` parser）
MTP	內建 MTP 權重（`qwen3_next_mtp` 方法）
DFlash	相容 z-lab/Qwen3.6-27B-DFlash

量化詳情

此模型在 llm-compressor 官方流程上堆疊三項策略（ACD）：

策略	說明
A. 官方基線	`Qwen3ForCausalLM` + `save_mtp_tensors_to_checkpoint`（量化後保留 MTP 權重）
C. 混合領域校準	`ultrachat_200k`（128 對話）+ `Nemotron-Post-Training-Dataset-v2`（128 推理）共 256
D. 黃金比例參數	`num_calibration_samples=256`、`max_seq_length=4096`（品質 > 數量）

B 策略（最後層保護）與 vLLM fused kernel 不相容：對 transformer block 內的 linear 層做 partial ignore 會觸發 ValueError: All projections need to have same quantization scheme but found multiple。

E 策略（SpinQuant R1+R2）與 multi-modal config 不相容：llm-compressor 的 get_head_dim 只讀頂層 config，不讀 Qwen3.6 巢狀的 text_config。

項目	數值
方法	llm-compressor（main）+ compressed-tensors（main）
方案	NVFP4 W4A4（E2M1 + FP8 逐群縮放，群組大小 16）
格式	compressed-tensors
校準資料集	HuggingFaceH4/ultrachat_200k（128）+ nvidia/Nemotron-Post-Training-Dataset-v2（128）
校準樣本總數	256
校準序列長度	4096
量化硬體	NVIDIA DGX Spark（GB10, 128GB 統一記憶體）
環境	`transformers>=5.0,<6` + `llm-compressor` main + `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`

保留 BF16 的層

層 pattern	原因
`re:.*lm_head`	輸出頭，對量化雜訊敏感
`re:.*embed_tokens$`	輸入嵌入
`re:visual.` / `re:model.visual.`	視覺編碼器
`re:.*mlp.gate$`	FFN 門控（dense 模型；必須 BF16）
`re:.linear_attn.`	Gated DeltaNet (Mamba) 層 — 量化後可能輸出零
`mtp.*`（所有 MTP 權重）	量化後透過 `save_mtp_tensors_to_checkpoint` 以 BF16 重新掛回

投機解碼

本模型支援兩種投機解碼方式：

MTP（內建權重，此 abliterated 變體的建議預設 — 詳見頂部警告）：

--speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'

DFlash（獨立 drafter，建議於非 abliterated 版本 / 單用戶 / 低併發使用）：

--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-27B-DFlash", "num_speculative_tokens": 15}'

注意：混合 GDN 架構下 MTP 可能觸發 state-rollback bug（vLLM #39273），高 rejection rate 時輸出可能退化。若遇輸出品質下降，請將 num_speculative_tokens 降為 1。

vLLM 部署