The following layers are not quantized to preserve model quality:
Layer
Reason
lm_head
Output head, sensitive to quantization noise (auto-excluded by shape)
embed_tokens
Input embeddings (auto-excluded by shape)
linear_attn.in_proj_a/b
Gated DeltaNet low-rank projections — shapes not divisible by 32 (auto-skipped)
linear_attn.* (other)
GDN/DeltaNet layers, may output zeros if quantized
mtp.fc
Multi-Token Prediction projection — quantized then dequantized back to BF16
Why dequantize mtp.fc? vLLM's Qwen3_5MTP loader expects unquantized weights for the MTP projection layer. AutoRound quantizes it during the main pass; we then convert it back to BF16 in a post-processing step so MTP speculative decoding works natively.
Speculative Decoding
This model supports two speculative decoding methods:
MTP (uses built-in weights, recommended for low-concurrency):
Note: On hybrid GDN architectures, MTP may hit a state-rollback bug (vLLM #39273) at high token-rejection rates. Reduce num_speculative_tokens to 1 if you observe degraded output quality. The DFlash drafter was trained on the original Qwen3.6-27B; acceptance rate on the abliterated variant may be lower.
Use --quantization auto_round for Marlin INT4 kernel (SM121 compatible via SM120 binary compat)
FP8 KV cache is not compatible with GDN non-causal attention layers; use --kv-cache-dtype auto
NVFP4 falls back to Marlin W4A16 on SM121 (missing cvt.e2m1x2 PTX instruction) — INT4 AutoRound has no such fallback issue
Runtime FP8 (--quantization fp8) is not compatible with DFlash (drafter inherits FP8 config and crashes)
Dense 27B is bandwidth-bound on DGX Spark's 273 GB/s LPDDR5X — expect ~15-20 tok/s without speculative decoding
--language-model-only skips vision encoder profiling for text-only inference
--performance-mode throughput enables CUDA graphs and kernels for throughput optimization
Clear page cache before starting on UMA: sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
Performance Note
Dense 27B activates all parameters per token (vs MoE 35B-A3B's 3B active), making it more accurate but slower than the 35B-A3B variant on the same hardware. Use this checkpoint when quality matters more than throughput; use 35B-A3B for high-throughput multi-user deployments.
Safety Warning
This model has safety filtering removed (abliterated) and may generate sensitive, controversial, or inappropriate content. Users are solely responsible for all consequences arising from its use. Please ensure usage complies with local laws and ethical standards. Not suitable for public-facing or production applications.
