About this repository: this is the 4-bit (W4A16) version of nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-F32 (Qwen3.6 27B multimodal, hybrid architecture with GatedDeltaNet linear attention + full attention). It is an AutoRound quantization ready for production serving on vLLM: the language model runs in INT4 (Marlin kernel), while the vision tower and the MTP (Multi-Token Prediction, used for speculative decoding) were fully preserved in BF16 — meaning this checkpoint accepts image/video inputs and speculative decoding with no functional loss compared to the original model. Size: ~19.5 GB (vs ~108 GB for the original FP32).

4-bit quantization (W4A16, symmetric, group_size 128) of nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-F32, produced with AutoRound via llm-compressor (compressed-tensors pack-quantized format → Marlin kernel on vLLM).

What is quantized and what stays in BF16

Component	Precision
Language model — 400 `Linear` layers (attn, MLP, GDN projections)	INT4 W4A16 g128
Vision tower (`model.visual.*`, 333 tensors)	BF16 (bit-identical to the original)
MTP (`mtp.*`, 15 tensors — speculative decoding)	BF16 (bit-identical to the original)
`linear_attn.in_proj_a/b` (out_dim 48 < group 128), norms, conv1d, embeddings, lm_head	BF16

Calibration

Dataset: neuralmagic/LLM_compression_calibration (128 samples, messages + the model's own chat template)
AutoRound: 200 iterations, batch size 4, seqlen 2048

Usage with vLLM

bash

vllm serve DoktorMincs/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-W4A16-AutoRound \
  --tensor-parallel-size 2 --max-model-len 65536 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

Speculative decoding (MTP) works out of the box: the Qwen3_5MTP draft is loaded from this same checkpoint.
Vision works (image/video inputs via the OpenAI-compatible API).
This is a thinking/chat model — use the chat endpoint (raw completion prompts may return empty output).

Suggested startup script

Note: this configuration was tested with 4× RTX 3060 12 GB (48 GB total VRAM).

bash

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export OMP_NUM_THREADS=4
export CUDA_VISIBLE_DEVICES=0,1,2,3
export PYTORCH_ALLOC_CONF=expandable_segments:True
export NCCL_P2P_DISABLE=1
export FLASHINFER_DISABLE_VERSION_CHECK=1

exec vllm serve \
    /root/models/DoktorMincs/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-W4A16-AutoRound \
    --served-model-name "qwen3.6-27b" \
    --tensor-parallel-size 4 \
    --max-model-len 163840 \
    --override-generation-config '{"temperature": 0.6, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0, "repetition_penalty": 1.0}' \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --trust-remote-code \
    --enable-prefix-caching \
    --max-num-batched-tokens 8192 \
    --gpu-memory-utilization 0.88 \
    --disable-custom-all-reduce \
    --max-num-seqs 16 \
    --host 0.0.0.0 \
    --port 3434 \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

Validation (vLLM 0.26, 2026-07-27)

Coherent text generation via chat ✓
MTP spec decode active (Qwen3_5MTP resolved, embeddings shared with target) ✓
Vision answers image inputs correctly ✓