This quant uses a custom mixed-precision recipe designed specifically for the Qwen3.6-27B dense architecture to maximize quality within a ~5.2 BPW budget. Attention layers use non-linear mapping (IQ4_NL), SSM state arrays are pinned to F32, and the FFN bottleneck (ffn_down) is boosted to IQ5_KS along with the first and last 8 "sandwich" layers.

Perplexity

Measured against wiki.test.raw with n_ctx=512 over 580 chunks:

Quant	Size	PPL
Hybrid IQ4_KS	16.6 GiB	6.9267 +/- 0.04554

*Note: Lower is better.

Quantization Recipe

The custom ruleset used for the mixed-precision tensor overrides:

bash

custom="
# SSM State Logic
blk\..*\.ssm_alpha\.weight=f32
blk\..*\.ssm_beta\.weight=f32
blk\..*\.ssm_out\.weight=q8_0

# 1. Non-linear mapping strictly for attention
blk\..*\.attn_.*\.weight=iq4_nl

# 2. Sandwich boost (First 8 / Last 8) -> iq5_ks for ALL FFN tensors
blk\.[0-7]\.ffn_.*\.weight=iq5_ks
blk\.(5[6-9]|6[0-3])\.ffn_.*\.weight=iq5_ks

# 3. Global bottleneck boost -> iq5_ks for remaining ffn_down
blk\..*\.ffn_down\.weight=iq5_ks

# 4. Fallback -> iq4_ks for remaining ffn_gate / ffn_up
blk\..*\.ffn_.*\.weight=iq4_ks

# 5. High precision anchors
token_embd\.weight=q8_0
output\.weight=q8_0
"

custom=$(
  echo "$custom" | grep -v '^#' | \
  sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
)

How This Was Made (Reproduction Steps)

Because generating an imatrix directly from a BF16 model requires massive system RAM, we use a Q8_0 intermediate step to generate the imatrix without running out of memory.

1. Convert HF Safetensors to BF16 GGUF

bash

python llama.cpp/convert_hf_to_gguf.py \
  --outtype bf16 \
  --outfile ./Qwen3.6-27B-BF16.gguf \
  /path/to/Qwen3.6-27B/

2. Quantize to Q8_0 (For Imatrix Generation)

bash

./ik_llama.cpp/build/bin/llama-quantize \
  ./Qwen3.6-27B-BF16.gguf \
  ./Qwen3.6-27B-Q8_0.gguf \
  Q8_0 16

3. Generate the Imatrix Note: GGML_CUDA_NO_PINNED=1 is used to prevent system RAM exhaustion on 24GB VRAM setups.

bash

GGML_CUDA_NO_PINNED=1 ./ik_llama.cpp/build/bin/llama-imatrix \
  -m ./Qwen3.6-27B-Q8_0.gguf \
  -f /path/to/ubergarm-imatrix-calibration-corpus-v02.txt \
  -o Qwen3.6-27B-imatrix.dat \
  --ctx-size 512 \
  -t 11 \
  --fit

4. Quantize BF16 to Hybrid IQ4_KS We use the original BF16 model here with the Q8_0-generated imatrix for maximum fidelity. IQ4_KS is passed as the base type.

bash

./ik_llama.cpp/build/bin/llama-quantize \
  --imatrix ./Qwen3.6-27B-imatrix.dat \
  --custom-q "$custom" \
  ./Qwen3.6-27B-BF16.gguf \
  ./Qwen3.6-27B-IQ4_KS.gguf \
  IQ4_KS 16

5. Test Perplexity

bash

wget https://huggingface.co/datasets/ikawrakow/validation-datasets-for-llama.cpp/resolve/main/wiki.test.raw.gz
gunzip wiki.test.raw.gz

./ik_llama.cpp/build/bin/llama-perplexity \
  -m ./Qwen3.6-27B-IQ4_KS.gguf \
  -f ./wiki.test.raw \
  -c 512 \
  --chunks 580 \
  -ngl 99 \
  -t 1

Quick Start Inference

Requires ik_llama.cpp.

bash

./ik_llama.cpp/build/bin/llama-server \
  -m ./Qwen3.6-27B-IQ4_KS.gguf \
  -c 131072 \
  -ngl 99

RDson/Qwen3.6-27B-IQ4_KS-GGUF

Qwen3.6-27B Hybrid IQ4_KS GGUF

Perplexity

Quantization Recipe

How This Was Made (Reproduction Steps)

Quick Start Inference

RDson/
Qwen3.6-27B-IQ4_KS-GGUF