This model was specifically created to run on consumer GPUs with 16GB VRAM. By utilizing q4_0 KV cache quantization, it allows pushing the context length up to 105k tokens.

⚠️ Note: This model is designed exclusively for nVidia GPUs and is based on the advanced KS quants developed by ikawrakow from the ik_llama.cpp repository.

!!!UPDATE!!!

I added a new quantization version which keeps the same file size but moves the model's weight from knowledge layers to logic layers. This results in a slight hit to PPL but should improve the model's performance on coding tasks. The new version is Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KS. I also added a version with an MTP head.

Quantization Details & Imatrix File

Quantization Base: KS Quants (ik_llama.cpp).
Imatrix File Used: Provided by mradermacher.
Other Tested Imatrix Files:
- bartowski – yielded significantly worse results.
- ubergarm – yielded comparable results.
- If you find or generate a better Imatrix file, please let me know in the Community tab!

I have also included the script I used for the quantization process in this repository. If you have any concerns regarding malware or security, you can download the official BF16 weights from Unsloth and quantize the model yourself using the provided script.

secret recipe Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KSS (test11)

text

# 1. Base Tensors & Embeddings
^output\.weight$ -> iq5_k
^output_norm\.weight$ -> f32
^token_embd\.weight$ -> iq4_ks

# 2. Exceptions (High precision for the first block's attention gate)
^blk\.0\.attn_gate\.weight$ -> iq6_k

# 3. Feed-Forward Networks (FFN)
# Aggressive compression (iq4_kss) for middle blocks 5-51
^blk\.([5-9]|[1-4][0-9]|50|51)\.ffn_(down|up)\.weight$ -> iq4_kss
.*ffn_(down|up)\.weight$ -> iq4_ks
.*ffn_gate\.weight$ -> iq4_ks

# 4. Attention Layers
.*attn_v\.weight$ -> iq5_ks
.*attn_(qkv|q|k|output)\.weight$ -> iq4_kss
.*attn_gate\.weight$ -> iq4_ks

# 5. Mamba / SSM Layers
.*ssm_out\.weight$ -> iq4_kss
.*ssm_(alpha|beta)\.weight$ -> iq4_ks
.*ssm_conv1d\.weight$ -> f32
.*ssm_a$ -> f32
.*ssm_dt\.bias$ -> f32

# 6. All Normalization Layers (Kept in f32 for stability)
.*norm\.weight$ -> f32

secret recipe Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KS (test29)

text

# Hybrid Sandwich quantization — from quantize_hybrid_sandwich_logic_test29.sh
# Pattern: alternating fused QKV (with SSM) / split QKV (no SSM) blocks
#   - Blocks n%4 == 0,1,2 (n<60): fused QKV + SSM
#   - Blocks n%4 == 3:            split Q (iq4_kss) / K / V (iq5_ks) / output — no SSM
#   - Blocks 60-62: fused QKV + SSM (no iq4_kss FFN)
#   - Blocks 63:  split QKV — no SSM

# 1. Base Tensors & Embeddings
^output\.weight$ -> iq5_k
^output_norm\.weight$ -> f32
^token_embd\.weight$ -> iq4_ks

# 2. Exceptions — first blocks & tail blocks (higher precision gates)
^blk\.0\.attn_gate\.weight$ -> q8_0
^blk\.([1-2]|6[01])\.attn_gate\.weight$ -> iq6_k
^blk\.62\.attn_gate\.weight$ -> q8_0

# 3. Feed-Forward Networks (FFN)
# Aggressive compression (iq4_kss) for middle blocks 4-59
^blk\.([4-9]|[1-5][0-9])\.ffn_(down|up)\.weight$ -> iq4_kss
.*ffn_(down|up)\.weight$ -> iq4_ks
.*ffn_gate\.weight$ -> iq4_ks

# 4. Attention Layers
# 4a. Fused QKV blocks: 0-2, 4,8,12..56, 60-62
^blk\.(([0-2])|([0-4][048])|(5[26])|(6[0-2]))\.attn_qkv\.weight$ -> iq4_ks

# 4b. Split QKV blocks: 3,7,11,15..59,63
^blk\.(([37])|([1-5][13579])|(63))\.attn_q\.weight$ -> iq4_kss
^blk\.(([37])|([1-5][13579])|(63))\.attn_k\.weight$ -> iq4_ks
^blk\.(([37])|([1-5][13579])|(63))\.attn_v\.weight$ -> iq5_ks
^blk\.(([37])|([1-5][13579])|(63))\.attn_output\.weight$ -> iq4_ks

# 4c. Default attention gate
.*attn_gate\.weight$ -> iq4_ks

# 5. Mamba / SSM Layers (only in fused QKV blocks)
.*ssm_out\.weight$ -> iq4_ks
.*ssm_(alpha|beta)\.weight$ -> iq4_ks
.*ssm_conv1d\.weight$ -> f32
.*ssm_a$ -> f32
.*ssm_dt\.bias$ -> f32

# 6. All Normalization Layers (Kept in f32 for stability)
.*norm\.weight$ -> f32

Quantization Recipe Comparison

Aspect	Original (test11)	New (test29)
Block 0 attn_gate	`iq6_k`	`q8_0`
Blocks 1-2 attn_gate	`iq6_k`	`iq6_k`
FFN `iq4_kss` range	blocks 5-51	blocks 4-59
Attention pattern	uniform `iq4_kss` for all qkv/q/k	alternating fused/split per block
Fused QKV blocks	—	`iq4_ks` (blocks 0-2, 4,8,12..56, 60-62)
Split Q blocks	`iq4_kss` (all)	`iq4_kss` (blocks 3,7,11..59,63 only)
Split V blocks	`iq5_ks` (all)	`iq5_ks` (blocks 3,7,11..59,63 only)
SSM `ssm_out`	`iq4_kss`	`iq4_ks`
Blocks 60-62	not specified	fused QKV + SSM, `iq4_ks` FFN
Block 62 attn_gate	—	`q8_0`
Blocks 60-61 attn_gate	—	`iq6_k`

Benchmark Results & Real-World Impressions

The model was heavily tested in daily production workflows for several days. It runs much faster (1.5x-1.75x) and more reliably than the previous iteration—completely eliminating the issue of "blank outputs", while the search-replace functionality works flawlessly.

Qwen Benchmark: Successfully passed the performance evaluations on qwen3-6-27b-benchmark.vercel.app.
Needle In A Haystack: Successfully evaluated with satisfying results across the full 100k context window.
Comparison: In direct testing, this model performs slightly better than my previous variant: Qwen3.6-27B-i1-IQ4_XS-GGUF.

Perplexity (PPL) — Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KSS (test11)

Perplexity evaluations were conducted focusing exclusively on the KV Cache quantization setup (q4_0), as this is the primary target use case:

bash

wget [https://www.gutenberg.org/files/2600/2600-0.txt](https://www.gutenberg.org/files/2600/2600-0.txt) -O pg19.txt \

./llama-perplexity -m Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KSS.gguf -f pg19.txt -c 65536 --chunks 32 -ngl 99 -khad -vhad -ctk q4_0 -ctv q4_0 -fa 1 -b 512 -ub 512

Test Log Output:

text

perplexity: calculating perplexity over 12 chunks, n_ctx=65536, batch_size=512, n_seq=1
perplexity: 71.10 seconds per pass - ETA 14.22 minutes
[1]6.6897,[2]7.0032,[3]7.1989,[4]7.3327,[5]7.4816,[6]7.3770,[7]7.4325,[8]7.4378,[9]7.4754,[10]7.5192,[11]7.5669,[12]7.4040,

Final estimate: PPL over 12 chunks for n_ctx=65536 = 7.4040 +/- 0.02773

Perplexity (PPL) — Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KS (test29)

./llama-perplexity -m Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KS.gguf -f /mnt/Samsung4TB/models/pg19.txt -c 65536 --chunks 32 -ngl 99 -khad -vhad -ctk q4_0 -ctv q4_0 -fa 1 -b 512 -ub 256

Test Log Output:

text

[1]6.6926,[2]7.0049,[3]7.2043,[4]7.3382,[5]7.4861,[6]7.3838,[7]7.4411,[8]7.4459,[9]7.4857,[10]7.5303,[11]7.5779,[12]7.4131,

Final estimate: PPL over 12 chunks for n_ctx=65536 = 7.4131 +/- 0.02774

PPL Comparison:

Test	Recipe	PPL (n_ctx=65536)	Margin of Error
test11	Original v1 (attn_qkv-IQ4_KSS)	7.4040	+/- 0.02773
test29	Hybrid Sandwich v2	7.4131	+/- 0.02774

The difference of ~0.009 is well within the margin of error, indicating statistically equivalent perplexity between both quantization strategies.

Production Deployment Configuration (ik_llama.cpp)

Below is the verified and stable execution configuration used for llama-server. It utilizes the corrected chat template (chat.template) by ex-arman68, sourced from froggeric/Qwen-Fixed-Chat-Templates.

bash

llama-server \
        -m "$MODEL_PATH" \
        -a Qwen3.6-27B \
        --ctx-size 105000 \
        --chat-template-file chat_template.jinja \
        --n-gpu-layers 99 \
        --cache-type-k q4_0 \
        --cache-type-v q4_0 \
        --batch-size 512 \
        --ubatch-size 256 \
        --flash-attn on \
        --no-mmap \
        --host 0.0.0.0 \
        --port 8081 \
        --reasoning on \
        --reasoning-format deepseek \
        -t 8 \
        --parallel 1 \
        -khad \
        -vhad \
        --chat-template-kwargs '{"preserve_thinking": true}' \
        --defrag-thold 0.3 \
        --jinja \
        --cont-batching \
        --temp 0.15 \
        --top-k 1 \
        --min-p 0.1 \
        --keep -1 \
        --repeat-last-n 512 \
        --repeat-penalty 1.05

Versions of the model with MTP have also been prepared. i1_MTP with iq4_ks quants and MPT with q8_0. Here is an example command to run the model with MTP:

bash

llama-server \
        -m "$MODEL_PATH" \
        -a Qwen3.6-27B \
        --ctx-size 50000 \
        --chat-template-file chat_template.jinja \
        --n-gpu-layers 99 \
        -mtp \
        --spec-type mtp \
        --draft-n 3 \
        --cache-type-k q4_0 \
        --cache-type-v q4_0 \
        --batch-size 512 \
        --ubatch-size 256 \
        --flash-attn on \
        --no-mmap \
        --host 0.0.0.0 \
        --port 8081 \
        --reasoning on \
        --reasoning-format deepseek \
        -t 8 \
        --parallel 1 \
        -khad \
        -vhad \
        --chat-template-kwargs '{"preserve_thinking": true}' \
        --defrag-thold 0.3 \
        --jinja \
        --cont-batching \
        --temp 0.15 \
        --top-k 1 \
        --min-p 0.1 \
        --keep -1 \
        --repeat-last-n 512 \
        --repeat-penalty 1.05

cHunter789/Qwen3.6-27B-i1-IQ4_KS-GGUF

Qwen3.6-27B-i1-IQ4_KS-GGUF

Quantization Details & Imatrix File

Quantization Recipe Comparison

Benchmark Results & Real-World Impressions

Perplexity (PPL) — Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KSS (test11)

Perplexity (PPL) — Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KS (test29)

Production Deployment Configuration (ik_llama.cpp)

cHunter789/
Qwen3.6-27B-i1-IQ4_KS-GGUF