Qwen3.6-27B-AEON-Ultimate-Uncensored — GGUF (UD Quants)
Unsloth Dynamic-style (UD) GGUF quantizations of
AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-BF16
Every quant uses per-tensor overrides (sensitivity-driven) + importance matrix (multi-domain calibration). All SSM recurrence tensors are preserved at source precision. MTP speculative decoding and vision (mmproj) are preserved.
Quant Comparison
| File | Quant | Size | tg t/s | PPL | KL mean | KL max | KL p99.9 |
|---|
| F16 | F16 | 50.9 GB | 30.9 | 5.7215 | — | — | — |
| UD-Q8_0 | Q8_0 | 34.7 GB | 44.7 | 5.7055 | 0.0029 | 5.23 | 0.22 |
| UD-Q6_K | Q6_K | 30.6 GB | 49.1 | 5.7046 | 0.0045 | 7.64 | 0.33 |
| UD-Q5_K_M | Q5_K_M | 28.7 GB | 46.9 | 5.7589 | 0.0111 | 5.09 | 2.04 |
| UD-IQ4_XS | IQ4_XS | 25.9 GB | 56.5 | 5.7630 | 0.0236 | 4.53 | 1.97 |
Benchmarked on
NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM), llama.cpp fork (
a4501150/llama.cpp), pp=512, tg=128.
What Makes These Different
SSM Recurrence Preservation
Qwen3.6 is a hybrid GatedDeltaNet + attention model. 48 of 64 layers use a recurrent SSM where quantization error compounds across token positions. All SSM recurrence tensors are preserved at source precision (F16) — never quantized.
| Tensor | Count | Precision | Rationale |
|---|
ssm_alpha, ssm_beta | 96 | F16 | State update projections — error accumulates in recurrence |
ssm_out | 48 | F16 | Output projection feeds directly into residual stream |
ssm_a, ssm_conv1d, ssm_dt, ssm_norm | 192 | F32 | Small state tensors (llama-quantize keeps 1D/small tensors at F32) |
attn_qkv (SSM input projection) | 48 | F16 | Highest measured KL sensitivity |
attn_gate (SSM gate projection) | 48 | F16 | Second-highest measured KL sensitivity |
Per-Tensor Sensitivity Analysis
Each tensor group was probed by quantizing only that group to Q4_0 while keeping the rest at F16, then measuring KL divergence. The override generator assigns precision based on measured sensitivity:
| Precision | Tensor Groups | Override Count |
|---|
| F16 | SSM recurrence, norms, biases, MTP layer | 512 |
| F16 | All attention tensors (attn_qkv, attn_gate, attn_v, attn_q, attn_k, attn_output), ffn_down edge | 173 |
| Base quant | FFN middle layers, FFN edge gate/up, embeddings | ~181 |
685 total overrides — every non-FFN tensor has an explicit precision assignment. No dependence on llama-quantize's internal promotion rules.
Multi-Domain Calibration + GPU Imatrix
Calibrated on a balanced mix across 4 domains from 13 HF datasets:
| Domain | Token Budget | Sources |
|---|
| General | 1M | ultrachat, OpenHermes, COIG-CQIA, LongAlpaca, pg19, froggeric/imatrix |
| Code | 750K | Magicoder-Evol-Instruct-110K |
| Reasoning | 750K | OpenMathInstruct-2, OpenR1-Math-220k |
| Agentic | 500K | glaive-function-calling-v2, xlam-function-calling-60k, hermes-function-calling-v1 |
Special tokens from source datasets are stripped automatically. Samples are kept whole — never truncated mid-conversation.
The importance matrix is generated with a PyTorch GPU-native generator (src/generate_imatrix.py) at 65,536 context — uses forward hooks to accumulate squared activations on GPU with zero PCIe D2H copies. Supports multi-GPU via device_map="auto".
Per-domain imatrices are merged with equal weights (DI-MATRIX approach).
MTP + Vision Preserved
- MTP (Multi-Token Prediction): Draft head (blk.64) pinned at F16. Use
--spec-type draft-mtp --spec-draft-n-max 3 for ~1.5-2x faster generation.
- Vision: mmproj file contains the full vision encoder. Use
--mmproj flag with llama-server for image/video understanding.
Files
| File | Description | Size |
|---|
Qwen3.6-27B-AEON-UD-Q8_0.gguf | Highest quality quantization | 34.7 GB |
Qwen3.6-27B-AEON-UD-Q6_K.gguf | Recommended — best quality/size | 30.6 GB |
Qwen3.6-27B-AEON-UD-Q5_K_M.gguf | Balanced | 28.7 GB |
Qwen3.6-27B-AEON-UD-IQ4_XS.gguf | Smallest, for constrained VRAM | 25.9 GB |
Qwen3.6-27B-AEON-mmproj-F16.gguf | Vision encoder (use with --mmproj) | 885 MB |
imatrix_merged.dat | Importance matrix for requantization | 13 MB |
Usage
llama-server (recommended)
1# Q6_K with YaRN 512k context, 5 concurrent slots, MTP + vision
2llama-server \
3 -m Qwen3.6-27B-AEON-UD-Q6_K.gguf \
4 --mmproj Qwen3.6-27B-AEON-mmproj-F16.gguf \
5 -ngl 99 \
6 --flash-attn \
7 -c 524288 \
8 --parallel 5 \
9 --cache-type-k q8_0 \
10 --cache-type-v q8_0 \
11 -kvu \
12 --cache-ram -1 \
13 --rope-scaling yarn \
14 --rope-scale 2.0 \
15 --yarn-orig-ctx 262144 \
16 --override-kv "qwen35.context_length=int:524288" \
17 --spec-type draft-mtp \
18 --spec-draft-n-max 3 \
19 --jinja \
20 --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true}' \
21 --host 0.0.0.0 --port 8080
Note: --spec-type draft-mtp requires llama.cpp b9375+. A
custom fork adds DFlash speculative decoding and Blackwell-tuned flash attention.
llama-cli
1llama-cli \
2 -m Qwen3.6-27B-AEON-UD-Q6_K.gguf \
3 -ngl 99 \
4 --flash-attn \
5 -c 524288 \
6 --rope-scaling yarn \
7 --rope-scale 2.0 \
8 --yarn-orig-ctx 262144 \
9 --jinja \
10 --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true}'
Chat Template Notes
enable_thinking activates reasoning mode (chain-of-thought in <think> blocks)preserve_thinking retains reasoning blocks in conversation history- No spaces after colons in the JSON — Qwen3.6's template parser is whitespace-sensitive
Architecture
Qwen3.6-27B is a hybrid SSM-attention model:
- 64 transformer layers + 1 MTP layer (blk.0-64)
- 48 SSM layers (GatedDeltaNet, no KV cache) + 16 full attention layers (every 4th layer)
- 27B parameters, 24 attention heads, 4 KV heads, head dim 256
- Vocab: 248,320 tokens, native context: 262,144 tokens
Quantization Pipeline
Built with super-quant:
- Convert HF to F16 GGUF (with MTP tensors) + mmproj GGUF (vision)
- Multi-domain calibration data from 13 HF datasets, special tokens stripped
- GPU-native importance matrix generation (PyTorch, 65k context) + weighted merge
- Per-tensor sensitivity analysis (KL divergence probing against F16 logits)
- Hybrid override generation — SSM at source precision, sensitivity-driven for the rest
- Quantize with per-tensor overrides + imatrix
- Benchmark: throughput + perplexity + KL divergence vs F16
Key Differences from Previous Release
| Aspect | Previous | Current |
|---|
| Calibration | Had <|endoftext|> token leak (2,628 occurrences) | Special tokens stripped automatically |
| Imatrix context | 32,768 | 65,536 |
| Imatrix generator | llama-imatrix (slow — PCIe D2H per tensor per chunk) | PyTorch GPU-native (zero D2H during generation) |
| Sensitivity analysis | llama.cpp subprocesses, ~600 GB disk I/O per run | PyTorch in-place weight perturbation, zero disk I/O |
| SSM alpha/beta | F32 (beyond source precision) | F16 (matches source BF16) |
| SSM out | Q8_0 (quantized) | F16 (source precision preserved) |
| Attention tensors | Mixed (f16/q8_0/q6_k) | All F16 (sensitivity-confirmed) |
| Norms/biases | Implicit (llama-quantize internal rules) | Explicit F16 overrides |
| Total overrides | 339 | 685 |
Links
- Base model: AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-BF16
- Quantization pipeline: super-quant
- llama.cpp fork: a4501150/llama.cpp (DFlash, MTP fixes, Blackwell FA4)
Credits
- Base model: AEON-7
- Architecture: Qwen Team
- Quantization: llama.cpp
- Sensitivity methodology inspired by APEX quant research
- Calibration datasets: HuggingFaceH4, teknium, NousResearch, nvidia, open-r1, Salesforce, glaiveai, froggeric
License: Apache-2.0
