Adaptive quantization family for Qwen3.5-9B hybrid Gated DeltaNet + Full Attention architecture. Each variant uses real imatrix data to selectively assign precision per-tensor.

Model: empero-ai/Qwable-9B-Claude-Fable-5 — Claude Fable 5 reasoning fine-tune of Qwen3.5-9B.

Note: The model file SHQ8-Q5_K_M.gguf is named for HF parser compatibility. This is NOT a pure Q5_K_M quantization — it's a hybrid using Q5_K_M base with Q8_0, Q5_K, Q4_K, and F16 selectively applied across different tensor types. Check the config for exact per-tensor types.

Note: SHQ8-v2-Q5_K_M.gguf is also a hybrid — Q5_K_M base with Q8_0 on critical attention, Q6_K on mid attention, IQ4_XS on low-importance tensors. See configs/SHQ8_v2.sh for the exact per-tensor map.

Speed: ~22 tokens/sec on GTX 1070 (8 GB VRAM) with flash-attn for SHQ-OptA, ~26 tokens/sec for SHQ8-v2 (fewer Q8_0 tensors → less data through Pascal's PCIe).

I've put a lot of work into hand-tuning these quants — let me know how they run on your hardware! Drop a comment or open a discussion with your setup and any feedback.

SHQ Family Tree

SHQ Series (Selective Hybrid Quantization)
│
├── SHQ6 ── Gemma 4 12B IT QAT
│   (QAT model, IQ4_XS all FFN + Q8_0 attention)
│   └── gemma4/README.md
│
├── SHQ7 ── Gemma 4 12B OBLITERATED
│   (non-QAT, gradient IQ4_NL + Q4_K + IQ4_XS + Q8_0)
│   └── gemma4/OBLITERATED_SHQ7_README.md
│
├── SHQ8 ── Qwen3.5-9B fine-tunes (non-QAT, Gated DeltaNet hybrid)
│   │
│   ├── Qwable-9B-Claude-Fable-5 (this)
│   │   ├── SHQ-OptA ★ WINNER
│   │   │   Q5_K_M base + Q8_0 on attn_gate/qkv/ssm_alpha/beta + imatrix
│   │   │   └── 6303 MiB, PPL 7.3747 — beats Q6_K in size + quality
│   │   ├── SHQ8-v2 — compact champion
│   │   │   Q5_K_M + selective Q8_0/Q6_K/IQ4_XS by imatrix
│   │   │   └── 5726 MiB, PPL 7.4559 — 18% smaller than OptA
│   │   ├── SHQ8-v3 — IQ4_NL experiment
│   │   │   Q5_K_M + all attn Q8_0 + IQ4_NL on imp ffn_down
│   │   │   └── 6041 MiB, PPL 7.4456 — 4% smaller than OptA
│   │   ├── SHQ8-v3a — all-attn Q8_0 test
│   │   │   Q5_K_M + all attn Q8_0 + ffn_down blk.20-30 Q8_0
│   │   │   └── 6301 MiB, PPL 7.4310 — same size, +0.056 vs OptA
│   │   └── Exp1 ── aborted
│   │       IQ4_XS all FFN (wrong for non-QAT)
│   │
│   └── Qwythos-9B-Claude-Mythos-5-1M
│       ├── SHQ8-OptA ★ quality champion
│       │   Same formula as OptA above
│       │   └── 6303 MiB, PPL 7.4831 — best PPL on Qwythos
│       ├── SHQ8-v2 — compact champion
│       │   Same tiered IMatrix approach as v2 above
│       │   └── 5726 MiB, PPL 7.6542 — 577 MiB less than OptA
│       └── MTP variants (built-in draft head for speculative decoding)
│           ├── M-SHQ8-MTP-OptA
│           │   OptA + blk.32 (MTP) at Q8_0
│           │   └── 6.40 GB, PPL 7.4831 — quality champion
│           └── M-SHQ8-MTP-v2
│               v2 + blk.32 (MTP) at Q8_0
│               └── 5.83 GB, PPL 7.6542 — compact champion

TL;DR

Quant	Size	PPL (ctx=1024)	vs Q6_K	vs OptA
i1-Q6_K (baseline)	7008 MiB	7.4394 ± 0.05	—	—
SHQ-OptA ★ quality	6303 MiB	7.3747 ± 0.05	best PPL	—
SHQ8-v2 ★ compact	5726 MiB	7.4559 ± 0.05	+0.017, -1282 MiB	+0.081, -577 MiB
SHQ8-v3	6041 MiB	7.4456 ± 0.05	+0.006, -967 MiB	+0.071, -262 MiB
SHQ8-v3a	6301 MiB	7.4310 ± 0.05	-0.008, -707 MiB	+0.056, -2 MiB

SHQ-OptA is the quality champion (best PPL). SHQ8-v2 is the compact champion (18% smaller than OptA at +0.081 PPL).

Key Result

Q5_K_M base + Q8_0 on attn_gate/qkv/ssm_alpha/ssm_beta + blk.31.ffn_down Q8_0 + imatrix per-row refinement.

PPL 7.3747 vs Q6_K 7.4394 (Δ = -0.065, within margin but directionally better)
Size 6303 MiB vs Q6_K 7008 MiB (10% smaller)
Imatrix auto-promotes high-importance tensors (ffn_down in deep layers → Q6_K)
Manual overrides take precedence: attn_gate/qkv → Q8_0 everywhere, blk.31.ffn_down → Q8_0

SHQ8-v2 — Real IMatrix Tiered Precision

Q5_K_M base + imatrix per-row + tiered precision by real importance data.

Real importance values from the actual Qwable-9B-Claude-Fable-5.imatrix.gguf (248 tensors, 319 chunks) revealed a 33,000× range — from blk.31 ffn_down at 323K to blk.0-5 ssm_out at 15. This enables safe aggressive quantization on low-importance tensors.

Tier	Types	Size Impact	Rationale
Q8_0	blk.0 attn (L0), blk.26-31 attn, blk.31 ffn_down, blk.31 attn_output, norms	+~155 MiB vs Q5_K	Highest importance layer edges
Q6_K	blk.1-25 attn_gate/qkv/ssm_alpha/beta	-~170 MiB vs Q8_0	Mid layers don't need Q8_0
IQ4_XS	ssm_out (all), attn_output (all), ffn_down (except blk.31)	-~350 MiB vs Q5_K	Low importance (43-21K mean)
F16	ssm_dt, ssm_a, all norms	Negligible	Tiny tensors
Q5_K_M	Everything else + imatrix per-row	Base	Proven design

Key insight — real imatrix ≠ README claims: The README's importance matrix table (1M+ for attention) was from a different file. The actual model's imatrix shows ~2× lower values and a different layer ordering (L0 dominates at 494K, not mid layers). This changes which tensors need high precision.

Size: 5,726 MiB (5.36 BPW) — 577 MiB (-9.2%) less than SHQ-OptA.

PPL: 7.4559 ± 0.05 — matches Q6_K (7.4394) within margin at 18% smaller size. Slightly higher than SHQ-OptA (7.3747) but saves 577 MiB (9.2%). A favorable size/quality trade-off.

Speed: ~26 tokens/sec on GTX 1070 — 18% faster than SHQ-OptA thanks to fewer Q8_0 tensors reducing VRAM bandwidth pressure.

Architecture

Same as Qwen3.5-9B — hybrid Gated DeltaNet + Gated Attention (3:1 pattern):

Property	Value
Layers	32 (24 DeltaNet + 8 Full Attention)
Hidden dim	4096
FFN intermediate	12288
Vocabulary	248,320
Attention-only	blk.3, 7, 11, 15, 19, 23, 27, 31
DeltaNet layers	all others

Full architecture details in SHQ-QWEN35-INSIGHTS_CODER.md.

How We Made It

Step 1: Q6_K baseline (PPL 7.4394)

Standard Q6_K with mradermacher's imatrix. Fits 8 GB VRAM at 7008 MiB.

Step 2: The IQ4_XS trap (aborted)

We tried IQ4_XS on all FFN tensors across all layers — blk.\d+.ffn_(gate|up|down)=IQ4_XS. This was a bad design for a non-QAT model: ffn_gate/up have 572K importance (far above the 150K IQ4_XS safety threshold). The run was aborted before completion.

Step 3: CODER insights rescue

Existing experiments on Qwopus3.5-9B-Coder-MTP (same architecture, different fine-tune) provided critical evidence (tested at ctx=2048):

Experiment	Result	Lesson
SHQv2 (Q5_K base + Q8_0 gates)	PPL 6.5456	Beats Q5_K_M (6.57)
SHQv3-C (IQ4_XS FFN + Q8_0 attn)	PPL 6.6534	+0.11 PPL vs SHQv2
SHQv3-D (Q5_K + IQ4_XS on low-imp)	Not tested	Predicted between SHQv2 and v3-C

150K importance threshold: IQ4_XS safe only on tensors with importance < 150K:

Tensor	Importance	IQ4_XS safe?
attn_q/k/v (Full Attn)	1,020,156	❌
attn_gate/qkv (DeltaNet)	879,647	❌
ssm_alpha/beta	879,647	❌
ffn_gate/up	572,159	❌
attn_output	127,631	✅
ffn_down	84,123	✅
ssm_out	43,393	✅

Full analysis in QUANT_HISTORY_CODER.md.

Step 4: SHQ-OptA — Winner

Chose the proven SHQv2 formula adapted for Qwable:

Base:        Q5_K_M (imatrix per-row refinement)
Attention:   Q8_0 — attn_gate, attn_qkv, ssm_alpha, ssm_beta (ALL layers)
Outlier:     Q8_0 — blk.31.ffn_down (most important tensor)
Output:      Q5_K
Token embd:  Q4_K
Norms:       F16, attn_q/k_norm Q8_0

Imatrix auto-promotes:

ffn_down in deep layers (18-30) → Q6_K (higher importance)
attn_v in some layers → Q6_K

Manual Q8_0 overrides are never downgraded.

Imatrix Notes

The importance matrix (Qwable-9B-Claude-Fable-5.imatrix.gguf) was computed by mradermacher from imatrix-training-full-3 — the same dataset used for fine-tuning the model itself. This gives unusually accurate importance signals.

Key findings from imatrix:

Inverted importance: Upper layers more important (blk.31 = 12× blk.1)
blk.31.ffn_down is the single most important tensor (3.97e+09)
Attention = 42.5% of total importance, FFN = 30.1%, SSM = 27.4%
Layer 1 is anomalously low (7.26e+08) — embedding boundary

Quantization Commands

SHQ-OptA

llama-quantize \
  --imatrix Qwable-9B-Claude-Fable-5.imatrix.gguf \
  --output-tensor-type Q5_K \
  --token-embedding-type Q4_K \
  --tensor-type "output_norm.*=Q8_0" \
  --tensor-type "blk\.\d+\.attn_gate=Q8_0" \
  --tensor-type "blk\.\d+\.attn_qkv=Q8_0" \
  --tensor-type "blk\.\d+\.ssm_alpha=Q8_0" \
  --tensor-type "blk\.\d+\.ssm_beta=Q8_0" \
  --tensor-type "blk\.31\.ffn_down=Q8_0" \
  --tensor-type ".*attn_q_norm.*=Q8_0" \
  --tensor-type ".*attn_k_norm.*=Q8_0" \
  --tensor-type ".*ssm_conv1d.*=Q8_0" \
  --tensor-type ".*ssm_dt.*=F16" \
  --tensor-type ".*ssm_a=F16" \
  --tensor-type ".*attn_norm.*=F16" \
  --tensor-type ".*post_attention_norm.*=F16" \
  --tensor-type ".*ssm_norm.*=F16" \
  Qwable-9B-Claude-Fable-5-bf16.gguf \
  Qwable-9B-Claude-Fable-5-SHQ-OptA.gguf \
  Q5_K_M

SHQ8-v2

Critical: --tensor-type rules are first-match-wins. Specific overrides must come before generic.

llama-quantize \
  --imatrix Qwable-9B-Claude-Fable-5.imatrix.gguf \
  --output-tensor-type Q5_K \
  --token-embedding-type Q4_K \
  --tensor-type "output_norm.*=Q8_0" \
  --tensor-type ".*attn_q_norm.*=Q8_0" \
  --tensor-type ".*attn_k_norm.*=Q8_0" \
  --tensor-type ".*ssm_conv1d.*=Q8_0" \
  --tensor-type "blk\.31\.ffn_down=Q8_0" \
  --tensor-type "blk\.31\.attn_output=Q8_0" \
  --tensor-type "blk\.0\.attn_gate=Q8_0" \
  --tensor-type "blk\.0\.attn_qkv=Q8_0" \
  --tensor-type "blk\.0\.ssm_alpha=Q8_0" \
  --tensor-type "blk\.0\.ssm_beta=Q8_0" \
  --tensor-type "blk\.(26|27|28|29|30|31)\.attn_gate=Q8_0" \
  --tensor-type "blk\.(26|27|28|29|30|31)\.attn_qkv=Q8_0" \
  --tensor-type "blk\.(26|27|28|29|30|31)\.ssm_alpha=Q8_0" \
  --tensor-type "blk\.(26|27|28|29|30|31)\.ssm_beta=Q8_0" \
  --tensor-type "blk\.(3|27|31)\.attn_q\.weight=Q8_0" \
  --tensor-type "blk\.(3|27|31)\.attn_k\.weight=Q8_0" \
  --tensor-type "blk\.(3|27|31)\.attn_v\.weight=Q8_0" \
  --tensor-type "blk\.([1-9]|1[0-9]|2[0-5])\.attn_gate=Q6_K" \
  --tensor-type "blk\.([1-9]|1[0-9]|2[0-5])\.attn_qkv=Q6_K" \
  --tensor-type "blk\.([1-9]|1[0-9]|2[0-5])\.ssm_alpha=Q6_K" \
  --tensor-type "blk\.([1-9]|1[0-9]|2[0-5])\.ssm_beta=Q6_K" \
  --tensor-type ".*attn_norm.*=F16" \
  --tensor-type ".*post_attention_norm.*=F16" \
  --tensor-type ".*ssm_norm.*=F16" \
  --tensor-type ".*ssm_dt.*=F16" \
  --tensor-type ".*ssm_a$=F16" \
  --tensor-type ".*ssm_out=IQ4_XS" \
  --tensor-type ".*attn_output=IQ4_XS" \
  --tensor-type ".*ffn_down=IQ4_XS" \
  Qwable-9B-Claude-Fable-5-bf16.gguf \
  SHQ8-v2-Q5_K_M.gguf \
  Q5_K_M

All Experiments

Experiment	Config	Size	PPL	vs Q6_K	Verdict
i1-Q6_K (baseline)	mradermacher imatrix	7008 MiB	7.4394 ± 0.05	—	Baseline
SHQ-OptA ★	Q5_K + Q8_0 gates + imatrix	6303 MiB	7.3747 ± 0.05	best PPL	Quality champion
SHQ8-v2 ★	Real imatrix tiered Q8_0/Q6_K/IQ4_XS	5726 MiB	7.4559 ± 0.05	-0.017, -1282 MiB	Compact champion
SHQ8-v3	All attn Q8_0 + IQ4_NL ffn_down	6041 MiB	7.4456 ± 0.05	+0.006, -967 MiB	Close to Q6_K
SHQ8-v3a	All attn Q8_0 + ffn_down blk.20-30 Q8_0	6301 MiB	7.4310 ± 0.05	-0.008, -707 MiB	Same size as OptA
SHQ-Exp1 (aborted)	IQ4_XS all FFN	—	—	—	Aborted (bad design)

Usage

llama.cpp

bash

llama-cli \
  -m SHQ8-Q5_K_M.gguf \
  -p "Write a Python function for merge sort" \
  -ngl 99 \
  --flash-attn on \
  -c 4096

For server mode (recommended for agent/coding use):

bash

llama-server \
  -m SHQ8-Q5_K_M.gguf \
  -c 65536 \
  -fit off \
  -ngl 99 \
  --jinja --chat-template-file /path/to/chat_template.jinja \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --port 8080 \
  --mmap \
  --temp 1 \
  --top-p 0.95 \
  --min-p 0 \
  --top-k 20 \
  --seed -1 \
  --parallel 1

Note: The chat template is from Unsloth (chat_template.jinja). For agent/coding tasks, temp 1 + top_k 20 + top_p 0.95 works well. Use -c 65536 for long context tasks.

VRAM usage: ~7399 MiB / 8192 MiB (90%) at -c 65536 with flash-attn + q8_0 KV cache. Room for more context thanks to the Gated DeltaNet architecture's efficient memory footprint.

From wepiqx: I would love to see this fine-tune released with MTP (Multi-Token Prediction) support — would make for an even better quantization. If Empero adds MTP, I'll update this quant with a specialized MTP draft model for faster inference.

Ollama

Create a Modelfile:

FROM ./SHQ8-Q5_K_M.gguf

TEMPLATE """{{ .System }}
{{ .Prompt }}"""

SYSTEM """You are a helpful coding assistant."""

PARAMETER num_ctx 8192
PARAMETER temperature 1
PARAMETER top_k 20
PARAMETER top_p 0.95

Build and run:

bash

ollama create qwable-SHQ8 -f Modelfile
ollama run qwable-SHQ8

Important — sampling (wepiqx): I strongly recommend not using repeat_penalty (keep at 1.0 / off). Testing shows it degrades output quality on this quant. Use temperature 1 (general) or temperature 0.6 (precise coding tasks). top_k 20 recommended. The model's native repetition handling is deliberately tuned and doesn't need penalizing.

LM Studio

Open LM Studio
Drag SHQ8-Q5_K_M.gguf into the app
Set GPU Offload to 99 layers
Enable flash-attention
Set context length to 8192+
Crucial — disable repeat_penalty (set to 1.0 or off). Use temperature 1 (general) or 0.6 (precise tasks), top_k 20
Start chatting

Coding Examples

These models generate full, working HTML/CSS/JS websites in a single pass. Both examples used the same prompt at temperature 1:

"I'm a dev, my audience is youth. I like a creative/tech style. Write the full website code. This HTML will be our foundation."

SHQ-OptA — SHQ8_temp-1.html

A complete, styled developer portfolio in 324 lines:

CSS variables, background grid effect
Typewriter animation (pure JS)
Card grid with hover glow
Terminal-style contact form
Responsive design (mobile tweaks)
JetBrains Mono font integration

SHQ8-v2 — SHQ8-v2_temp-1.html

Same prompt, 355 lines — cleaner code, no external dependencies:

Ambient floating orb background (pure CSS)
Typewriter effect + 3D tilt card (mousemove parallax)
Code window with syntax-highlighted preview
Custom cursor glow trail
Responsive design with mobile-safe tilt disable
Zero external dependencies — no Google Fonts, everything shipped inline

Lessons Learned

Imatrix from same dataset as training = gold. Importance signals are unusually clean.
IQ4_XS on FFN gate/up destroys non-QAT models. 150K importance threshold is real.
CODER experiments transferred 1:1. Same architecture, different fine-tune, same optimal strategy.
Q5_K_M + imatrix > Q6_K. The per-row refinement compensates for lower base bpw.
Manual --tensor-type overrides beat imatrix promotions. Q8_0 stays Q8_0.
Temperature 2 + SHQ-OptA gives incredible coding output. User reported impressive results.
Real imatrix data matters more than README claims. The actual importance matrix had 2× lower values and different layer ordering than documented, leading to a different optimal strategy.
ssm_out is quantization noise at 15-155 importance (10,000× below top tensors). Safe for IQ4_XS or even IQ2_XXS.
L0 and deep layers (26-31) need Q8_0; middle layers (1-25) suffice at Q6_K. Attention position matters more than attention type.

Future Work

Try IQ2_XXS on ssm_out (importance 15-155 — near zero)
Profile speed impact of IQ4_XS tensors on GTX 1070 (Pascal lacks IQ4 hardware)

Files

File	Size	Description
`Qwable-9B-Claude-Fable-5-bf16.gguf`	17 GB	BF16 source
`Qwable-9B-Claude-Fable-5.imatrix.gguf`	5 MB	Importance matrix (mradermacher)
`Qwable-9B-Claude-Fable-5-i1-Q6_K.gguf`	6.9 GB	Q6_K baseline
`SHQ8-Q5_K_M.gguf`	6.2 GB	Winner — recommended for inference
`SHQ8-v2-Q5_K_M.gguf`	5.7 GB	SHQ8-v2 — hybrid Q5_K_M base with tiered precision (compact champion)
`SHQ8_temp-1.html`	--	Coding example (OptA): full website at temp 1
`SHQ8-v2_temp-1.html`	--	Coding example (v2): same prompt, no external deps

wepiqx/Qwable-9B-Claude-Fable-5-SHQ8-GGUF

Qwable-9B-Claude-Fable-5 — SHQ Family (Selective Hybrid Quants)

SHQ Family Tree

TL;DR

Key Result

SHQ8-v2 — Real IMatrix Tiered Precision

Architecture

How We Made It

Step 1: Q6_K baseline (PPL 7.4394)

Step 2: The IQ4_XS trap (aborted)

Step 3: CODER insights rescue

Step 4: SHQ-OptA — Winner

Imatrix Notes

Quantization Commands

SHQ-OptA

SHQ8-v2

All Experiments

Usage

llama.cpp

Ollama

LM Studio

Coding Examples

SHQ-OptA — SHQ8_temp-1.html

SHQ8-v2 — SHQ8-v2_temp-1.html

Lessons Learned

Future Work

Files

References

wepiqx/
Qwable-9B-Claude-Fable-5-SHQ8-GGUF