AWQ 4-bit (W4A16) quantization of TeichAI/Qwen3.6-27B-Fable-5-Experimental — a Fable-5 reasoning fine-tune of Qwen3.6-27B (qwen3_5: a hybrid GatedDeltaNet (linear-attention) + full-attention model with a vision tower and an MTP head).

Variant: AWQ W4A16 — 4-bit symmetric integer weights, group size 128, with activation-aware scaling. Activations stay BF16. Disk size: ~25 GB (24.98 GiB; vs ~52 GB BF16, ~2.1×) Quantized by: sahilchachra Tooling: llm-compressor 0.12 (AWQModifier + QuantizationModifier) → compressed-tensors (pack-quantized)

Why this model and not Mia-AiLab/Qwable-3.6-27b: that repo ships only a Q4_K_M GGUF (no full-precision weights), so it cannot be honestly AWQ-quantized (AWQ needs BF16/FP16 source weights). This checkpoint quantizes the genuine Fable-5 BF16 fine-tune instead — the same Fable-5 reasoning lineage.

What is quantized

Only the language tower's standard linear layers are quantized (int4):

mlp.{gate,up,down}_proj on all 64 decoder layers
self_attn.{q,k,v,o}_proj on the 16 full-attention layers
= 256 quantized modules

Kept in BF16 (not quantized): the 48 GatedDeltaNet linear_attn (mamba) layers (no standard AWQ mapping, quantization-sensitive), the vision tower (model.visual.*), the MTP head (mtp.*), token embeddings, lm_head, and all norms. The headline name reflects the LM-tower quantization; on-disk size averages the int4 and BF16 parts.

Calibration (task-specific)

AWQ is calibration-based. This checkpoint was calibrated on 256 sequences of 2048 tokens drawn from armand0e/claude-fable-5-claude-code — the fine-tune's own Fable-5 reasoning training data (Claude Code agent traces). Transcripts are reconstructed turn-by-turn (user prompts + assistant reasoning/answers), rendered through the model's chat template, and windowed — so AWQ's per-channel scales reflect the model's real reasoning workload. Group size 128, AWQ scale search, duo_scaling.

Verification (smoke test)

Structure verified: 256 LM-tower modules carry weight_packed (int4) + weight_scale; the 48 GatedDeltaNet layers, vision tower, MTP head, embeddings and lm_head remain BF16. Functionally verified on an NVIDIA Thor by dequantizing the int4 weights to dense BF16 and generating: the model reasons correctly (e.g. the "17 sheep, all but 9 run away" trick → answers 9, and recognizes 17 is irrelevant) and produces coherent code reasoning, matching the BF16 source's behavior. This is a functional smoke test, not a quality benchmark.

Usage

Load with trust_remote_code and sdpa attention. The GatedDeltaNet kernels (causal-conv1d / flash-linear-attention) fall back to a pure-PyTorch path where unavailable (e.g. ARM/Jetson) — correct, just slower.

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "sahilchachra/Qwen3.6-27B-Fable-5-AWQ-W4A16"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, dtype=torch.bfloat16, device_map="cuda",
    attn_implementation="sdpa", trust_remote_code=True).eval()

msgs = [{"role": "user", "content": "Explain why the sky is blue, briefly."}]
enc = tok.apply_chat_template(msgs, add_generation_prompt=True,
                              return_tensors="pt", return_dict=True).to("cuda")
print(tok.decode(model.generate(**enc, max_new_tokens=256)[0]))

Loader note: compressed-tensors == 0.17.1 has a bug in its eager decompress_model path (it mis-reads group_size as 0 and raises strategy group requires group_size to be set to a positive value). This affects all pack-quantized checkpoints, not just this one. Use a compressed-tensors version where this is fixed, or serve with vLLM (its own loader is unaffected). The weights/scales on disk are standard and correct.

Notes

AWQ W4A16: LM-tower weights are 4-bit (symmetric, group 128, activation-aware); all other modules stay BF16.
Format: compressed-tensors pack-quantized. Custom code, processor/tokenizer and config carried over from the source.
Symmetric int4 chosen for broad loadability (transformers / vLLM / compressed-tensors).
Smoke-tested only; not a quality benchmark.

Original model

See TeichAI/Qwen3.6-27B-Fable-5-Experimental and base Qwen/Qwen3.6-27B for architecture, intended use, capabilities and license (Apache-2.0, inherited).