component	precision
`mlp.{gate,up,down}_proj`, layers 0–55	INT4, group-32, asymmetric (~4.6 effective bits)
`mlp.{gate,up,down}_proj`, layers 56–63	INT8 W8A16
`self_attn.{q,k,v,o}_proj`	INT8 W8A16
`linear_attn.{in_proj_qkv,in_proj_z,out_proj}` (GDN)	INT8 W8A16
`lm_head`, `embed_tokens`, norms, GDN state params, vision tower	BF16

AWQ per-input-channel scaling (folds into the norm weights — zero size cost), then GPTQ Hessian error compensation. Runs on anything Turing or newer; no FP8/FP4 hardware needed.

Calibration: self-distilled from the abliterated model on a balanced Nemotron-v2 prompt blend (25% code, 25% math, 20% STEM, 20% chat, 10% multilingual).

Benchmarks

Measured against the abliterated BF16 model as its own reference — not stock Qwen — so the numbers reflect quantization damage only, not the effect of abliteration. 142,727 tokens plus 200 free greedy generations. vLLM 0.27.1, TP=2, 2×B300.

build	size ↓	top-1 ↑	near-tie ↓	moderate ↓	confident ↓	certain ↓	divmed ↑	tok/s ↑
this model (INT4 AWQ+GPTQ)	25.1 GB	96.45%	21.76%	2.92%	0.88%	0.12%	41	4551
NVFP4 sibling (AWQ+AutoRound)	24.7 GB	92.98%	34.02%	9.83%	1.80%	0.20%	27	10702
earlier build (base-model calibration)	24.7 GB	91.79%	37.58%	11.21%	3.54%	0.21%	20	10685

Sizes are on-disk tensor bytes and include the ~0.85 GB BF16 MTP head.

Columns. top-1 is raw argmax agreement with the BF16 abliterated model. The bucket columns are disagreement rates split by how confident the reference was at that position (top1−top2 logprob margin): near-tie <0.5, moderate 0.5–2, confident 2–5, certain >5. Only confident and certain are real damage. divmed is the median token index at which free greedy generation first diverges.

Perplexity is excluded — on this model family it is anti-correlated with quality.

Calibration matters more than abliteration

An earlier build of this model used the stock-Qwen calibration set and a weaker recipe, and landed at 3.54% confident damage. Regenerating the calibration from the abliterated model itself brings that to 0.88%.

That also answers a question worth stating plainly: abliterated weights are not intrinsically harder to quantize. With matched recipe and self-distilled calibration this model reaches 0.88% confident damage, against 0.93% for the same recipe on stock Qwen3.8-27B. The earlier gap was the calibration and recipe, not the abliteration.

INT4 vs NVFP4 on this model

The INT4 build is substantially more faithful than the NVFP4 one (confident 0.88% vs 1.80%, paired McNemar z = 13.2; certain 0.12% vs 0.20%, z = 4.3), reproducing the same gap measured on stock Qwen3.8-27B (0.93% vs 1.85%). At ~4.6 effective bits INT4 group-32 asymmetric gives 16 uniform levels plus a per-group zero point, against FP4's 8 non-uniform levels at 4.5 bits.

NVFP4's advantage is hardware: it decodes at 10702 tok/s here versus 4551. Pick the NVFP4 sibling if you are throughput-bound on Blackwell; pick this for fidelity, or on Ampere/Ada.

Usage

python

from vllm import LLM
llm = LLM("TelperionAI/Huihui-Qwen3.8-27B-abliterated-INT4-AWQ-GPTQ", tensor_parallel_size=2)

Speculative decoding (MTP)

The MTP head is included, in BF16, grafted from the abliterated base (not stock Qwen):

python

llm = LLM("TelperionAI/Huihui-Qwen3.8-27B-abliterated-INT4-AWQ-GPTQ", tensor_parallel_size=2,
          speculative_config={"method": "mtp", "num_speculative_tokens": 2})

Qwen3_5ForConditionalGeneration does not carry mtp.* in its state dict, so llm-compressor silently drops it even though config.json declares mtp_num_hidden_layers: 1. It is excluded from quantization via re:.*mtp.*. Acceptance rate has not been measured; the head is verified to load and generate.

Limitations

Single evaluation corpus, and no downstream task benchmarks.
Abliterated base. This model has had its refusal directions removed upstream; that behaviour is inherited here and is not something quantization changes.
The abliterated calibration set is ~18% smaller than the stock one (the same length filter kept fewer generations), so it is not perfectly matched to the stock-model builds.
Vision tower untouched (BF16); evaluated as a text model.
MTP acceptance rate unmeasured.