On this hardware int8 is SLOWER than BF16 — roughly half the decode rate — and MTP speculative decoding does not work. Both are measured below. Use these only if you need the smaller memory footprint.

Scheme

Weights 8-bit, per-channel, symmetric. Activations 8-bit, per-token dynamic (int-quantized). Per-token dynamic scaling is computed at runtime, which avoids the activation-outlier failure mode of per-tensor static scaling.

Quantized with llm-compressor 0.12.1a20260701, QuantizationModifier (round-to-nearest). No calibration data was used and none is needed: weights are round-to-nearest and activation scales are computed dynamically at inference time.

Measured performance — read this before choosing int8

Benchmarked on 2x AMD MI210 (gfx90a), vLLM 0.27.2rc0+mi210.1, TP=2, prefix caching disabled (leaving it on inflates decode, because the harness derives decode time by subtracting a prefill that the second, cached request never performs).

Decode, tokens/s, at three context depths:

config	@0	@8k	@32k	weights
BF16 + MTP	54.1	33.3	16.0	52 GB
BF16	31.3	17.5	7.7	52 GB
W8A8 (this family)	17.3	12.0	6.4	34 GB
W8A16 (this family)	16.1	11.3	6.2	34 GB

INT8 is roughly half the decode rate of BF16 on this hardware, not faster.

gfx90a has no fused dequantization path and its INT8 peak equals its BF16 peak (181 TOPS vs 181 TFLOP/s), so every quantized weight is converted to bf16 before the MFMA units can use it, and there is no compute headroom to recover that cost from. The same pattern held for int8 activations and for an int8 KV cache, which lost 38-48% of throughput at depth.

What int8 does buy is memory: 34 GB against 52 GB, and roughly 830k KV tokens against 578k at 64K context (12.7x vs 8.8x concurrency). If you are capacity-bound rather than latency-bound, that is the trade on offer.

Of the two, W8A8 is the faster — about 6% on decode and 36% on prefill — despite quantizing more. Prefer it over W8A16 unless you specifically need weight-only.

On a CUDA GPU with real INT8 tensor-core paths the picture is likely different; none of the above should be read as a claim about other hardware.

MTP does not work on these quants

The base model ships an MTP draft head and mtp_num_hidden_layers: 1, and vLLM registers qwen3_5_mtp as a speculative method. On BF16 it is a large win — 86-100% draft acceptance, ~2x decode.

On any compressed-tensors quantization of this model it produces 0.0% acceptance: the draft runs every step and every token is rejected, so it is pure overhead and leaves you slower than not using it.

Isolated by elimination, all on the same hardware and vLLM build:

target	acceptance
official BF16	86-100%
abliterated BF16	83.8%
W8A8, `mtp.*` kept BF16	0.0%
W8A8, `mtp.*` also quantized	0.0%
W8A16 (weight-only)	0.0%
W8A8 target, draft pointed at a BF16 checkpoint	0.0%

So it is not abliteration, not the quantization scheme, not how the MTP head itself is treated, and not the draft checkpoint path — vLLM reads mtp.* from the target regardless. Exactly 0.0% rather than a degraded rate points to something structurally broken rather than quality loss.

Possibly relevant: vLLM copies the target's quantization into the draft config for the dspark method and has no equivalent for MTP.

Do not pass --speculative-config with these checkpoints.

What is quantized — and what deliberately is not

Qwen3.8-27B is a hybrid: layer_types is 48 linear_attention (GDN) layers and 16 full_attention, full_attention_interval: 4. Only 16 of 64 layers hold a KV cache, which is why its context is cheaper than the layer count suggests.

Of 985 modules, 256 are quantized and 729 are left at BF16:

quantized	count
`model.language_model.layers.N.mlp.{gate,up,down}_proj`	192 (all 64 layers)
`model.language_model.layers.N.self_attn.{q,k,v,o}_proj`	64 (the 16 full-attention layers)

protected	count	why
`re:.linear_attn.`	384	all 48 GDN layers, incl. `conv1d` / `in_proj_*` / `out_proj`
`re:.visual.`	167	vision tower
`re:.norm.`	271	norms
`re:^mtp\..*`	15	MTP draft head
`lm_head`, `embed_tokens`	2

MLPs dominate parameter count, so quantizing them captures most of the footprint reduction while the sensitive linear-attention path stays untouched.

Loading gotcha

On Qwen3_5Config, AutoModelForCausalLM maps to Qwen3_5ForCausalLM — the text-only class. It loads without error and silently drops all 54 vision blocks. Use AutoModelForImageTextToText, which maps to Qwen3_5ForConditionalGeneration.

Serving (vLLM)

bash

vllm serve davetha/Qwen3.8-27B-abliterated-W8A8 \
  --tensor-parallel-size 2 --max-model-len 131072 \
  --gpu-memory-utilization 0.85 --trust-remote-code \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3

qwen3_xml, not hermes: the chat template emits <tool_call><function=name><parameter=x> XML, which the hermes parser does not match.

On gfx90a note that head_dim = 256, which upstream vLLM's ROCm paged-attention gate rejects (it accepts only 64 or 128).

Verified

Loads under vLLM 0.27.2rc0+mi210.1, TP=2 on 2x MI210
Correct at temperature 0: 17*23 -> 391; 2:15pm-6:40pm -> 265 minutes; string-reverse one-liner -> s = s[::-1]

Not measured: perplexity, and no benchmark suite has been run. Quality claims beyond the spot checks above are unverified.

Note

The base model is abliterated, so the usual refusal behaviour is substantially removed. You are responsible for how you use it.