Calibrated FP8 quantization of DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP that preserves vision-language capabilities, unlike vLLM's dynamic FP8 which destroys them on this architecture family.

TL;DR

~30 GB FP8 (down from ~52 GB BF16), serves on a single 48 GB+ GPU with 16k context.
Vision verified intact after quantization (correctly reads fine image details — eye color, clothing trim, props — on real multimodal workloads).
Do NOT pass --quantization fp8 to vLLM — quantization config is baked in via compressed-tensors.
For structured/non-reasoning tasks, pass chat_template_kwargs: {"enable_thinking": false} — see Usage. Without it the model emits long tag-less "Thinking Process:" reasoning before answering.

About the source model

The source is DavidAU's multi-stage merge on the Qwen3.6-27B base (Qwen qwen3_5 architecture class: hybrid attention with Gated DeltaNet linear-attention layers + full attention every 4th layer, vision encoder included, 256k native context). Decensoring is via Heretic v1.2 with Arbitrary-Rank Ablation (ARA) — the source card measures 4/100 refusals vs 99/100 for the base model.

Note: the source repo ships a sidecar MTP (multi-token prediction) tensor file that is not part of the model index. It is not included in this quant; vLLM does not use it for standard serving.

Why dynamic FP8 destroys vision on this architecture

vLLM's runtime --quantization fp8 uses a single tensor-wide scale per Linear layer. The vision merger — the only bridge between the 1152-dim visual tower and the 5120-dim LM embedding space — has a much wider weight distribution than LM layers, so single-scale FP8 rounds its small-magnitude weights to zero. The LM then receives noise at image-token positions and silently hallucinates image descriptions from text alone. See the detailed write-up in the sibling repo tacodevs/Huihui-Qwen3.5-27B-Claude-4.6-Opus-abliterated-FP8.

What this checkpoint does differently

Per-channel weight scales (one scale per output channel, computed from actual weight distributions) instead of one global scale per layer.
Entire visual tower and merger kept in BF16 via the ignore list.
All Gated DeltaNet linear_attn modules kept in BF16 (hybrid-attention internals are not plain GEMMs and are excluded).
lm_head kept in BF16.
Dynamic per-token activation quantization at inference time.

Only the language model body (full-attention projections and MLP linears) is FP8.

Quantization details

Tool: llmcompressor (see recipe.yaml in this repo)
Scheme: FP8_DYNAMIC (per-channel weight scales, dynamic per-token activation scales)
Targets: all Linear layers
Excluded modules (207 total): re:.*visual.* (visual tower + merger), all linear_attn modules and norms, lm_head
Original size: ~52 GB BF16 → FP8 size: ~30 GB

Measured performance (RTX PRO 6000 Blackwell 96 GB, vLLM 0.26)

Decode: ~49 tok/s single-stream warm (identical to the BF16-recipe sibling Qwen3.5-27B FP8 on the same GPU).
TTFT ~0.8–1.0 s on multimodal requests (one 900px image + ~1k text tokens).
Vision: correct fine-grained image reading across all test runs (hair/eye color, clothing details, held objects, background).

Usage with vLLM

bash

python -m vllm.entrypoints.openai.api_server \
    --model tacodevs/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-FP8 \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.90 \
    --trust-remote-code

IMPORTANT: Do NOT pass --quantization fp8. The model already has its quantization config baked in via compressed-tensors; vLLM detects and uses the proper FP8 path automatically. Passing --quantization fp8 would re-quantize the already-FP8 weights and break everything.

Controlling thinking mode

The chat template supports Qwen's enable_thinking switch. By default the model produces extended reasoning without <think> tags (plain "Thinking Process:" markdown), which is easy to overrun token budgets with and hard to strip in streaming pipelines. For structured-output or latency-sensitive tasks, disable it per request:

json

{
  "model": "tacodevs/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-FP8",
  "chat_template_kwargs": {"enable_thinking": false},
  "messages": [...]
}

In our structured image-prompt workload this cut output from a truncated 500+ tokens to a complete 130–210 tokens and total latency from ~11 s to ~4.7 s, with format-perfect results.

Credits

Source merge: DavidAU
Decensoring: Heretic v1.2 (Arbitrary-Rank Ablation)
Base model: Qwen/Qwen3.6-27B
Quantization: tacodevs