Quantizations of trohrbaugh/Qwen3.8-27B-heretic-ara with MTP (Multi-Token Prediction) tensors preserved (866 tensors) and vision projector (mmproj) for multimodal image understanding.

Includes ROCmFPX quants for the ROCmFPX fork of llama.cpp — ~42 t/s average decode on AMD Strix Halo (50–75% faster than stock Vulkan).

Files

File	Size	BPW	Format	Description
`Qwen3.8-27B-heretic-ara-ROCmFP4-FAST.gguf`	14 GB	4.26	Q4_0_ROCMFP4_FAST	Fastest — ROCmFPX Vulkan, ~42 t/s decode with MTP
`Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf`	16 GB	4.83	Q4_K_M	Stock llama.cpp, ~27–28 t/s decode with MTP
`Qwen3.8-27B-heretic-ara-Q6_K-MTP.gguf`	21 GB	6.56	Q6_K	Higher quality, stock llama.cpp
`Qwen3.8-27B-heretic-ara-ROCmFP6.gguf`	21 GB	6.50	Q6_0_ROCMFPX	Higher quality, ROCmFPX Vulkan
`mmproj-Qwen3.8-27B-heretic-ara-BF16.gguf`	931 MB	—	BF16	Vision encoder/projector (shared by all quants)

Which quant should I use?

ROCmFP4_FAST (14 GB): Fastest option on RDNA 3.5 (Strix Halo, gfx1151). Requires ROCmFPX fork with -dev Vulkan0. ~42 t/s average decode. Best choice if you have Strix Halo and want maximum speed.
Q4_K_M (16 GB): Best for stock llama.cpp on any hardware. ~27–28 t/s on Strix Halo. Full vision + MTP support. Best choice for general use.
Q6_K (21 GB): Higher quality for creative writing and nuanced reasoning. ~30% slower on unified memory due to larger file size. Choose if quality matters more than speed.
ROCmFP6 (21 GB): Higher quality ROCmFPX variant. ~24 t/s on Strix Halo. Use if you want both ROCmFPX speed and higher quality, and have memory headroom.

Source model

trohrbaugh/Qwen3.8-27B-heretic-ara — Arbitrary-Rank Ablation (ARA) uncensoring of Qwen3.8-27B using heretic v1.2.0+custom. Parameters: start_layer 26, end_layer 56, preserve_good_behavior_weight 0.9432, steer_bad_behavior_weight 0.0009. KL divergence 0.0535, 0/100 refusals.

How to use

Stock llama.cpp — Text-only with MTP (fastest stock decode)

bash

llama-server \
  -m Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf \
  -ngl 99 -ctk f16 -ctv f16 \
  -c 32768 \
  --spec-type draft-mtp \
  --spec-draft-n-max 3

Swap in Q6_K-MTP.gguf for higher quality. --spec-draft-n-max 3 is optimal on AMD Strix Halo (Vulkan). Try 4 on other hardware.

Stock llama.cpp — Vision + MTP (multimodal)

bash

llama-server \
  -m Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf \
  --mmproj mmproj-Qwen3.8-27B-heretic-ara-BF16.gguf \
  -ngl 99 -c 8192 -fa on \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --jinja

Then send images via the OpenAI-compatible API:

bash

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
  "messages": [{
    "role": "user",
    "content": [
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},
      {"type": "text", "text": "Describe this image."}
    ]
  }],
  "max_tokens": 300
}'

Note: mmproj has zero impact on text-only request performance — the vision encoder only activates for image-bearing requests. Safe to load mmproj permanently.

ROCmFPX — Maximum speed (text-only)

Requires ROCmFPX fork built for your GPU.

bash

llama-server \
  -m Qwen3.8-27B-heretic-ara-ROCmFP4-FAST.gguf \
  -ngl 99 -dev Vulkan0 \
  -c 32768 \
  -ctk q8_0 -ctv turbo4 \
  -fa on \
  --spec-type draft-mtp \
  --spec-draft-n-max 6

Critical flags:

-dev Vulkan0 — forces RADV cooperative matrix path. Without it, ROCm backend runs at ~18 t/s instead of ~42.
-ctk q8_0 -ctv turbo4 — TurboQuant KV cache. Speed-neutral, dramatically reduces KV memory (enables 262K context in 20 GB vs 61 GB with f16 KV).
--spec-draft-n-max 6 — optimal for ROCmFPX (vs 3 for stock Vulkan).

ROCmFPX — Vision

bash

llama-server \
  -m Qwen3.8-27B-heretic-ara-ROCmFP4-FAST.gguf \
  --mmproj mmproj-Qwen3.8-27B-heretic-ara-BF16.gguf \
  -ngl 99 -dev Vulkan0 \
  -c 8192 -np 1 \
  -ctk q8_0 -ctv turbo4 \
  -fa on \
  --jinja

Important: MTP speculative decoding must be disabled for vision requests on ROCmFPX (b213) — the non-consecutive token positions from image embeddings crash MTP boundary tracking. Omit --spec-type draft-mtp when using --mmproj. Vision decode runs at ~13.6 t/s without MTP. Text-only requests will not benefit from MTP in this configuration; use a separate text-only server with MTP for maximum text speed.

Without MTP (slower but works everywhere)

bash

llama-server \
  -m Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf \
  -ngl 99 -c 32768

Benchmarks (AMD Ryzen AI Max+ 395, 122 GB unified, gfx1151)

ROCmFPX (Vulkan0, MTP n-max 6, TurboQuant KV)

Task	ROCmFP4_FAST Decode	ROCmFP4_FAST Prefill	ROCmFP6 Decode	ROCmFP6 Prefill
Code gen #1	34.5 t/s	42.0	25.7 t/s	29.5
Code gen #2	41.6 t/s	61.5	29.3 t/s	29.7
Code gen #3	49.5 t/s	61.5	—	—
Math	37.5 t/s	97.4	22.9 t/s	50.4
JSON extraction	44.7 t/s	137.1	—	—
Technical	43.8 t/s	59.6	18.1 t/s	21.4
Average	~42 t/s		~24 t/s*

*ROCmFP6 benchmarked under swap pressure (21 GB model at 8K ctx). Would likely hit ~30+ t/s without swap contention.

Stock Vulkan (MTP n-max 3)

Quant	Prefill (t/s)	Decode (t/s)	BPW
Q4_K_M-MTP	168–172	27–28	4.83
Q6_K-MTP	~65	~17–23	6.56
Q4_K_M-MTP, no MTP	200	12.2	4.83

Vision (with mmproj, Q4_K_M, stock Vulkan)

Config	Prompt tokens	Prefill (t/s)	Decode (t/s)	MTP acceptance
Vision + MTP n-max 3 (simple image)	79	23.7	13.5	62.6%
Vision + MTP n-max 3 (complex image)	830	93.4	12.7	54.6%
Vision, no MTP	79	44.9	12.1	—

Vision decode is ~13 t/s (vs 28 text-only) due to vision encoder overhead. Text-only requests are not affected by having mmproj loaded.

ROCmFPX vision (without MTP): ~13.6 t/s decode, ~85.9 t/s prefill. MTP must be disabled for vision on ROCmFPX b213.

Stock Vulkan benchmarked on llama.cpp build 0ef6e55ed (b10254). ROCmFPX benchmarked on b213/b2f5829.

Speed comparison summary

Config	Avg Decode (MTP)	Avg Prefill	File Size	Memory fit (122 GB)
ROCmFPX FP4_FAST	~42 t/s	~77 t/s	14 GB	✅ Comfortable
Stock Q4_K_M	~27–28 t/s	~170 t/s	16 GB	✅ Comfortable
ROCmFPX FP6	~24 t/s*	~33 t/s*	21 GB	⚠️ Swap at 32K
Stock Q6_K	~17–23 t/s	~65 t/s	21 GB	⚠️ Swap at 131K

Code quality — EvalPlus HumanEval+ (164 problems, pass@1, temp=0)

Quant	HumanEval	HumanEval+	BPW
Q6_K-MTP	88.4%	81.7%	6.56
Q4_K_M-MTP	82.3%	75.6%	4.83
ROCmFP4_FAST	75.6%	71.3%	4.26
ROCmFP6	73.8%	67.7%	6.50

Stock llama.cpp quants (Q6_K, Q4_K_M) outperform ROCmFPX quants at similar BPW — Q6_K-MTP (6.56 BPW) scores 14% higher on HumanEval+ than ROCmFP6 (6.50 BPW). The ROCmFPX quantization format trades code quality for decode speed. All quants handle math/reasoning, creative writing, and vision correctly on qualitative tests.

Qualitative comparison

All quants tested on identical prompts at temperature 0.0:

Math/reasoning: All produce correct step-by-step solutions
Creative writing: All produce strong prose. Q6 variants show marginally richer vocabulary.
Vision: All correctly identify shapes, colors, and text in test images.

Architecture

Qwen3.8-27B uses the qwen3.5 architecture — a hybrid DeltaNet + attention model (48 linear + 16 full-attention layers) with an integrated vision encoder. Key notes:

Natively multimodal — 333 vision tensors for image understanding (extracted as mmproj)
Prompt caching is broken in llama.cpp for hybrid-memory models (every turn reprocesses full context)
Only 16 layers grow KV cache → cheap context scaling
MTP provides ~2.3× decode speedup via speculative decoding (text-only, stock Vulkan)
MTP on ROCmFPX provides ~3.2× decode speedup (13 → 42 t/s)

Quantization details

ROCmFP4_FAST: Q4_0_ROCMFP4_FAST, 4.26 BPW, ~14 GB, 866 tensors (MTP nextn.* → q8_0 auto)
Q4_K_M: 4.83 BPW, ~16 GB, 866 tensors (851 model + 15 MTP)
Q6_K: 6.56 BPW, ~21 GB, 866 tensors (851 model + 15 MTP)
ROCmFP6: Q6_0_ROCMFPX, 6.50 BPW, ~21 GB, 866 tensors (MTP nextn.* → q8_0 auto)
mmproj: BF16, 931 MB, 334 tensors (vision encoder — CLIP arch, cannot be quantized)
Source: BF16 safetensors → BF16 GGUF (51 GB) → all quant variants
Stock quants: convert_hf_to_gguf.py → llama-quantize
ROCmFPX quants: convert_hf_to_gguf.py → ROCmFPX llama-quantize (b213/b2f5829)
mmproj: convert_hf_to_gguf.py --mmproj --outtype bf16

MTP tensors included (all quants)

blk.64.nextn.eh_proj.weight
blk.64.nextn.enorm.weight
blk.64.nextn.hnorm.weight
blk.64.nextn.shared_head_norm.weight
blk.64.nextn.attn_k.weight
blk.64.nextn.attn_k_norm.weight
blk.64.nextn.attn_norm.weight
blk.64.nextn.attn_output.weight
blk.64.nextn.attn_q.weight
blk.64.nextn.attn_q_norm.weight
blk.64.nextn.attn_v.weight
blk.64.nextn.ffn_down.weight
blk.64.nextn.ffn_gate.weight
blk.64.nextn.ffn_up.weight
blk.64.nextn.ssm_out.weight

cygnal/Qwen3.8-27B-heretic-ara-Q4_K_M-MTP-GGUF

Qwen3.8-27B-heretic-ara — GGUF with MTP + Vision

Files

Which quant should I use?

Source model

How to use

Stock llama.cpp — Text-only with MTP (fastest stock decode)

Stock llama.cpp — Vision + MTP (multimodal)

ROCmFPX — Maximum speed (text-only)

ROCmFPX — Vision

Without MTP (slower but works everywhere)

Benchmarks (AMD Ryzen AI Max+ 395, 122 GB unified, gfx1151)

ROCmFPX (Vulkan0, MTP n-max 6, TurboQuant KV)

Stock Vulkan (MTP n-max 3)

Vision (with mmproj, Q4_K_M, stock Vulkan)

Speed comparison summary

Code quality — EvalPlus HumanEval+ (164 problems, pass@1, temp=0)

Qualitative comparison

Architecture

Quantization details

MTP tensors included (all quants)

cygnal/
Qwen3.8-27B-heretic-ara-Q4_K_M-MTP-GGUF