Views
No views yet
unsloth/Qwen3.8-27B-GGUF (BF16)
(original model: Qwen/Qwen3.8-27B, the hybrid
attention/SSM qwen35 architecture with an MTP nextn head).[!IMPORTANT] You need the ROCmFPX fork of llama.cpp (or a llama.cpp build with ROCmFPX support). These files use the experimentalq8_0_rocmfpx(type 103) and/orq4_0_rocmfp4_fast(type 101) weight formats, which stock llama.cpp releases do not understand — loading them elsewhere will fail with an unknown tensor type error.
| File | Recipe | Size | For |
|---|---|---|---|
Qwen3.8-27B-Q8_0_ROCMFPX.gguf | pure q8_0_rocmfpx, all weights | 26.9 GB (8.25 bpw) | large-VRAM systems (e.g. Strix Halo) |
Qwen3.8-27B-Q4FAST-Q8-sensitive.gguf | bulk q4_0_rocmfp4_fast + sensitive tensors at q8_0_rocmfpx | 16.4 GB (5.15 bpw) | ~24 GB VRAM laptops (leaves room for KV cache) |
UD-Q4_K_XL dynamic recipe (Q6_K tier =
most sensitive, Q5_K = next), rebalanced onto a strict two-type q4/q8 mix to hit ~16 GB:Tensors at q8_0_rocmfpx (165 tensors) | Why |
|---|---|
attn_q/k/v/output (17 full-attention layers) | attention projections (Unsloth Q5_K/Q6_K tier; attn_v is Q6_K there) |
attn_gate + ssm_out (48 linear-attn/SSM layers) | Q5_K tier in UD-Q4_K_XL |
output.weight head | Q6_K tier in UD-Q4_K_XL |
q4_0_rocmfp4_fast (340 tensors, incl. ffn_gate/up/down, attn_qkv,
token_embd — Unsloth keeps embeddings at Q4_K too), norms/biases stay f32 (360 tensors).
The MTP head (nextn.eh_proj) is auto-protected at q8_0 by the quantizer's draft-sensitive logic.1# pure Q8 (from the ROCmFPX fork; CPU-only build works fine for quantization)
2llama-quantize Qwen3.8-27B-BF16-00001-of-00002.gguf \
3 Qwen3.8-27B-Q8_0_ROCMFPX.gguf Q8_0_ROCMFPX
4
5# 16 GB hybrid
6llama-quantize \
7 --tensor-type "attn_q.weight=q8_0_rocmfpx" \
8 --tensor-type "attn_k.weight=q8_0_rocmfpx" \
9 --tensor-type "attn_v.weight=q8_0_rocmfpx" \
10 --tensor-type "attn_output.weight=q8_0_rocmfpx" \
11 --tensor-type "attn_gate.weight=q8_0_rocmfpx" \
12 --tensor-type "ssm_out.weight=q8_0_rocmfpx" \
13 --tensor-type "^output.weight=q8_0_rocmfpx" \
14 Qwen3.8-27B-BF16-00001-of-00002.gguf \
15 Qwen3.8-27B-Q4FAST-Q8-sensitive.gguf Q4_0_ROCMFP4_FASTmmproj-F16.gguf (mirrored from
unsloth/Qwen3.8-27B-GGUF, which also offers
mmproj-BF16.gguf).1# build ROCmFPX for your GPU (see the repo README; e.g. Strix Halo):
2env JOBS=16 scripts/build-strix-rocmfp4-mtp.sh
3
4./build-strix-rocmfp4/bin/llama-cli \
5 -m Qwen3.8-27B-Q8_0_ROCMFPX.gguf --mmproj mmproj-F16.gguf \
6 -dev Vulkan0 -ngl 999 -fa on --jinja[!NOTE] All Benchmarks were run on an AI max 395+ with 128GB RAM
| Model | Size | Depth | pp512 t/s | tg128 t/s |
|---|---|---|---|---|
| Q8_0_ROCMFPX | 26.25 GiB | 4096 | 304.65 ± 1.28 | 7.47 ± 0.03 |
| Q8_0_ROCMFPX | 26.25 GiB | 16384 | 247.26 ± 1.00 | 7.37 ± 0.10 |
| Q8_0_ROCMFPX | 26.25 GiB | 65536 | 143.50 ± 0.43 | 6.84 ± 0.00 |
| Q4_0_ROCMFP4_FAST | 16.38 GiB | 4096 | 329.88 ± 4.00 | 11.15 ± 0.13 |
| Q4_0_ROCMFP4_FAST | 16.38 GiB | 16384 | 262.43 ± 1.17 | 10.95 ± 0.02 |
| Q4_0_ROCMFP4_FAST | 16.38 GiB | 65536 | 147.77 ± 0.59 | 9.49 ± 0.02 |
unsloth/Qwen3.8-27B-GGUF (BF16 shards);
sensitivity tiers referenced from their UD-Q4_K_XLQwen/Qwen3.8-27B