Two custom mixed-tensor quantizations of Qwen/Qwen3.8-27B-FP8, the dense 27B multimodal model with a hybrid GatedDeltaNet + Gated Attention architecture and Multi-Token Prediction (MTP) head.

Built with llama.cpp at commit 1692f9e, using an importance matrix (imatrix) and per-tensor quantization overrides.

Both quantizations are designed around the 16 GB VRAM budget of consumer GPUs such as the AMD Radeon RX 9070 XT.

On 24 GB GPUs(3090, 4090, 7900 XTX), both quants have substantial headroom — full-context runs with Q8_0 KV cache throughout should fit somewhat comfortably.

PPL comparison

The following comparison was measured against other Q4 quantizations:

Rank	Model	Final PPL↓	Reported uncertainty	Difference
1	Unsloth Qwen3.8-27B IQ4_XS	7.9642	± 0.03107	—
2	vmarcelo Qwen3.8-27B IQ4-MIX	8.0315	± 0.03136	+0.0673 PPL
3	Bucoid Qwen3.8-27B-IQ4_XS_4BPW.gguf	8.0609	± 0.03151	+0.0967 PPL
4	cHunter789 IQ4_KT	8.2890	± 0.03224	+0.3248 PPL
5	mradermacher IQ4_XS	8.2955	± 0.03221	+0.3313 PPL

NOTE: cHunter789 IQ4_KT and mradermacher IQ4_XS PPL test I didn't run. I used the numbers that cHunter789 provided and they might not match real result since I used a different version of llama-perplexity.

Files

File	Size	BPW	Category	Best for
`Qwen3.8-27B-IQ4-MIX.gguf`	14.10 GB	4.13	Q4	Best quality while fitting within 16 GB VRAM
`Qwen3.8-27B-IQ3-MIX.gguf`	12.58 GB	3.68	Q3	Larger context headroom (64K–128K)
`mmproj-F16.gguf`	885 MB	—	—	Vision encoder; load with `--mmproj`
`qwen38_iq4_types.txt`	269 B	—	—	IQ4-MIX recipe for reproducibility
`qwen38_iq3_types.txt`	319 B	—	—	IQ3-MIX recipe for reproducibility
`config.json`	3.7 KB	—	—	Architecture metadata for LM Studio / HF Hub

Which one should I pick?

	IQ4-MIX (Q4)	IQ3-MIX (Q3)
Size	14.10 GB	12.58 GB
BPW	4.13	3.68
VRAM (model + 16K context)	~15.7 GB	~14.6 GB
VRAM headroom	~0.6 GB	~1.4 GB
Max context on 16 GB VRAM	16K–64K	64K–128K
Generation speed	33 tok/s	37 tok/s
Quality	Higher	Slightly lower, mainly due to linear attention at Q3
Best for	Code, reasoning, vision	Long context, agentic workflows

IQ4-MIX

Choose IQ4-MIX if you want the best quality and don't need context beyond ~32K.

IQ3-MIX

Choose IQ3-MIX if you need larger context windows (64K–128K), want more VRAM headroom, or need to run other GPU workloads alongside the model.

Performance

Tested on an AMD Radeon RX 9070 XT (gfx1201, 16 GB VRAM) using LM Studio's Vulkan build:

llama.cpp-linux-x86_64-vulkan-avx2-2.29.0

Generation speed (TG)

Test	IQ4-MIX	IQ3-MIX
Text PT (fotossíntese)	32.0 tok/s	36.7 tok/s
Text EN (photosynthesis)	32.2 tok/s	37.5 tok/s
Reasoning (count "r" in strawberry)	32.8 tok/s	37.9 tok/s
Code explain (lambda)	33.4 tok/s	37.4 tok/s
Code write (`is_palindrome`)	33.3 tok/s	37.4 tok/s
Multilingual (FR/DE/JA)	32.8 tok/s	37.5 tok/s
Longer reasoning (5 LLM uses)	33.2 tok/s	37.9 tok/s
Vision (image description)	33.4 tok/s	37.9 tok/s
Average	33.0 tok/s	37.5 tok/s

IQ3-MIX is approximately 14% faster than IQ4-MIX due to its lower average bits-per-weight and reduced memory bandwidth requirements.

Prompt processing (PP)

Test	IQ4-MIX	IQ3-MIX
Short prompts (20–30 tokens)	75–108 tok/s	65–131 tok/s
Cached (2000+ tokens)	638 tok/s	635 tok/s
Vision (image + text)	655 tok/s	548 tok/s

Context vs. VRAM

Measured with the indicated KV-cache quantization:

Context	KV cache	IQ4-MIX VRAM	IQ3-MIX VRAM
4K	Q8_0	14.6 GB ✅	13.5 GB ✅
16K	Q8_0	15.7 GB ⚠️	14.6 GB ✅
64K	Q8_0	15.4 GB ⚠️	14.3 GB ✅
128K	Q8_0 + Q4_0	16.0 GB ❌	15.0 GB ⚠️
256K	Q4_0	15.7 GB ⚠️	14.6 GB ✅

Note: The VRAM figures above depend on the backend, runtime configuration, batch size, and other loaded GPU resources. They should be treated as practical measurements rather than hard minimums.

With Q8_0 + Q4_0 KV cache, IQ3-MIX can reach 128K context on a 16 GB GPU, while IQ4-MIX is more constrained by VRAM at larger context sizes.

Quantization Recipes

Both recipes use an importance matrix (imatrix) computed from 10 chunks × 2048 context of wikitext-103.

IQ4-MIX — Q4, 4.13 BPW

Tensor family	Quant type	BPW	% of size
FFN gate/up/down	IQ4_XS	4.25	71.5%
Linear attention qkv/gate	IQ3_S	3.44	12.3%
Full attention q/k/v/output	Q4_K	4.58	7.1%
LM head (`output.weight`)	Q5_K	5.33	6.2%
Token embedding	IQ2_S	2.50	2.9%

IQ3-MIX — Q3, 3.68 BPW

Tensor family	Quant type	BPW	% of size
FFN gate/up	IQ3_XXS	3.06	50.3%
FFN down	Q5_K	5.33	13.8%
Full attention q/k/output	IQ3_XXS	3.06	5.2%
Full attention v	Q5_K	5.33	1.8%
Linear attention qkv/gate	IQ2_S	2.50	10.6%
MTP layer (`blk.64.*`)	Q4_K	4.58	8.9%
LM head (`output.weight`)	Q5_K	5.33	6.7%
Token embedding	Q3_K	3.55	2.8%

Both recipes follow the "sensitive trio" philosophy inspired by Unsloth Dynamic:

output.weight
attention value projections (attn_v)
FFN down projections (ffn_down)

These tensors are kept at Q5_K or higher, while more tolerant tensors such as embeddings and linear-attention projections are pushed to lower bit-widths.

Usage

Text + Vision

bash

RADV_PERFTEST=nogttspill llama-server \
  -m Qwen3.8-27B-IQ4-MIX.gguf \
  --mmproj mmproj-F16.gguf \
  -ngl 999 -c 16384 -b 2048 -ub 2048 -t 12 -np 1 \
  --cont-batching --jinja --flash-attn on \
  --cache-type-k q8_0 --cache-type-v q8_0 --kv-unified \
  --host 0.0.0.0 --port 8080

Replace IQ4-MIX with IQ3-MIX when using the Q3 version.

For larger context windows, increase -c. IQ3-MIX supports up to 128K context with Q8_0 + Q4_0 KV cache under the tested configuration.

Download

bash

hf download vmarcelo/Qwen3.8-27B_IQ4-MIX_GGUF \
  --include \
    "Qwen3.8-27B-IQ4-MIX.gguf" \
    "Qwen3.8-27B-IQ3-MIX.gguf" \
    "mmproj-F16.gguf" \
  --local-dir ./

About the Source Model

Qwen3.8-27B-FP8 is a dense multimodal model with the following architecture:

64 transformer layers + 1 MTP layer
Hybrid attention:
- 48 layers use linear attention (GatedDeltaNet-style SSM)
- 16 layers use standard full attention
Vision encoder
Multi-Token Prediction (MTP) head
Vocabulary size: 248,320
Hidden size: 5,120
Context length: 262,144 tokens

Every fourth transformer layer uses standard full attention, with the remaining transformer layers using the linear-attention architecture.

All supported model features, including MTP and multimodal vision, are preserved in both GGUFs.

Acknowledgements

Source model: Qwen/Qwen3.8-27B-FP8 — Qwen team
Quantization tool: llama.cpp — ggml-org
Vulkan build: LM Studio
mmproj: unsloth/Qwen3.8-27B-GGUF

vmarcelo/Qwen3.8-27B-MIX_GGUF

Qwen3.8-27B Custom Mixed-Tensor GGUF Quantizations