This quant was built specifically for high-throughput local inference on a 16 GB GPU while preserving the model's built-in MTP (Multi-Token Prediction) head for speculative decoding in llama.cpp.

The main GGUF contains the MTP tensors. No separate draft model is required when using:

--spec-type draft-mtp

File

File	Size	BPW
`Qwen3.8-27B-Abliterated-IQ4-MIX-MTP.gguf`	~13.26 GiB	4.17

Quantizer output:

text

model size = 52115.19 MiB (16.00 BPW)
quant size = 13573.45 MiB (4.17 BPW)

Quantization layout

This is not a uniform IQ4_XS quant. Different tensor groups use different precisions.

Tensor group	Quant
`token_embd.weight`	`IQ2_S`
`output.weight`	`Q5_K`
body `ffn_gate / ffn_up / ffn_down`	`IQ4_XS`
linear-attention `attn_qkv`	`IQ3_S`
linear-attention `attn_gate`	`IQ3_S`
full-attention `attn_q / k / v / output`	`Q4_K`
remaining quantizable SSM/body tensors	`IQ4_XS`
MTP block 64 `attn_q / k / v / output`	`Q6_K`
MTP block 64 `ffn_gate / up / down`	`Q6_K`
MTP `nextn.eh_proj`	`Q8_0`

Small F32 tensors, norms, biases and state tensors that were not suitable for quantization remain in their original precision.

Why the MTP block is higher precision

The MTP block was intentionally protected with Q6_K, and nextn.eh_proj with Q8_0, instead of compressing it to the body quantization level.

The goal is to preserve speculative-drafting quality and acceptance rate. The additional VRAM/storage cost is small relative to the full 27B model.

Importance matrix

An importance matrix was generated from:

Dataset: Salesforce/wikitext
Config: wikitext-103-raw-v1
Context per chunk: 2048
Chunks: 10
Importance-matrix entries: 496

Command used:

powershell

.\llama-imatrix.exe `
  -m .\Qwen3.8-27B-Abliterated-BF16.gguf `
  -f .\calibration.txt `
  -o .\Qwen3.8-27B-Abliterated-imatrix.gguf `
  -c 2048 `
  --chunks 10 `
  --no-ppl `
  -ngl 18

The MTP block itself is not exercised by the standard imatrix forward pass, so it was explicitly assigned conservative Q6_K/Q8_0 types during quantization.

Quantization command

Built with llama.cpp build 10441 / commit 0177dcc73.

powershell

.\llama-quantize.exe `
  --imatrix .\Qwen3.8-27B-Abliterated-imatrix.gguf `
  --tensor-type '^blk\.64\.nextn\.eh_proj\.weight$=q8_0' `
  --tensor-type '^blk\.64\.attn_q\.weight$=q6_k' `
  --tensor-type '^blk\.64\.attn_k\.weight$=q6_k' `
  --tensor-type '^blk\.64\.attn_v\.weight$=q6_k' `
  --tensor-type '^blk\.64\.attn_output\.weight$=q6_k' `
  --tensor-type '^blk\.64\.ffn_gate\.weight$=q6_k' `
  --tensor-type '^blk\.64\.ffn_up\.weight$=q6_k' `
  --tensor-type '^blk\.64\.ffn_down\.weight$=q6_k' `
  --tensor-type 'ffn_gate\.weight$=iq4_xs' `
  --tensor-type 'ffn_up\.weight$=iq4_xs' `
  --tensor-type 'ffn_down\.weight$=iq4_xs' `
  --tensor-type 'attn_qkv\.weight$=iq3_s' `
  --tensor-type 'attn_gate\.weight$=iq3_s' `
  --tensor-type 'attn_q\.weight$=q4_k' `
  --tensor-type 'attn_k\.weight$=q4_k' `
  --tensor-type 'attn_v\.weight$=q4_k' `
  --tensor-type 'attn_output\.weight$=q4_k' `
  --token-embedding-type iq2_s `
  --output-tensor-type q5_k `
  .\Qwen3.8-27B-Abliterated-BF16.gguf `
  .\Qwen3.8-27B-Abliterated-IQ4-MIX-MTP.gguf `
  iq4_xs `
  14

Tested hardware

Tested on:

GPU: AMD Radeon RX 9070 XT 16 GB
Backend: Vulkan
OS: Windows
llama.cpp: build 10441, commit 0177dcc73
GPU offload: all model layers
Parallel slots: 1

Measured performance

These numbers are from local tests and should be treated as hardware/workload-specific, not universal benchmarks.

32K allocated context, MTP-3

text

eval time = 81200.04 ms / 4070 tokens
generation = 50.11 tokens/s
draft acceptance = 0.74563
2814 accepted / 3774 generated
mean len = 3.24

64K allocated context, MTP-3

text

eval time = 182513.24 ms / 9018 tokens
generation = 49.40 tokens/s
draft acceptance = 0.77141
6297 accepted / 8163 generated
mean len = 3.31

These tests started with relatively short prompts. Decode speed falls as the live context grows. In agentic coding workloads with roughly 25K–30K tokens of active history, observed generation was closer to the mid-20 tokens/s range depending on MTP acceptance.

Recommended llama.cpp server command — 64K

powershell

.\llama-server.exe `
  -m .\Qwen3.8-27B-Abliterated-IQ4-MIX-MTP.gguf `
  --alias qwen38-local `
  -ngl all `
  --fit off `
  --jinja `
  -fa on `
  -np 1 `
  -c 65536 `
  -b 2048 `
  -ub 256 `
  -ctk q8_0 `
  -ctv q4_0 `
  --kv-unified `
  --reasoning on `
  --reasoning-budget 2048 `
  --spec-type draft-mtp `
  --spec-draft-n-max 3 `
  --spec-draft-type-k q8_0 `
  --spec-draft-type-v q4_0 `
  --host 127.0.0.1 `
  --port 8080

For maximum raw throughput, keep the active conversation/context reasonably short. A 64K allocation does not mean generation speed will remain constant when all 64K tokens are occupied.

MTP notes

The most reliable tested configuration for this quant was:

text

--spec-type draft-mtp
--spec-draft-n-max 3

Higher speculative draft lengths are not automatically faster. In local testing, n-max 8 with a high p-min performed substantially worse on the tested Windows/Vulkan setup.

Download with Hugging Face CLI

Replace YOUR_USERNAME with the repository owner:

powershell

hf download YOUR_USERNAME/Qwen3.8-27B-Abliterated-IQ4-MIX-MTP-GGUF `
  Qwen3.8-27B-Abliterated-IQ4-MIX-MTP.gguf `
  --local-dir .

Notes on quality

This is a lossy quantization of the BF16 source model.
The embedding tensor is aggressively compressed to IQ2_S.
The output head is retained at Q5_K.
The MTP block is deliberately higher precision than the main body.
No comprehensive downstream benchmark suite has been run on this exact GGUF quantization.
Results can differ between Vulkan, HIP/ROCm, CUDA, CPU backends, context lengths and sampling settings.

Abliterated model notice

The upstream checkpoint is an abliterated variant. This repository does not perform additional fine-tuning or alignment changes; it provides a GGUF quantization of that source checkpoint.

Users are responsible for evaluating the model for their intended use and complying with applicable laws, licenses and platform policies.

Credits

Base model: Qwen/Qwen3.8-27B
Abliterated checkpoint: windowsxp811203/Qwen3.8-27B-Abliterated
Inference / quantization: llama.cpp
Importance-matrix calibration text: Salesforce/wikitext

License

The source checkpoint metadata reports Apache-2.0. Please also review the upstream model repositories and their terms before redistribution or use.

finex666/Qwen3.8-27B-Abliterated-IQ4-MIX-MTP-GGUF

Qwen3.8-27B-Abliterated — IQ4-MIX-MTP GGUF

File

Quantization layout

Why the MTP block is higher precision

Importance matrix

Quantization command

Tested hardware

Measured performance

32K allocated context, MTP-3

64K allocated context, MTP-3

Recommended llama.cpp server command — 64K

MTP notes

Download with Hugging Face CLI

Notes on quality

Abliterated model notice

Credits

License

finex666/
Qwen3.8-27B-Abliterated-IQ4-MIX-MTP-GGUF