GGUF conversion of huihui-ai / Qwen3.8-27B-abliterated (uncensored) quantized to NVFP4, converted from the source NVFP4 safetensors by sakamakismile.

Upstream lineage:

Qwen/Qwen3.8-27B (Apache-2.0) — base model
huihui-ai/Huihui-Qwen3.8-27B-abliterated — abliterated (uncensored) fine-tune
sakamakismile/Huihui-Qwen3.8-27B-abliterated-NVFP4 — NVFP4 quantized safetensors (compressed-tensors, nvfp4-pack-quantized)
This repo — lossless repack to GGUF via convert_hf_to_gguf.py (no re-quantization, no precision loss)

Files

File	Size	Description
`Qwen3.8-27B-huihui-NVFP4.gguf`	19.65 GB	Main model. NVFP4 MLP + attention, Q5_K embeddings, BF16 MTP head, 262K native context
`mmproj-huihui.gguf`	931 MB	BF16 vision projector (mmproj) for image input

Architecture: qwen35 (Qwen3_5ForConditionalGeneration), 64 layers + 1 MTP layer (nextn_predict_layers=1)
Bits per weight: ~5.6 BPW
The MTP (multi-token prediction) head ships in the file — speculative decoding works out of the box in llama.cpp / LM Studio
Vision projector enables image input (Qwen3-VL path); load it alongside the main model.

Load in LM Studio / llama.cpp

bash

# llama.cpp: text-only
llama-server \
  -m Qwen3.8-27B-huihui-NVFP4.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  -c 163840 \
  -ngl 999

# llama.cpp: with vision
llama-server \
  -m Qwen3.8-27B-huihui-NVFP4.gguf \
  --mmproj mmproj-huihui.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  -c 131072 \
  -ngl 999

LM Studio settings

Draft probability: 0 — critical. Default 0.75 rejects ~60% of correct MTP drafts and collapses speculative speed to base rate (60-70 t/s). Setting 0 accepts all drafts and unlocks ~120 t/s.
Min draft tokens: 0–2
Max draft tokens: 2–3 (acceptance collapses at ≥4)
KV cache quant: q8_0 recommended

Measured speed — RTX 5090 (32 GB), LM Studio, 2026-08

MTP acceptance rate drives speed; content type drives acceptance rate:

Content	Draft acceptance	Speed
Code / JSON	80–96%	117–129 t/s
Math reasoning	71%	114 t/s
Chinese / English prose	37–48%	88–91 t/s

Notes:

Short outputs (<50 tokens) measure low due to prefill amortization — irrelevant to real use.
The ~8% gap vs Q8-attention builds (e.g. utautako/Qwen3.8-27B-NVFP4-MTP-Q8attn-GGUF) is the price of full-NVFP4 attention; it buys smaller size and full abliteration.

How it was converted

Lossless repack from compressed-tensors NVFP4 safetensors — no dequantization → requantization round trip:

bash

git clone --depth 1 https://github.com/ggml-org/llama.cpp
pip install -r requirements.txt  # torch + numpy + pyyaml + transformers

python convert_hf_to_gguf.py ./Huihui-Qwen3.8-27B-abliterated-NVFP4 \
  --outfile Qwen3.8-27B-huihui-NVFP4.gguf --outtype auto
python convert_hf_to_gguf.py ./Huihui-Qwen3.8-27B-abliterated-NVFP4 \
  --outfile mmproj-huihui.gguf --mmproj

Conversion is pure CPU (mmap, no VRAM), ~1 minute for 27B on a modern desktop.

Why this model

Uncensored (abliterated) — no safety refusals
NVIDIA NVFP4 — native Blackwell FP4 tensor cores, faster than GGUF Q4/Q5 k-quants at similar size
MTP speculative decoding — roughly doubles throughput over autoregressive baseline (65 → ~120 t/s on code)
Vision — image input supported via mmproj

License

Apache-2.0. Base model: Qwen/Qwen3.8-27B. Fine-tune: huihui-ai. Quantization source: sakamakismile. GGUF conversion: renketong.

renketong/Huihui-Qwen3.8-27B-abliterated-NVFP4-GGUF

Huihui-Qwen3.8-27B-abliterated-NVFP4-GGUF