You can deploy this model and start earning money today!

sakamakismile/
Huihui-Qwen3.6-27B-abliterated-NVFP4

Name: Huihui-Qwen3.6-27B-abliterated-NVFP4
Availability: InStock

Views

No views yet

Huihui-Qwen3.6-27B-abliterated-NVFP4

NVFP4 quantized version of huihui-ai/Huihui-Qwen3.6-27B-abliterated — an abliterated (uncensored) variant of Qwen3.6-27B, the dense 27B VLM with Gated DeltaNet hybrid attention.

Quantized to NVIDIA FP4 by Lna-Lab using custom Blackwell NVFP4 GEMM kernels (lna-lab/blackwell-geforce-nvfp4-gemm).

55.6 GB → 19.7 GB (0.35x) — vision tower preserved in BF16. Runs on a single NVIDIA Blackwell GPU.

Key Specs


Base model	huihui-ai/Huihui-Qwen3.6-27B-abliterated
Original	Qwen/Qwen3.6-27B
Architecture	Dense 27B, Gated DeltaNet + Gated Attention hybrid, VLM
Quantization	NVFP4 (W4A4 — weights FP4, activations FP4, scales FP8)
Format	`compressed-tensors` (native vLLM support)
Tool	vllm-project/llm-compressor + blackwell-geforce-nvfp4-gemm
Size	19.7 GB
Requires	NVIDIA Blackwell GPU (SM 120), vLLM >= 0.19

Benchmark Results

Tested on a single NVIDIA RTX PRO 6000 Blackwell (96 GB), vLLM 0.19.1+, 128K context, FP8 KV cache.

Task	Tokens	Speed (tok/s)	Status
English reasoning	1,024	56.2	PASS
Japanese essay (方丈記)	2,048	59.7	PASS
Python code generation	2,048	59.1	PASS
Contradictory instructions	1,500	59.5	PASS
VLM image description	947	58.1	PASS
Math proof (√2 irrationality)	1,024	59.3	PASS

Sustained throughput: ~58 tok/s (single GPU, 128K context, FP8 KV cache)

VRAM Usage

State	GPU Memory
After model load	92,142 MiB
Peak (during inference)	92,150 MiB

Quick Start — From Scratch with Docker

1. Pull the model

bash

hf download sakamakismile/Huihui-Qwen3.6-27B-abliterated-NVFP4 \
  --local-dir /models/Huihui-Qwen3.6-27B-abliterated-NVFP4

2. Run with Docker (128K context + FP8 KV cache)

bash

docker run -d --name huihui-qwen36-27b \
  --gpus '"device=0"' --shm-size=16g \
  -v /models/Huihui-Qwen3.6-27B-abliterated-NVFP4:/models/current:ro \
  -p 8000:8000 \
  vllm/vllm-openai:cu130-nightly \
  --model /models/current \
  --trust-remote-code --quantization modelopt --language-model-only \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --default-chat-template-kwargs '{"preserve_thinking":true}' \
  --enable-prefix-caching --enable-chunked-prefill \
  --max-model-len 131072 --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8_e4m3

3. Run with vLLM directly

bash

vllm serve /models/Huihui-Qwen3.6-27B-abliterated-NVFP4 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.95 \
  --dtype auto \
  --kv-cache-dtype fp8_e4m3 \
  --trust-remote-code

4. Test inference

bash

# Text
curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Huihui-Qwen3.6-27B-abliterated-NVFP4",
    "messages": [{"role": "user", "content": "Write a haiku about quantization."}],
    "max_tokens": 256,
    "temperature": 0.0
  }'

python

# VLM (image input)
import base64, requests
from pathlib import Path

img_b64 = base64.b64encode(Path("photo.jpg").read_bytes()).decode()
resp = requests.post("http://localhost:8000/v1/chat/completions", json={
    "model": "Huihui-Qwen3.6-27B-abliterated-NVFP4",
    "messages": [{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}},
        {"type": "text", "text": "Describe this image."},
    ]}],
    "max_tokens": 1024,
})
print(resp.json()["choices"][0]["message"]["content"])

Quantization Details

Recipe

yaml

QuantizationModifier:
  targets: [Linear]
  ignore: [lm_head, 're:.*visual.*', 're:.*mlp.gate$', 're:.*mlp.shared_expert_gate$']
  scheme: NVFP4

What's Quantized / What's Not

Quantized (NVFP4): All Linear layers in the language model
Kept in BF16: lm_head, all vision layers (model.visual.*), MLP gates

Reproduction

python

from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
import torch

model = Qwen3_5ForConditionalGeneration.from_pretrained(
    "huihui-ai/Huihui-Qwen3.6-27B-abliterated",
    torch_dtype=torch.bfloat16, trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained(
    "huihui-ai/Huihui-Qwen3.6-27B-abliterated", trust_remote_code=True,
)

recipe = QuantizationModifier(
    targets="Linear", scheme="NVFP4",
    ignore=["lm_head", "re:.*visual.*", "re:.*mlp.gate$", "re:.*mlp.shared_expert_gate$"],
)

# Calibration with neuralmagic/calibration dataset (20 samples, 8192 seq len)
# ... (see quantization script in repo)

model.save_pretrained("output-NVFP4", save_compressed=True)
processor.save_pretrained("output-NVFP4")

Note: After saving, verify vision tower keys use model.visual.* prefix (not model.language_model.visual.*). See this fix for details.

Tested Environment

Component	Version
vLLM	0.19.1rc1+ (nightly)
Transformers	5.5.4
PyTorch	2.11.0+cu130
llm-compressor	0.1.dev5
CUDA	13.0
GPU	NVIDIA RTX PRO 6000 Blackwell (96 GB)
OS	Ubuntu 24.04, Linux 6.17

Credits

Original model: Qwen Team (Alibaba Group) — Qwen3.6-27B
Abliteration: huihui-ai — Huihui-Qwen3.6-27B-abliterated
NVFP4 quantization & benchmarking: Lna-Lab
Blackwell NVFP4 GEMM kernels: lna-lab/blackwell-geforce-nvfp4-gemm
Quantization framework: vllm-project/llm-compressor

Support the Base Model Authors

If you find this model useful, please consider supporting:

huihui-ai (abliteration): Ko-fi | BTC: bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge
Qwen Team (original model): Star the Qwen repo

License

This model inherits the Apache 2.0 license.