Quantizing the abliterated checkpoint (not the base model) means the refusal-removal weight edit is preserved through quantization — this is not a generic base-model AWQ with the refusal behaviour intact.

⚠️ Disclaimer. The underlying model does not refuse requests. This build is published for alignment and safety research — measuring what refusal training protects, red-teaming, and studying refusal-direction mechanics. You are responsible for your use of it and for complying with all applicable laws. Inherits the Apache-2.0 license of the base model.

Latest bug fixes as of 2026-08-16

What changed

checkpoint	size (safetensors)	bits/weight
`Qwen/Qwen3.8-27B` (BF16)	~52 GB	16
`hotdogs/Qwen3.8-27B-abliterated` (BF16)	~52 GB	16
this repo (AWQ-INT4)	~17.6 GB	4

~66 % smaller while keeping the abliterated weights and (for the quantized linear layers) near-lossless W4A16 accuracy.

Quantization details

field	value
tool	llm-compressor (GPTQModifier)
format	`compressed-tensors`, `pack-quantized`
scheme	W4A16 (4-bit weights, 16-bit activations)
group size	128
symmetric	yes
weight dtype	int4
calibration	custom corpus — v6 standard + real Hermes usage (Qwen3.8 chat template) + Thai
ignored tensors	`lm_head` + 64× `linear_attn` (Gated DeltaNet) layers (kept BF16)

Why `linear_attn` is left in BF16

Qwen3.8-27B uses a hybrid architecture — full attention on some layers and Gated DeltaNet linear attention on others. The GDN linear-attention path has no INT4 kernel, so the 64 linear_attn modules are excluded from quantization and remain at full BF16 precision. Everything else (attention QKV/proj, MLP, embeddings) is quantized to INT4.

Usage

vLLM (recommended — Marlin kernel)

bash

CUDA_HOME=/usr/local/cuda-12.8 \
VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve ./Qwen3.8-27B-abliterated-AWQ-INT4 \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --host 0.0.0.0 --port 8000

Then query it:

bash

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.8-27B-abliterated-AWQ-INT4",
    "messages": [{"role": "user", "content": "บอก 3 ข้อดีของประเทศไทย สั้นๆ ภาษาไทย"}],
    "max_tokens": 150
  }'

Notes

--tensor-parallel-size 2 is required on 24 GB GPUs: the model occupies ~22.7 GB per GPU at load (Marlin workspace + BF16 linear_attn), leaving too little for the KV cache on a single 3090. On a bigger GPU (≥ 48 GB) TP=1 works.

VLLM_USE_FLASHINFER_SAMPLER=0 disables the FlashInfer JIT sampler to avoid a CUDA-toolkit/CCCL version mismatch crash on some hosts.

Transformers / Hugging Face

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "hotdogs/Qwen3.8-27B-abliterated-AWQ-INT4",
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("hotdogs/Qwen3.8-27B-abliterated-AWQ-INT4")

Memory requirements

hardware	TP	fits?
2 × RTX 3090 / 4090 (24 GB)	2	✅ recommended
1 × RTX 4090 (24 GB)	1	⚠️ tight — KV cache very small
1 × RTX 5090 / A6000 (48 GB)	1	✅ comfortable

Benchmark (A/B vs base)

This INT4 build quantizes the λ = 1.2 abliterated checkpoint, so its capability tracks the λ = 1.2 base model. The A/B benchmark below was measured on the bf16 λ = 1.2 model (identical lm-eval HF backend, same prompt/config, no chat template); the INT4 quant is expected to land within a point or two of these numbers on the quantized linear layers:

benchmark	base (Qwen3.8-27B)	abliterated (λ=1.2, bf16)	Δ
MMLU (0-shot)	0.8388	0.8342	−0.005
GSM8K (5-shot, strict)	0.62	0.59	−0.03
ARC-Challenge (0-shot, 300)	0.4433	0.4533	+0.010

The λ = 1.2 build preserves capability essentially at base level (MMLU −0.005, GSM8K −0.03, ARC +0.01) while cutting refusal from 98 % → 39 % with KL 0.0001. See the base model card for the full method and the refusal-direction / r̂ weight-level verification.

Reproduction

The quantization was done with llm-compressor on the abliterated checkpoint:

python

from llmcompressor import Oneshot
from llmcompressor.modifiers.quantization import GPTQModifier

recipe = GPTQModifier(
    scheme="W4A16",
    targets=["Linear"],
    ignore=["lm_head"],   # linear_attn auto-excluded by the hybrid arch
)

oneshot = Oneshot(
    model="hotdogs/Qwen3.8-27B-abliterated",
    recipe=recipe,
    dataset="<your calibration corpus>",
    output_dir="./Qwen3.8-27B-abliterated-AWQ-INT4",
    max_seq_length=2048,
    num_calibration_samples=128,
)
oneshot()
oneshot.save()

Model card for the base (abliterated) checkpoint

For full details on the abliteration method, the refusal-direction edit, and reproduction with LLM-abliterate, see the base model card: hotdogs/Qwen3.8-27B-abliterated.

TL;DR — a single forward-only rank-1 weight edit removes the refusal direction from the residual-stream writers (λ = 1.2, hidden-state index 46). No fine-tuning, no distillation, no data poisoning. The vision tower and lm_head are untouched. Same Apache-2.0 license as the base model.

hotdogs/Qwen3.8-27B-abliterated-AWQ-INT4

Qwen3.8-27B-Abliterated-AWQ-INT4