We provide a spectrum of quantizations to suit different hardware capabilities. If you are unsure which to choose, Q4_K_M is the recommended standard for the best balance of speed, size, and reasoning retention.

Filename	Quant Type	File Size	Recommended RAM/VRAM	Use Case
`Huihui-Qwen3.5-9B-abliterated-Q4_K_S.gguf`	Q4_K_S	~5.1 GB	8 GB	Fastest inference, lowest memory footprint.
`Huihui-Qwen3.5-9B-abliterated-Q4_K_M.gguf`	Q4_K_M	~5.4 GB	8 GB	Recommended. Best balance of quality and size.
`Huihui-Qwen3.5-9B-abliterated-Q5_K_S.gguf`	Q5_K_S	~6.1 GB	10 GB	Slightly higher quality, needs more memory.
`Huihui-Qwen3.5-9B-abliterated-Q5_K_M.gguf`	Q5_K_M	~6.3 GB	10 GB	Excellent reasoning retention for 12GB+ GPUs.
`Huihui-Qwen3.5-9B-abliterated-Q6_K.gguf`	Q6_K	~7.4 GB	12 GB	Near-lossless quality.
`Huihui-Qwen3.5-9B-abliterated-Q8_0.gguf`	Q8_0	~9.6 GB	16 GB	Virtually indistinguishable from the F16 base model.

🚀 How to Use

These GGUF files are fully compatible with popular local AI software, including:

LM Studio
Ollama
Jan
text-generation-webui (Oobabooga)
llama.cpp

Prompt Format (ChatML)

Qwen models strictly use the ChatML format for prompting. Your system prompt and user inputs must be wrapped exactly like this:

text

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Write a simple Python script.<|im_end|>
<|im_start|>assistant

Abiray/Huihui-Qwen3.5-9B-abliterated-GGUF

Huihui-Qwen3.5-9B-abliterated - GGUF

💾 Provided Files and Quantizations

🚀 How to Use

Prompt Format (ChatML)

Abiray/
Huihui-Qwen3.5-9B-abliterated-GGUF