File	Quant	Size	BPW	Quality
`Qwen3.5-27B-heretic-Q8_0.gguf`	Q8_0	27 GB	8.50	Excellent — near-lossless
`Qwen3.5-27B-heretic-Q6_K.gguf`	Q6_K	21 GB	6.57	Very good
`Qwen3.5-27B-heretic-Q5_K_M.gguf`	Q5_K_M	18 GB	5.77	Good
`Qwen3.5-27B-heretic-Q4_K_M.gguf`	Q4_K_M	15.4 GB	4.92	Recommended for most users
`Qwen3.5-27B-heretic-Q3_K_M.gguf`	Q3_K_M	12.4 GB	3.95	Usable — for low VRAM setups

Multimodal projector (vision)

File	Format	Size
`Qwen3.5-27B-mmproj-BF16.gguf`	BF16	889 MB

How to use

llama-server (recommended for API usage)

bash

llama-server \
    -m Qwen3.5-27B-heretic-Q8_0.gguf \
    --mmproj Qwen3.5-27B-mmproj-BF16.gguf \
    --host 0.0.0.0 \
    --ctx-size 32768 \
    --gpu-layers 99 \
    --tensor-split 1,1 \
    --chat-template-kwargs '{"enable_thinking": false}' \
    --repeat-penalty 1.0 \
    --presence-penalty 1.5 \
    --min-p 0.0 \
    --top-k 20 \
    --top-p 0.8 \
    --temp 0.7

llama-cli

llama-cli -m Qwen3.5-27B-heretic-Q4_K_M.gguf -ngl 99 -c 4096

Which quantization should I use?

Q8_0: Best quality, needs ~28 GB VRAM. Fits on 2x RTX 3090.
Q6_K: Great quality, needs ~22 GB VRAM. Fits on 1x RTX 3090.
Q5_K_M: Good balance of quality and size. Fits on 1x RTX 3090 with room for context.
Q4_K_M: Most popular choice. Fits on 1x RTX 4070 Ti Super / RTX 3090.
Q3_K_M: For constrained VRAM setups (~13 GB needed).

Quantization details

Quantized with llama.cpp convert_hf_to_gguf.py + llama-quantize
Original model: llmfan46/Qwen3.5-27B-heretic

alexdenton/Qwen3.5-27B-heretic-GGUF

Qwen3.5-27B-heretic GGUF

Available quantizations

Multimodal projector (vision)

How to use

llama-server (recommended for API usage)

llama-cli

Which quantization should I use?

Quantization details

alexdenton/
Qwen3.5-27B-heretic-GGUF