File	Quant	Size	BPW	Quality
`Qwen3.5-35B-A3B-heretic-Q8_0.gguf`	Q8_0	34.4 GB	8.52	Excellent — near-lossless
`Qwen3.5-35B-A3B-heretic-Q6_K.gguf`	Q6_K	26.5 GB	6.58	Very good
`Qwen3.5-35B-A3B-heretic-Q5_K_M.gguf`	Q5_K_M	23 GB	5.71	Good
`Qwen3.5-35B-A3B-heretic-Q4_K_M.gguf`	Q4_K_M	19.7 GB	4.88	Recommended for most users
`Qwen3.5-35B-A3B-heretic-Q3_K_M.gguf`	Q3_K_M	15.6 GB	3.87	Usable — for low VRAM setups

Multimodal projector (vision)

File	Format	Size
`Qwen3.5-35B-A3B-mmproj-F16.gguf`	F16	~889 MB

How to use

llama-server (recommended for API usage)

bash

llama-server \
    -m Qwen3.5-35B-A3B-heretic-Q8_0.gguf \
    --mmproj Qwen3.5-35B-A3B-mmproj-F16.gguf \
    --host 0.0.0.0 \
    --ctx-size 32768 \
    --gpu-layers 99 \
    --tensor-split 1,1 \
    --chat-template-kwargs '{"enable_thinking": false}' \
    --repeat-penalty 1.0 \
    --presence-penalty 1.5 \
    --min-p 0.0 \
    --top-k 20 \
    --top-p 0.8 \
    --temp 0.7

llama-cli

llama-cli -m Qwen3.5-35B-A3B-heretic-Q4_K_M.gguf -ngl 99 -c 4096

Which quantization should I use?

Q8_0: Best quality, needs ~35 GB VRAM. Fits on 2x RTX 3090.
Q6_K: Great quality, needs ~27 GB VRAM. Fits on 2x RTX 3090 with room for context.
Q5_K_M: Good balance. Fits on 1x RTX 3090 (24 GB).
Q4_K_M: Most popular choice. Fits on 1x RTX 3090 with context room.
Q3_K_M: For constrained VRAM setups (~16 GB needed).

Quantization details

Quantized with llama.cpp convert_hf_to_gguf.py + llama-quantize
Original model: brayniac/Qwen3.5-35B-A3B-heretic

alexdenton/Qwen3.5-35B-A3B-heretic-GGUF

Qwen3.5-35B-A3B-heretic GGUF

Available quantizations

Multimodal projector (vision)

How to use

llama-server (recommended for API usage)

llama-cli

Which quantization should I use?

Quantization details

alexdenton/
Qwen3.5-35B-A3B-heretic-GGUF