These quantizations were compiled and converted using high-performance compute environments to ensure broad compatibility with modern GGML-based inferencing engines.

Available Quantizations

The following quantization methods are provided to suit various hardware capabilities, from VRAM-constrained setups to high-end rigs.

Format	Characteristics	Recommended Use
Q3_K_M	Smallest, highest perplexity loss	Maximum space savings; quality degradation is noticeable.
Q4_K_S	Small, slightly higher perplexity	Good balance for tight VRAM limits.
Q4_K_M	Medium, optimal balance	Recommended - Excellent balance of size, speed, and quality.
Q5_K_M	Large, very low perplexity loss	High quality, requires more VRAM.
Q6_K	Very large, near-lossless	Premium quality for high-VRAM systems.
Q8_0	Largest, practically lossless	Best for compute-heavy setups; requires massive VRAM.

Usage with llama.cpp

You can run these models using the latest versions of llama.cpp. Note that recent versions of llama.cpp use the cmake build system.

CLI Example

To run inference via the command line:

./build/bin/llama-cli -m Qwen3.6-35B-A3B-heretic-Q4_K_M.gguf -p "Explain the concept of quantum entanglement." -n 512

Abiray/Qwen3.6-35B-A3B-heretic-GGUF

Qwen3.6-35B-A3B-heretic GGUF

Available Quantizations

Usage with llama.cpp

CLI Example

Abiray/
Qwen3.6-35B-A3B-heretic-GGUF