[!TIP] KV-cache quantization without any fork (recommended, 2026): upstream llama.cpp/Ollama now cover this natively — use -ctk q8_0 -ctv q8_0 (~half KV memory, negligible quality loss: perplexity +0.002–0.05) or -ctk q4_0 -ctv q4_0 (~quarter memory, ≈7.6% perplexity increase). In Ollama: OLLAMA_KV_CACHE_TYPE=q8_0 with OLLAMA_FLASH_ATTENTION=1. Keep K and V types symmetric to stay on the fast fused Flash-Attention path. Since April 2026, mainline llama.cpp also applies Hadamard rotation to KV activations (PR #21038), which greatly improves low-bit KV quality (opt-out: LLAMA_ATTN_ROT_DISABLE=1).

The RotorQuant/TurboQuant fork flow below is experimental/legacy: the TurboQuant llama.cpp PR was closed without merging (June 2026) and the fork is unmaintained relative to mainline. It is NOT required to use this model.

Qwen3.6-35B-A3B-RotorQuant-GGUF-Q5_K_M

GGUF Q5_K_M weight-quantized variant of Qwen/Qwen3.6-35B-A3B optimised for use with RotorQuant KV cache compression via a dedicated llama.cpp fork.

Important: RotorQuant KV cache types (planar3, iso3) are not available in upstream llama.cpp, standard Ollama, or LM Studio. They require a specific llama.cpp fork. The GGUF file itself is a standard GGUF and works with any llama.cpp-compatible runtime using normal KV cache types (f16, q8_0, q4_0, etc.).

Hardware compatibility

Device	VRAM / RAM	Recommendation
CPU host with ≥26 GB RAM	~26.2 GB	works via llama.cpp; slower than GPU but no accelerator required
Apple Silicon (Metal)	~28.6 GB	llama.cpp Metal backend; fast on M-series unified memory
NVIDIA GPU (partial offload)	split between GPU + RAM	offload as many layers as VRAM allows; rest on CPU

Overview

This model combines two independent compression techniques:

Technique	What it does	Requirement
GGUF Q5_K_M weight quantization	Reduces model size from ~70 GB (BF16) to ~23.8 GB	Any llama.cpp-compatible runtime
RotorQuant KV cache compression — block-diagonal Clifford-algebra rotors for 3-bit KV cache (`--cache-type-k iso3 --cache-type-v iso3`)	Block-diagonal rotations / random rotation for compressed KV cache	llama-cpp-turboquant fork only

Quickstart

Option A — RotorQuant KV cache (experimental fork — not required)

You must build from the RotorQuant-enabled llama.cpp fork:

bash

# Clone and build the fork
git clone https://github.com/johndpope/llama-cpp-turboquant.git
cd llama-cpp-turboquant && git checkout feature/planarquant-kv-cache

# CUDA (Windows/Linux)
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build -j

# Metal (Apple Silicon)
cmake -B build -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build -j

# Run with RotorQuant KV cache
./build/bin/llama-cli -m Qwen3.6-35B-A3B-RotorQuant-GGUF-Q5_K_M.gguf \
  --cache-type-k iso3 --cache-type-v iso3 \
  -ngl 99 -fa \
  -p "Explain quantum computing"

# Or run as a server
./build/bin/llama-server -m Qwen3.6-35B-A3B-RotorQuant-GGUF-Q5_K_M.gguf \
  --cache-type-k iso3 --cache-type-v iso3 \
  -ngl 99 -fa --jinja

Option B — With standard llama.cpp / LM Studio / Ollama

The GGUF works as a normal quantised model. You won't get RotorQuant-specific KV cache benefits, but standard KV cache quantization (q8_0, q4_0) still reduces VRAM significantly.

llama.cpp (upstream)

bash

llama-cli -m Qwen3.6-35B-A3B-RotorQuant-GGUF-Q5_K_M.gguf \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -ngl 99 -fa \
  -p "Explain quantum computing"

LM Studio

Download the GGUF file and load in LM Studio.
Enable Developer Mode (Settings → Developer).
In the model loader's advanced settings, set Flash Attention to ON.
Set K Cache Quantization and V Cache Quantization to q8_0 (or q4_0 for more aggressive VRAM savings).
Note: LM Studio does not currently support RotorQuant's iso3 cache types. Track this feature request for updates.

Ollama

bash

# Standard Ollama does not support RotorQuant cache types.
# Use with default or q8_0 KV cache via OLLAMA_KV_CACHE_TYPE=q8_0
OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_FLASH_ATTENTION=1 ollama run majentik/Qwen3.6-35B-A3B-RotorQuant-GGUF-Q5_K_M

Specifications

Property	Value
Base Model	Qwen/Qwen3.6-35B-A3B
Architecture	Sparse MoE (3B active per token), instruct-tuned
Parameters	35B total, 3B active per token (MoE)
Context Length	262K native
Weight Quantization	GGUF Q5_K_M (high quality, balanced 5-bit)
Original Size (BF16)	~70 GB (approx.)
Quantized File Size	~23.8 GB
KV Cache (RotorQuant)	3-bit via `--cache-type-k iso3 --cache-type-v iso3` (fork only)
KV Cache (standard)	q8_0, q4_0, f16, etc. (any llama.cpp runtime)
License	apache-2.0
Modalities	Text + Image + Video (multimodal)
Compatible Runtimes	llama.cpp, LM Studio, Ollama, koboldcpp

About the RotorQuant / TurboQuant labels

RotorQuant and TurboQuant are this project's release labels, not distinct quantization algorithms — for any given tier, both brand repos carry byte-identical weights produced with the standard MLX / llama.cpp quantizers. No brand-specific speedup is claimed or measured. The KV-cache fork these labels originally referred to is legacy; for KV-cache memory savings use the upstream options described above (-ctk/-ctv q8_0, OLLAMA_KV_CACHE_TYPE).

Current Status of RotorQuant in the Ecosystem

Runtime	RotorQuant Support	Standard KV Quant
llama.cpp (upstream)	❌ Not merged	✅ q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1
llama-cpp-turboquant fork	✅ planar3, iso3	✅ All standard types
LM Studio	❌ Requested	✅ Via advanced settings
Ollama	❌ Not supported	✅ Via OLLAMA_KV_CACHE_TYPE
koboldcpp	❌ Not supported	✅ Standard types

Recommended Settings

For VRAM-constrained setups, standard q8_0 KV cache quantization already halves KV cache memory with negligible quality impact. Flash Attention should always be enabled — it is required for V cache quantization and improves memory efficiency regardless.

VRAM	Suggested Configuration
24 GB (RTX 4090)	Q5_K_M + q8_0 KV cache + Flash Attention, 8K–16K context
16 GB	Q5_K_M + q4_0 KV cache + Flash Attention, 4K–8K context
48+ GB	Q5_K_M + f16 KV cache, full 32K+ context

Quant trade-off (GGUF lane)

Quant	Approx size	Use case	Recommendation
Q2_K	~19 GB	Lossy, low-RAM CPU/edge	Resource-constrained inference
Q3_K_M	~21 GB	Smaller-than-Q4, modest quality drop	Edge devices with ~16 GB RAM
IQ4_XS	~18 GB	Importance-quant 4-bit, smaller than Q4_K_M	Best size/quality at 4-bit
Q4_K_M	~26 GB	Balanced default	Recommended for most users
Q5_K_M	~27 GB	Higher fidelity than Q4	Quality-sensitive applications
Q6_K	~32 GB	Approaching FP16 quality	High-fidelity CPU/edge
Q8_0	~36 GB	Near-lossless reference	Fidelity-critical work
MXFP4_MOE	~19 GB	Microscaling FP4 (MoE-aware)	vLLM / transformers users

(Current variant — Q5_K_M — is bolded.)

Variants in this family

(Showing 24 sibling variants under majentik/qwen3.6-35b-a3b-*. The current variant — RotorQuant-GGUF-Q5_K_M — is bolded.)

Variant	Runtime	Approx size	Use case
RotorQuant-GGUF-IQ4_XS	llama.cpp	~30 GB	Lossy 4-bit, low-RAM CPU/edge
RotorQuant-GGUF-Q2_K	llama.cpp	~21 GB	Lossy, low-RAM CPU/edge
RotorQuant-GGUF-Q3_K_M	llama.cpp	~27 GB	Smaller 3-bit, CPU-friendly
RotorQuant-GGUF-Q4_K_M	llama.cpp	~38 GB	Balanced default
RotorQuant-GGUF-Q5_K_M	llama.cpp	~46 GB	Higher fidelity, more RAM
RotorQuant-GGUF-Q8_0	llama.cpp	~74 GB	Near-lossless reference
RotorQuant-MLX-2bit	mlx-lm	~11 GB	Apple Silicon, smallest
RotorQuant-MLX-5bit	mlx-lm	~27 GB	Apple Silicon, higher fidelity
RotorQuant-MLX-6bit	mlx-lm	~32 GB	Apple Silicon, near-lossless
RotorQuant-MLX-8bit	mlx-lm	~41 GB	Apple Silicon reference
TurboQuant-MLX-3bit	mlx-lm	~16 GB	Apple Silicon, small
TurboQuant-MLX-4bit	mlx-lm	~22 GB	Apple Silicon balanced

majentik/Qwen3.6-35B-A3B-RotorQuant-GGUF-Q5_K_M

Qwen3.6-35B-A3B-RotorQuant-GGUF-Q5_K_M

Hardware compatibility

Overview

Quickstart

Option A — RotorQuant KV cache (experimental fork — not required)

Option B — With standard llama.cpp / LM Studio / Ollama

Specifications

About the RotorQuant / TurboQuant labels

Current Status of RotorQuant in the Ecosystem

Recommended Settings

See Also

Quant trade-off (GGUF lane)

Variants in this family

majentik/
Qwen3.6-35B-A3B-RotorQuant-GGUF-Q5_K_M