Architecture: Mixture-of-Experts (MoE), 35B total / ~3B active per forward pass
Quant format: IQ4_K_R4 (ik_llama.cpp-specific)
File size: ~19 GB
Source: Re-quantized from Q8_0

Hardware Requirements

Fits on a single 16 GB consumer GPU (e.g. RTX 5060 Ti / 5070 Ti) with some CPU offload for MoE layers.

Recommended launch flags:

bash

/path/to/ik_llama.cpp/build/bin/llama-server \
  --model Qwen3.6-35B-A3B-IQ4_K_R4.gguf \
  -ngl 99 --n-cpu-moe 3 -fa 1 -ctk q4_0 -ctv q4_0 \
  -c 131072 --temp 0.6 --jinja --port 8080

--n-cpu-moe 3 offloads 3 expert layers to CPU, keeping VRAM usage within 16 GB while maintaining full GPU utilization for attention and dense layers.

Context Length

Native context is 262k tokens. Tested flat throughput to 131k context on 16 GB VRAM.

Thinking Mode

Thinking mode is controlled via the enable_thinking parameter:

Via API:

"chat_template_kwargs": {"enable_thinking": false}

Via system prompt (jinja template required): Use the custom template qwen3_nonthinking.jinja to suppress thinking by default.

IQ3_K_R4 version — smaller, ~15 GB, slightly lower quality
ik_llama.cpp — required inference engine

abovespec/Qwen3.6-35B-A3B-IQ4_K_R4-GGUF

Qwen3.6-35B-A3B — IQ4_K_R4 GGUF (ik_llama.cpp)

Model Details

Hardware Requirements

Context Length

Thinking Mode

Related

abovespec/
Qwen3.6-35B-A3B-IQ4_K_R4-GGUF