Native 3-bit TurboQuant checkpoint of Qwen/Qwen3.6-35B-A3B. Pre-fused expert weights stored as packed 3-bit indices with per-group norms. ~16 GB on disk instead of ~70 GB BF16.

This is the open-weight Qwen3.6 family variant: 35B total / 3B active MoE with 256 experts, hybrid GatedDeltaNet + partial-rotary attention. Tested end-to-end on A100 80GB, RTX PRO 6000 Blackwell 96GB, H100 80GB, and L40S 48GB.

Usage

Requires turboquant-plus-vllm v0.13.5 or later and vLLM 0.20.2+.

bash

pip install vllm>=0.20.2
pip install 'turboquant-plus-vllm@git+https://github.com/varjoranta/turboquant-vllm.git'

bash

vllm serve varjosoft/Qwen3.6-35B-A3B-TQ3-native \
    --quantization turboquant \
    --trust-remote-code \
    --max-model-len 4096

Python:

python

from vllm import LLM, SamplingParams

llm = LLM(
    model="varjosoft/Qwen3.6-35B-A3B-TQ3-native",
    quantization="turboquant",
    trust_remote_code=True,
    max_model_len=4096,
)
out = llm.generate(["Explain quantum entanglement in one paragraph."],
                   SamplingParams(temperature=0, max_tokens=200))
print(out[0].outputs[0].text)

Results

Validated on A100 80GB (sm_80), vLLM 0.20.2, CUDA 13.0, Triton 3.6.0:

Metric	Value
Checkpoint size	~16 GB (vs ~70 GB BF16)
Compression ratio	4.4×
Throughput, CUDA graphs ON	16.0 tok/s at bs=1 (reproduced across 4 runs: 16.02 / 15.94 / 15.88 / 16.0)
Throughput, eager mode	9.81 tok/s at bs=1
GSM8K-200 (5-shot CoT)	eval workflow exceeds the validation harness's polling window; standalone eval to follow
Load time	~10 min from cold (includes vLLM init + kernel compile)
Minimum GPU	A100 80GB tested. RTX PRO 6000 Blackwell 96GB also validated end-to-end

Output sample (greedy, 4 prompts, max_tokens=128):

"Q: Explain quantum entanglement in one short paragraph. A: Here's a thinking process: 1. Analyze User Input: Topic: Quantum entanglement, Format: ..."

(Qwen3.6 generates a visible reasoning trace by default; production use should set the system prompt accordingly.)

What This Checkpoint Contains

model-0000{1..4}-of-00004.safetensors: packed 3-bit weight indices (.tq_packed) + per-group norms (.tq_norms) for all 41 MoE layers (pre-fused per layer as experts.gate_up_proj.tq_* and experts.down_proj.tq_*) and the 64 attention layers; FP16 for embeddings, RMSNorms, biases, GatedDeltaNet conv1d/A_log/dt_bias.
tq_config.json: {"bits": 3, "group_size": 128, "format": "tq3_native"}.
config.json, chat_template.jinja, tokenizer.json, tokenizer_config.json, preprocessor_config.json, generation_config.json: standard HuggingFace artifacts.

How It Was Made

python

from turboquant_vllm.checkpoint import save_tq3_checkpoint

save_tq3_checkpoint("Qwen/Qwen3.6-35B-A3B", "./qwen3.6-35b-tq3", bits=3)
# CPU only, ~80 GB RAM during compression, ~10 minutes. No GPU needed.

Each weight tensor is read lazily from the source safetensors, rotated with a Walsh-Hadamard transform, quantized to 3 bits against a Gaussian Lloyd-Max codebook, and saved with per-group norms. For MoE layers, the expert tensors are saved pre-fused per layer (gate_up_proj = gate+up stacked, down_proj = down across all experts) — the v0.13.5 plugin loader detects this layout and skips the per-expert regroup.

Non-weight tensors (embeddings, norms, biases, GatedDeltaNet state tensors) are kept in FP16.

Architecture Specifics

Qwen3.6-35B-A3B uses three patterns the loader must handle correctly:

Multimodal name-mapping — Qwen3_5MoeForConditionalGeneration exposes hf_to_vllm_mapper that rewrites model.language_model.* → language_model.model.*. v0.13.5 applies the mapper before regroup target lookup.
Pre-fused MoE storage — per-layer experts saved as .experts.gate_up_proj.tq_packed (no per-expert index, no .weight suffix). v0.13.5's _NATIVE_MOE_PRE_FUSED_PATTERN detects this and yields directly to the placeholder param names (w13_weight_tq_packed / w2_weight_tq_packed).
Partial-rotary attention (partial_rotary_factor=0.25) — uses the block-diagonal WHT CUDA kernel from v0.13.0 to dequant the rotary projections without falling back to Python.

The 16 tok/s graphs-on result resolves the v0.13.0 graphs regression: on vLLM 0.20.2 CUDA graphs are now 1.6× FASTER than eager for this checkpoint, not 3.3× slower.

Algorithm

Inspired by TurboQuant (Zandieh, Daliri, Hadian, Mirrokni; ICLR 2026). Our implementation uses a Gaussian Lloyd-Max codebook as an approximation of the paper's distortion-rate framework. Norm correction stores original_norm / reconstruction_norm per group to fix magnitude shrinkage at 3-bit.

The weight scheme matches the scalar case of HIGGS (Cornell + Yandex, NeurIPS 2024); the reference implementation is in HuggingFace Transformers. This package's role is the production vLLM integration plus the MoE infrastructure (pre-fused checkpoints, sparse expert dequant, mapper-aware loading).

Citation

bibtex

@inproceedings{zandieh2026turboquant,
  title={TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate},
  author={Zandieh, Amir and Daliri, Majid and Hadian, Majid and Mirrokni, Vahab},
  booktitle={International Conference on Learning Representations},
  year={2026}
}

Compressed by Varjosoft Oy using turboquant-plus-vllm v0.13.5.