Qwen3.5-35B-A3B — APEX GGUF (torch imatrix)
MoE-aware, mixed-precision
APEX quantization of
Qwen/Qwen3.5-35B-A3B — a 35B-total /
~3B-active MoE (40 layers, 256 routed + 1 shared expert, hybrid GatedDeltaNet
linear-attention + periodic full attention, with a NextN/MTP self-speculative head).
The importance matrix here is generated with a PyTorch band-serialized generator
rather than llama-imatrix, for two concrete reasons on this architecture:
- llama.cpp's imatrix tool is impractical for
qwen35moe. The GatedDeltaNet
linear-attention is a serial state-space recurrence; the imatrix collection
callback breaks the GPU path that makes normal inference fast, so it falls back to
a single CPU thread that no thread count can parallelize.
- The torch generator covers the MTP/NextN head (
blk.40.*) that llama-imatrix
does not — it is memory-bounded (only band layers resident) so it runs on modest
hardware regardless of model size.
update:
-v2 files: identical bit allocation to the originals, except the 48 attn_gate coefficients are kept at F32 instead of the tier's attention band type. They parameterise a nonlinearity that multiplies the state path, so quantising them lets error compound along the sequence; F32 costs about 31 MB. Prefer -v2.
The -torch imatrix — what it is
Qwen3.5-35B-A3B-torch.imatrix is a standard GGUF-format importance matrix
(in_sum2 + counts per tensor), bit-compatible with llama-quantize --imatrix.
It was generated from the HF safetensors with a band-serialized PyTorch forward,
calibrated on a general text corpus, covering all 40 transformer layers plus the
NextN/MTP head.
Validation vs a reference llama.cpp imatrix
Compared per-tensor against
bartowski's canonical llama.cpp imatrix
(
Qwen_Qwen3.5-35B-A3B-imatrix.gguf), computed independently on different calibration data:
| Metric | Value |
|---|
| Tensors covered (torch) | 523 (incl. 13 MTP-head tensors) |
| Tensors covered (bartowski) | 510 (no MTP head) |
| Per-tensor correlation (median) | 0.966 |
| Per-tensor correlation (mean) | 0.877 |
| Lowest-correlation tensors | blk.*.ssm_out.weight only |
Median correlation 0.966 against an independently-computed imatrix (different
calibration data) validates the mapping and values. The only weakly-correlated tensors
are the ssm_out projections (post-nonlinearity inputs) — a known property that does
not affect quantization quality (the quantizer needs only relative per-channel
importance). The torch imatrix is a strict superset (only-real = []).
PPL parity (measured)
The truer test: does the torch imatrix produce a quant as good as the llama.cpp one?
Same i-compact recipe quantized with each imatrix, perplexity over 200×512-token
wikitext-2 windows:
| i-compact quantized with | PPL | Δ vs bf16 |
|---|
| bf16 (reference) | 6.620 | — |
| bartowski llama.cpp imatrix | 6.756 | +2.05% |
| this torch imatrix | 6.775 | +2.34% |
The torch and llama.cpp quants differ by 0.02 PPL — inside the ±0.073 error bars,
i.e. statistically indistinguishable. The torch-generated imatrix produces a quant as
good as the canonical llama.cpp one.
Repeated on a diverse code-heavy corpus (HumanEval + MBPP + GSM8K + prose, 342
windows) — more representative of a coding model than Wikipedia prose:
| i-compact quantized with | PPL (code-heavy) | Δ vs bf16 |
|---|
| bf16 (reference) | 2.247 | — |
| bartowski llama.cpp imatrix | 2.310 | +2.77% |
| this torch imatrix | 2.318 | +3.15% |
Parity holds on code/math too (Δ0.008, inside ±0.014) — the imatrix records per-channel
activation magnitudes, so the method is domain-agnostic. (Absolute PPL is far lower
here only because code is more predictable than prose; not comparable across corpora.)
Calibration used the diverse calibration_datav3 (prose + code + multilingual), so
these quants are not domain-handicapped. The imatrix's calibration corpus is a
knob: a code-weighted calibration would favor coding channels further, at a small
cost elsewhere — useful if you're specializing for a single domain.
Files
Qwen3.5-35B-A3B-torch.imatrix — the PyTorch-generated importance matrix (this repo).Qwen3.5-35B-A3B-APEX-i-quality-torch.gguf (23.5 GB) — largest/highest-fidelity tier,
quantized with the torch imatrix above.Qwen3.5-35B-A3B-APEX-i-compact-torch.gguf (17.4 GB) — smaller, more aggressive tier.*-v2.gguf — same tiers with the state/gate coefficients (ssm_alpha, ssm_beta) stored
at F32. All other tensors are unchanged.- llama.cpp reference imatrix — not re-hosted; see
bartowski/Qwen_Qwen3.5-35B-A3B-GGUF.
Attribution
- Base model: Qwen — Qwen/Qwen3.5-35B-A3B.
- Reference imatrix for validation: bartowski —
Qwen_Qwen3.5-35B-A3B-GGUF.
- APEX recipe & toolkit: LocalAI —
localai-org/apex-quant.
- Quantization engine: llama.cpp (ggml-org).
Unofficial community quantization; not affiliated with or endorsed by Qwen.
