Qwen3.6-35B-A3B 128-Expert Pruned GGUF
Experimental research artifact. This is a 50% expert-pruned model, not an official Qwen release. The pruning calibration is small and coding/agent biased. Test the Q8 artifact on your workload before trusting a lower quant.
This repository contains Q8 through one-bit-class GGUF variants of
Qwen/Qwen3.6-35B-A3B. The routed MoE was reduced from
256 to 128 experts per layer while retaining the original
top-k 8 active experts per token. Pruning reduces stored parameters and memory, but it does not reduce the eight active expert matmuls per token.
Quant files
| Quant | Size GiB | SHA-256 | QA | Notes |
|---|
| Q8_0 | 18.392 | b85b3ab29aabd631… | 8/8 (pass) | Reference pruned artifact; retained weights preserve their original Q8 bytes. |
| Q6_K | 14.212 | 7fa673151e2f653f… | 6/8 (fail) | Near-Q8 PPL, but missed 2/8 deterministic acceptance checks. |
| Q5_K_M | 12.347 | 495cc9fe01f33bea… | 8/8 (pass) | Best measured local balance: 8/8 checks and only +1.05% PPL. |
| Q4_K_M | 10.592 | b8fec3c109c561b2… | 7/8 (pass) | Smaller viable option: 7/8 checks and +2.38% PPL. |
| Q3_K_M | 8.434 | 0d4dfa3c58100e69… | 6/8 (fail) | Aggressive; verify on your workload. |
| Q2_K | 6.582 | 602bab193dd82587… | 7/8 (pass) | Passed 7/8 smoke checks, but held-out PPL rose 26.50%. |
| IQ2_XS | 5.396 | 9b631cca3ae23cf7… | 6/8 (fail) | Extreme ~2-bit-class build; imatrix required and workload QA strongly recommended. |
| IQ1_M | 4.312 | 4064257359d35174… | 2/8 (fail) | Experimental one-bit-class endpoint (1.75 bpw nominal for quantized tensors). |
| IQ1_S | 3.952 | c5caaa158eb73055… | 1/8 (fail) | Smallest experimental endpoint (1.56 bpw nominal for quantized tensors). |
GGUF has no conventional Q7_K family, so the ladder follows the supported practical families: Q8, Q6, Q5, Q4, Q3, Q2/IQ2, then IQ1. IQ1_M and IQ1_S are both included because they are distinct one-bit-class formats.
Requantization warning: every file below Q8_0 was produced from the pruned Q8_0 artifact with --allow-requantize, not from F16/BF16. The quantizer itself warns that this can reduce quality more than direct quantization from 16/32-bit weights. A remapped 128-expert importance matrix was used for every lower quant.
What was pruned
This checkpoint is
not a direct REAP implementation. REAP motivated the experiment, but the Qwen selection used low router-use frequency because that criterion performed better than REAP in the preceding local Granite-MoE control experiment.
See the
REAP paper and
official Cerebras implementation for the router-weighted activation-norm method that inspired this work.
The selection procedure was:
- Run 512 coding/agent calibration tokens through all 40 MoE layers.
- Record 20,480 layer-token observations and each token's top-8 routed expert IDs.
- In each layer, prune the 128 experts with the lowest routed top-k frequency; break ties deterministically by original expert ID.
- Copy retained expert rows and corresponding router rows in ascending original-ID order. Update
expert_count from 256 to 128.
- Preserve the raw quantized bytes of retained expert tensors. Shared experts and all other tensors remain byte-identical to the source Q8.
- Re-open the output and fully hash-check all 733 tensors before publication.
The exact retained IDs, per-expert route counts, source trace hash, and transformation report are included under method/.
Evaluation
The first table compares the unpruned Q8 baseline, the 50%-pruned Q8, and the existing all-expert mixed-Q2 deployment reference on the same eight deterministic coding, JSON, math, tool-call, and context checks.
| Artifact | Size GiB | Passed | Exact outputs vs baseline | Median decode tok/s | Gate |
|---|
| full-q8 | 34.369 | 7/8 | 8/8 | 33.84 | PASS |
| pruned-50-q8 | 18.392 | 8/8 | 7/8 | 104.38 | PASS |
| full-mixed-q2 | 16.930 | 5/8 | 4/8 | 118.01 | FAIL |
Exact-output equality is reported separately from task correctness. Runtime numbers are specific to a local RTX 3060 Ti + RTX 2080 SUPER + RTX 3080 setup; the unpruned Q8 baseline used CPU expert offload, while the smaller models fit across all three GPUs.
The complete per-prompt outputs and timing records are in evaluation/. These checks are a smoke/acceptance suite, not a replacement for HumanEval, BFCL, MMLU, long-context, or safety evaluation.
All-quant functional and speed benchmark
| Artifact | Size GiB | Passed | Exact outputs vs baseline | Median decode tok/s | Gate |
|---|
| Q8_0 | 18.392 | 8/8 | 8/8 | 104.11 | PASS |
| Q6_K | 14.212 | 6/8 | 6/8 | 108.44 | FAIL |
| Q5_K_M | 12.347 | 8/8 | 8/8 | 118.28 | PASS |
| Q4_K_M | 10.592 | 7/8 | 7/8 | 125.22 | PASS |
| Q3_K_M | 8.434 | 6/8 | 5/8 | 112.63 | FAIL |
| Q2_K | 6.582 | 7/8 | 6/8 | 124.36 | PASS |
| IQ2_XS | 5.396 | 6/8 | 4/8 | 109.60 | FAIL |
| IQ1_M | 4.312 | 2/8 | 1/8 | 100.48 | FAIL |
| IQ1_S | 3.952 | 1/8 | 1/8 | 114.08 | FAIL |
All functional-speed rows except IQ1_M used the local ik_llama CUDA build. That build aborts on IQ1_M with Unhandled type iq1_m (29), so IQ1_M was bench-tested with the local TurboQuant llama.cpp build instead. Its speed is therefore not directly comparable to the other rows; the engine and full command are preserved in evaluation/quant-ladder-qa-results.json.
Held-out perplexity by quant
Perplexity used the included evaluation/qwen36_ppl_heldout.txt, which is separate from both the pruning router trace and the importance-matrix calibration. Values are most useful as relative degradation within this table, not as a cross-model leaderboard.
| Quant | Size GiB | Held-out PPL | Delta vs Q8 | Estimate error |
|---|
| Q8_0 | 18.392 | 9.7356 | +0.00% | 0.81899 |
| Q6_K | 14.212 | 9.7435 | +0.08% | 0.81747 |
| Q5_K_M | 12.347 | 9.8377 | +1.05% | 0.82701 |
| Q4_K_M | 10.592 | 9.9673 | +2.38% | 0.84195 |
| Q3_K_M | 8.434 | 10.9941 | +12.93% | 0.96516 |
| Q2_K | 6.582 | 12.3158 | +26.50% | 1.09987 |
| IQ2_XS | 5.396 | 14.3985 | +47.90% | 1.27363 |
| IQ1_M | 4.312 | 21.4644 | +120.47% | 1.99295 |
| IQ1_S | 3.952 | 29.1534 | +199.45% | 2.88727 |
Practical selection from this sweep
- Q8_0 is the reference for judging pruning independently of further quantization.
- Q5_K_M is the measured recommendation for this local coding/agent workload: 8/8 checks, +1.05% held-out PPL, and 118.28 median decode tok/s.
- Q4_K_M is the smaller viable choice: 7/8 checks, +2.38% PPL, and 125.22 tok/s.
- Q3 and below are research/space-first builds, not general recommendations; their perplexity degradation is material even when a tiny smoke suite passes.
Usage
1llama-server \
2 -m Qwen3.6-35B-A3B-128E-Pruned-Q4_K_M.gguf \
3 --jinja --n-gpu-layers 99 --ctx-size 32768
Use a recent llama.cpp-compatible runtime with qwen35moe support. The Q8 file is the best reference for deciding whether pruning itself is acceptable; move down the quant ladder only after that comparison.
Limitations
- The 512-token selection trace is small and strongly biased toward coding/agent work.
- Experts rarely used by that trace may be important for languages, domains, knowledge, reasoning styles, or safety behavior absent from calibration.
- There was no recovery fine-tuning after pruning.
- Frequency pruning measures router use, not expert activation magnitude or REAP saliency.
- Halving stored experts does not halve decode compute because top-k remains 8.
- Low-bit requants compound pruning risk with Q8-to-lower-bit requantization error.
- Do not use this experiment for high-stakes decisions without independent evaluation.
Reproducibility files
quant-manifest.json: source, imatrix, quantizer hashes, commands, sizes, and hashes.SHA256SUMS: complete model-file checksums.method/expert-selection.json: retained/pruned IDs and route counts for every layer.method/pruning-report.json: Q8 rewrite and verification report.method/imatrix-remap-report.json: 256-to-128 imatrix transformation report.evaluation/: deterministic QA outputs and timing evidence.
License and attribution
The GGUF metadata identifies the base license as Apache-2.0. This derivative keeps that license metadata. The source Q8 metadata credits
Unsloth as its quantizer. Review the upstream model card and license before redistribution or use.
