Zero-shot expert carve of Qwen3.6-35B-A3B — no training, no calibration.
256 → 160 experts per layer (37.5% dropped), carved directly from the NVFP4
checkpoint by importance, specifically so the model fits and runs fast on a
single 24GB GPU (RTX 5090 laptop) where the full 35B cannot allocate KV.
The Kirigami family
Zero-shot expert carves of Qwen3.6-35B-A3B, sized for single consumer GPUs
(kirigami: the Japanese art of cutting paper to create structure — here,
cutting experts to fit the card). No training, no calibration, one CPU pass.
One-shot, shard-streamed CPU pass over unsloth/Qwen3.6-35B-A3B-NVFP4 —
top-K expert selection (keep/drop, no merging), described fully below so the
method is reimplementable:
Importance (checkpoint-resident, zero forward passes):
I_e = ||router_row_e||₂ × mean(weight_scale_e) — router propensity ×
NVFP4 scale energy, per layer.
Kept experts renumbered contiguously; router rows sliced in the same order
(routing semantics preserved). Shared expert and all attention untouched.
carve_manifest.json (shipped in this repo) records the kept-expert indices per layer.
26 GB → 17.3 GiB (160 experts/layer); boots at 16K context on 24GB. Carve time: ~40 min CPU.
Measured on RTX 5090 laptop (24GB), vLLM v0.24, 4.4K-token prompt
Mode
TTFT
Decode
4-way aggregate
compiled (CUDA graphs)
288 ms
163 tok/s
173 tok/s
This rung trades 12.5% more experts than the 28B for a 16K-context boot.
Sustained throughput under real concurrent load (12 in-flight requests,
long multi-thousand-token prompts — the most representative numbers):
Metric
Value
Peak prefill
4,764 tok/s
Peak generation (aggregate)
749 tok/s
Sustained generation (aggregate)
545 tok/s
Peak KV-cache utilization
—
Concurrent requests
12
Decode speed is nearly flat in both prompt depth and generation length
(hybrid linear-attention backbone); prefill ingests ~11K tok/s.
Includes a carved MTP draft head (fused tensors sliced consistently)
for speculative decoding.
MTP speculative decoding (included)
The parent's MTP draft head is carved consistently (fused draft-expert
tensors + draft router sliced with their own importance set) and works with
vLLM's qwen3_5_mtp speculative method. Measured on the 20B rung
(temp 0, log-analysis prompts — acceptance is workload-dependent):
Notes: NVFP4 MoE requires the cutlass/flashinfer backends (do NOT set
moe_backend=triton). Thinking-mode preamble is on by default — disable via
chat_template_kwargs: {"enable_thinking": false} if you want direct answers.
Honest caveats
Lossy by construction: 37.5% of experts are gone. Coherence smokes
(log RCA, constraint following, factual explanations) show no visible
degradation; no benchmark suite has been run. Certify before production.
Importance is uncalibrated (weight/scale statistics only). A
calibration-based selection (router activation statistics) would likely
select better; this artifact demonstrates the zero-cost floor.
Vision tower weights are carried over but untested post-carve.
