Views
No views yet
unsloth/Qwen3.5-9B-GGUF
(Q2_K_XL Unsloth Dynamic quant, 3.93 GB source) into per-stage GGUFs ready for
distributed pipeline inference (DPI). Each stage GGUF is a self-contained
GGUF that vanilla llama.cpp can load standalone — the splitter keeps
token_embd.weight on every stage so middle stages pass libllama's
load-time supported-tensor check.| Path | Stages | Layers per stage | Use case |
|---|---|---|---|
stages-2/ | 2 | 16 | DPI on 2 workers (~2.5 GB VRAM per stage) |
stages-4/ | 4 | 8 | DPI on 4 workers (~1.5 GB VRAM per stage) |
qwen35 arch). Auto-split evenly;
the last stage carries output_norm + lm_head so it's slightly larger
than the rest.stages-2/stage0 layers [0, 16) → 1805 MB, 214 tensors
stage1 layers [16, 32) → 2682 MB, 215 tensors (carries lm_head)stages-4/stage0 layers [0, 8) → 1181 MB, 108 tensors
stage1 layers [8, 16) → 1179 MB, 108 tensors (middle, vanilla-libllama loadable)
stage2 layers [16, 24) → 1214 MB, 108 tensors (middle, vanilla-libllama loadable)
stage3 layers [24, 32) → 2024 MB, 109 tensors (carries lm_head)1gguf_stage_split \
2 --input Qwen3.5-9B-UD-Q2_K_XL.gguf \
3 --output-dir stages-N \
4 --auto-split N \
5 --prefix qwen3.5-9b-q2kxlcc-headless-v2/src/bin/gguf_stage_split.rs). Each stage
GGUF has its layer indices renumbered starting at 0 (blk.0, blk.1,
…) so that llama.cpp's existing GGUF model loader works without
modification on each stage.token_embd.weight purely so the loader's
supported-tensor check passes; the embedding is not used at inference
time on those stages.unsloth/Qwen3.5-9B-GGUF model.