GGUF quantizations of Qwen/Qwen3.8-27B
(multimodal, hybrid SSM + attention, 64 layers + 1 MTP layer), converted with
upstream llama.cpp. All files include the built-in MTP block
(nextn_predict_layers=1), usable for speculative decoding.
What the XYZ series is (and isn't)
XYZ is 100% focused on coding. These quantizations are built and tuned
for programming work: the per-tensor recipe keeps the parts that matter for
code generation (attention K/V, SSM state, and the MTP head) in higher
precision, and the build is evaluated against a code test suite so the
models stay reliable on code tasks — even at low bits.
That focus is deliberate. General-knowledge domains are not the goal.
There are already plenty of generalist models out there — this one is the
coding one. If you need the best answer on biology, history, politics, or
other non-code topics, a general-purpose build of the base model will usually
serve you better. XYZ optimizes for coding behavior first, and that's the
trade you're opting into here.
TL;DR — reach for XYZ when the job is writing, reading, or debugging
code. For everything else it's still a capable general model, but don't
expect it to out-perform an untuned build on pure knowledge tasks.
Recommended sampling: --temp 0.6 for balanced, coherent output.
What I like to use, in a 24gb gpu
bash
1llama-server -m Qwen3.8-27B-Q5-XYZ.gguf \2--host 127.0.0.1 --port 8080 --ctx-size 128000 -b 2048 --parallel 1\3-ngl 999 --threads 8 -ub 512 -ctk q8_0 -ctv q8_0 -fa on -kvu \4--no-mmap --temp 0.6 --spec-type draft-mtp,ngram-mod \5--spec-draft-type-k q4_0 --spec-draft-type-v q4_0 --spec-draft-n-max 6\6--spec-draft-p-min 0.75 --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48\7--spec-ngram-mod-n-max 64 --reasoning-budget 2048 --reasoning-preserve \8-mm G:\LLM\Models\mmproj-Qwen3.8-27B-bf16.gguf
91011using --reasoning-budget 2048 is like set to medium the thinking, use 512for low thinking and 8196for high, and use nothing to xHigh.
What I use for 16GB, using mtp is an option but you will be limited to 64k. You can also use q4_0 and double context but there will be degradation, 128k is a sweet spot for most cases.
bash
12llama-server -m Qwen3.8-27B-Q3-XYZ-v2.gguf --host 0.0.0.0 --port 80803--ctx-size 128000 -b 2048 -ngl 999 --threads 4 -ub 5124-ctk q8_0 -ctv q8_0 -fa on -kvu --temp 0.65--spec-type ngram-mod --spec-ngram-mod-n-match 246--spec-ngram-mod-n-min 487--spec-ngram-mod-n-max 648--reasoning-preserve
91011using --reasoning-budget 2048 is like set to medium the thinking, use 512for low thinking and 8196for high, and use nothing to xHigh.
Extreme quantizations (Q1/Q2) — sampling to reduce hallucinations
The low-bit files (Q1Q/Q2) are more prone to hallucination — use this tuned
sampling with llama-server for much more stable output:
The DRY sampler (repetition-aware) is the key flag for low-bit models —
--dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 2 keeps generation
on-track without over-penalizing. Works for Q1Q-XYZ-v2 too.
Notes
Model is Apache-2.0, architecture Qwen3_5ForConditionalGeneration (hybrid
SSM + attention, full attention every 4th layer), vocab 248,320,
tie_word_embeddings=false.
Quantized with upstream llama.cpp, 4 threads, --tensor-type per-tensor
recipe with imatrix. Key tensors (attention K/V, SSM params, MTP head) are
kept in BF16 — full precision where it matters.
The Q1/Q2 files are experimental: below ~2.5 BPW a reasoning model can get
stuck in the thinking loop on some prompts.
Sizes are exact file sizes on disk (GiB, base 1024 — matches the sizes shown
on the HF file browser).
Files
Quant ladder (v2 builds) + the new Q2.5-v3. Sizes are exact on-disk GiB (base 1024).
