You can deploy this model and start earning money today!

logic65/
Qwen3.8-Whittle-MoE-27B-A17.8B-GGUF

Name: Qwen3.8-Whittle-MoE-27B-A17.8B-GGUF
Availability: InStock

logic65/Qwen3.8-Whittle-MoE-27B-A17.8B

Views

No views yet

v2.2.1 — the reasoning abort is fixed in every tier (29 Aug 2026)

All tiers here are rebuilt from v2.2.1. Served with reasoning enabled, v2.2 could end its turn inside its own chain of thought instead of answering: its stop gate had been trained on sequences where the chat template opened a <think> block that the training data never closed, so the most heavily weighted signal in the set landed on a terminator inside the thinking block.

v2.2.1 replaces the 64 shared_expert_gate tensors — 0.66 MB, nothing else changed — with a gate trained on real chain-of-thought traces from the parent, where the block is closed by the teacher's own words. On a 24-prompt structural probe with reasoning on, turns that ended without ever emitting </think> fell from 21/24 to 3/24. Zeroing the gate entirely scores 11/24, so the new gate is doing real work rather than merely undoing the damage.

What this does not fix. Measured like-for-like against v2.2 with reasoning off on 36 held-out counted prompts: turns that stop cleanly 32/36 → 34/36, distinct-item exact counts 20/36 → 24/36, mean overrun 1.69× → 1.14×. Mean repeated-4-gram is unchanged at 0.141, still above our 0.05 target. Long lists of 45+ items remain weak (4/12 → 5/12). Take v2.2.1 for the reasoning abort, not for the counting.

Tiers

file size notes
Q8_0 28.7 GB reference quality
Q6_K 23.1 GB
DQ5_K_XL 21.7 GB DQ recipe: q8_0 embeddings/head, q6_k attention and shared expert
DQ4_K_XL 20.1 GB DQ recipe
Q4_K_M 17.4 GB stock allocation
DQ3_K_XL 14.7 GB small, more attention bits than Q3_K_M
Q3_K_M 13.9 GB new — small, stock allocation, no DQ recipe

Q8_0, Q6_K, Q4_K_M, DQ5_K_XL and DQ4_K_XL reproduce the v2.2 tensor allocation exactly, tensor for tensor. DQ3_K_XL does not: on this llama.cpp the 64 expert down-projections fall back to q5_1 where the v2.2 build emitted q5_0, which is chosen by the quantiser for 192-wide rows and is not settable by a flag. It is a q3_k_l build rather than a byte-match of the old file. Q3_K_M is new, added so there is a small tier without the DQ recipe.

The shared_expert_gate is stored at f32 in every tier, so the fix is carried at full precision regardless of the quantisation level.

Whittle MoE 27B (A18B) v2.2 GGUF

Quantized builds of Whittle MoE 27B (A18B), a post hoc mixture of experts carved from Qwen3.8-27B and taught when to stop talking. v2.1 passed every release bar: 8 percent single turn loop rate (down from 69 at first release), 22 percent structured (down from 75), zero truncated answers, knowledge battery 28 of 39. Method, measurements, and the failures along the way are on the main model card.

v2.2 (28 Aug 2026)

All tiers here are v2.2: the 64 per-layer shared_expert_gate tensors now carry a trained stop signal (0.33M params changed, nothing else). Measured against v2.1 on held-out prompts: list termination 18/36 -> 27/36, worst-window list repetition 0.237 -> 0.048, conversation turns terminating 27/48 -> 41/48. One honest regression: exact item counts dropped (8/36 -> 2/36) - it ends lists reliably but not at the requested count; pin the v2.1 files (git history of this repo) if exact-N lists are your use case. Full numbers and the root-cause story are on the model card.

The gate tensor is 1-D, so llama-quantize keeps it at F32 in every tier automatically - the stop signal survives even DQ3. The BF16 tier is retired, and the v2.1 tiers were removed from the tree to keep one clear current version - pin revision d4b45cea of this repo if you need a v2.1 file.

Which file

file	size	fits	notes
`Q8_0`	26.7 GB	32 GB, or 24 GB with expert offload	best quality in practice
`Q6_K`	22.1 GB	24 GB	flat quant
`DQ5_K_XL`	20.2 GB	24 GB	dynamic, close to Q6 quality
`DQ4_K_XL`	18.8 GB	20 to 24 GB	dynamic, the size and quality sweet spot
`Q4_K_M`	16.2 GB	16 to 20 GB	flat quant, smallest standard build
`DQ3_K_XL`	13.9 GB	15 to 16 GB, no offload needed	dynamic, smallest usable tier
`mmproj-qwen38.gguf`	0.9 GB	adds about 1 GB	optional vision projector, see below

What the DQ tiers are

A dynamic mix rather than a flat quant. The routed experts carry the tier's nominal bit width, while the parts that suffer most under compression keep more bits:

The router stays at F16 in every DQ tier. It is tiny, around 20M parameters, but it decides which 16 of 64 expert slivers fire for each token. This model is its routing: it was brought back from gibberish by training the routers with every expert frozen, so they are the last thing to compromise.
Shared experts and attention sit one or two tiers above the base type.
Embeddings and the output head are protected in the upper tiers. In DQ3 they are deliberately dropped, because with a 248k vocabulary they alone were adding about 2.7 GB and pushing a supposed Q3 build above Q4_K_M.

One quirk worth knowing if you quantize this architecture yourself: the routed expert down projections have 192 wide rows, and k-quants need rows divisible by 256, so llama.cpp silently falls back to legacy q5_1 for them at 6 bits per weight no matter which tier you ask for. That is why a naive Q3 build of this model lands heavier than Q4_K_M. The DQ3 build accounts for it.

Naming note, because it matters: DQ is our own recipe, a fixed protection rule applied at quantize time. It is not Unsloth's UD, which uses closed source imatrix calibration to choose per layer bit widths from data. Same underlying idea, less sophisticated, and named differently on purpose.

Running it

llama-server -m Whittle-MoE-27B-A18B-v2.2-DQ4_K_XL.gguf -ngl 99 -c 8192 -fa on --jinja

Any recent llama.cpp build with Qwen3.5 MoE support works. No fork, no patches.

Short on VRAM? Keep the routed experts in system RAM instead of shrinking the model:

llama-server -m <file>.gguf -ngl 99 --n-cpu-moe 32 -c 8192 -fa on --jinja

--n-cpu-moe N moves the experts of N layers (of 64) out to RAM while attention, the GDN layers, the routers and the shared experts stay on the GPU. Raise N until it fits: 64 puts every routed expert in RAM and needs only about 5 GB of VRAM. Each token touches just 16 of 64 slivers, so it costs speed, not sanity. Older builds without that flag can use -ot '\.ffn_(gate|up|down)_exps\.=CPU'.

Also on Ollama:

ollama run Whittle/Qwen3.8-Whittle-MoE-27B-A17.8B

Vision (experimental)

The parent is a vision language model and this MoE kept the whole multimodal interface intact through the carve. mmproj-qwen38.gguf is the parent's vision tower converted for llama.cpp:

llama-server -m <file>.gguf --mmproj mmproj-qwen38.gguf -ngl 99 -c 8192 -fa on --jinja

Then attach images in the built in web UI or through the API. Honest status: the routers were healed on text only and vision has not been through any measured gate. First eyes on tests show detailed, structured image description including small texture details and composition judgments. Treat it as a research bonus until numbers land.

Check the numbers yourself

The harness that produced the loop rates on the model card is in this repo:

bash

curl -sLO https://huggingface.co/logic65/Qwen3.8-Whittle-MoE-27B-A17.8B-GGUF/resolve/main/loop_test.py
python3 loop_test.py http://localhost:8080
# ollama:  python3 loop_test.py http://localhost:11434 Whittle/Qwen3.8-Whittle-MoE-27B-A17.8B
# routers: pass the model name as the second argument or the server returns 400

Same prompts, thresholds, seeds and samplers used to gate every release. It runs about 90 generations, prints single turn, structured, multi turn and late turn failure rates, and writes every generation's scores to loop_test_results.json so you can inspect individual failures instead of trusting a summary. Standard library only.

Support this work

Every donation goes directly to GPU hours, and every GPU hour gets reported, including the ones that failed: ko-fi.com/davida81328

file	size	notes
`Q8_0`	28.7 GB	reference quality
`Q6_K`	23.1 GB
`DQ5_K_XL`	21.7 GB	DQ recipe: q8_0 embeddings/head, q6_k attention and shared expert
`DQ4_K_XL`	20.1 GB	DQ recipe
`Q4_K_M`	17.4 GB	stock allocation
`DQ3_K_XL`	14.7 GB	small, more attention bits than Q3_K_M
`Q3_K_M`	13.9 GB	new — small, stock allocation, no DQ recipe