⚠️ NEEDS POST TRAINING. A research preview, published as the record of a
compression method and its measurements rather than as a finished assistant. Further
healing and instruction tuning runs are planned over time, and these checkpoints will
improve as those land.
Qwen3.8-Whittle-16B
🔬 RESEARCH PREVIEW. This model is a compression-research artifact, evaluated
with field measurements rather than academic benchmarks. Expect rough edges, use the
serving settings below, and do not rely on it for factual reference or production systems.
🛑 NON-STARTER FOR AGENT AND LONG MULTI-TURN USE: repetition looping.
In extended generations, multi-turn conversations, and agent loops (coding
assistants, tool use) the model can fall into repetition loops badly enough to be
unusable, even with DRY sampling enabled. Single-turn use with the recommended
serving flags works well. Further training rounds targeting looping, multi-turn,
and agent formats are actively in progress; this card will be updated as fixes
land and pass real-world testing.
✅ v2 WEIGHTS: use the -v2- GGUFs. v2 fixed code fencing (8/8 on the fencing
suite; v1 emitted code outside markdown blocks) and reduced single-turn long-form
loops to 1/6 with mitigations disabled. Regression held: battery 35/39, recognition
5/7. v2 adds three.js fluency (trained on the official MIT examples + manual) at
the cost of one A100 hour on a 2.2M-token clean mix; recipe and mix in training/.
Serve with: --dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 4 --repeat-penalty 1.15 --repeat-last-n 512. The v1 files remain for reproducibility.
A 27B whittled down to 16.8B with a logit lens and a pricing table, then healed
with one A100 evening. It now outscores every intermediate cut, including ones 4B
larger, at 20 tokens/second on two consumer 8GB GPUs.
Whittle is Qwen3.8-27B minus 20 of its 64 layers (whole interval-blocks, chosen by
measured boundary cost) and minus 25% of every remaining MLP's width (weakest
neurons by ‖down_column‖ × activation-std). That is a −10.1B parameter cut performed
with zero training, followed by a single 11M-token QLoRA heal on a fully
clean-lineage mix (synthetic fact frames, programmatic short-think arithmetic,
hand-written code drills, public-domain long pages).
params
file
battery (39 greedy tasks)
speed (RTX 4060 + 3050)
Qwen3.8-27B (base)
26.9B
n/a
not measurable on ref. hardware
n/a
48-layer cut
20.8B
12.9GB
33/39
5 t/s
un-repaired Whittle
16.8B
10.1GB
25/39
20.5 t/s
Whittle (healed, this)
16.8B
10.1GB
36/39
18.5–20.9 t/s
The heal fixed 11 battery items and broke zero. Long-tail recall recovered
(recognition probe 2/7 → 5/7), arithmetic precedence and code completion returned to
textbook form, and the boiling point of water, the fragile fact that failed every
intermediate variant, answers correctly. Full measurement history, every pricing
run, and all scripts: see the companion research repo
Qwen3.8-p44w75-16.8B-unrepaired
and its research/ folder. Both heal adapters live in this repo under adapters/
(v1-heal applies to the un-repaired base; v2-fencing-threejs applies on top of v1).
Running it
The GGUFs run on stock llama.cpp (any build with Qwen3.5-series support).
Recommended serving command. The anti-loop sampling flags are part of the recipe,
not optional garnish:
The Q4_K_M fits entirely in 16GB of VRAM (or 2×8GB split). This is a thinking model:
it reasons in <think> before answering, so give it generous max_tokens (500+),
and keep --jinja (included above) if your client uses tool calling. Context can be
raised well past 8192 cheaply: only 11 of the 44 layers are full attention, the
rest carry fixed-size recurrent state.
Known limitations
Open-ended creative prompts (e.g. "write a haiku") can exhaust the thinking budget
in deliberation. Concrete instructions ("exactly 3 short lines, no preamble") work.
This is the one measured failure the heal did not close; it is targeted in the
next round.
Evaluated with a 39-prompt greedy battery plus probes, not academic benchmarks.
The numbers above are honest field measurements, comparable within this table.
The heal mix is small and targeted. Knowledge breadth beyond what a 16.8B carries
is not magically restored: this is an efficient model, not a 27B in disguise.
How it was made (short version)
Sound the model: stream per-layer FP8 shards through an 8GB GPU, record each
layer's identity cosine and a logit-lens sounding at every boundary.
Price the cuts: block-drops must keep the GDN:attention interval (GGUF
expressibility); price every candidate at the boundary, then at task level.
Findings: drop damage is non-additive, width damage compounds, and layers 32–35
hold arithmetic, not knowledge.
Cut: 20 layers + 25% of MLP width, zero training, single-width so stock
tooling serves it.
Heal: QLoRA r=64 on every linear (GDN projections included), 110 steps,
cosine annealed to completion, clean-lineage data only.
Full methodology in the research log (research/RESEARCH_activation_space_layer_merging.md).
Honest state of the model, and what it would take to finish it
Web development capability took real damage in the compression: current measurements
show the shape cut collapsed targeted web-token retrieval roughly 28-fold at the
boundary, and we are still isolating how much came from the depth cut versus the
width prune (restoration pricing runs are in progress). The heals so far were single
A100 hours and evenings; they recovered facts, arithmetic, and code fencing, but
turning this research preview into an actually usable daily model (agent-capable,
loop-free in long multi-turn use, restored web-dev depth) needs sustained
post-training that is beyond a grocery-money compute budget. The methods and data
are ready; the A100 hours are the missing ingredient.
Support this work
Independent research on consumer hardware. Every donation becomes A100 hours, and
every A100 hour ends up as a public model or a public measurement. If you want the
usable version of this model to exist, this is the lever.
☕ ko-fi.com/davida81328
Acknowledgements
Base model by the Qwen team (Apache 2.0). Whittled and healed by David Aylward with
Claude (Fable 5, Anthropic) as co-author. The instruments, pricing runs, builds,
training and evaluations were executed by Claude under David's direction, including
several load-bearing ideas of David's: the parallel-composition merge operator, the
slice-stack-merge width reduction, and the recognition-vs-recall damage probe.
