⚠️ NEEDS POST TRAINING. A research preview, published as the record of a

compression method and its measurements rather than as a finished assistant. Further healing and instruction tuning runs are planned over time, and these checkpoints will improve as those land.

Qwen3.8-p44w75 — a 16.8B cut of Qwen3.8-27B (zero training, ⚠️ UN-REPAIRED)

🪚 The healed release is here: Qwen3.8-Whittle-16B — same 16.8B, one QLoRA evening later: 36/39 on the battery (this un-repaired base scores 25/39), water boils at 100°C again, fibonacci is textbook. This repo remains the research artifact: the raw cut, the full measurement history, and every script.

What this is

1. Depth: 20 layers deleted in whole interval-blocks of 4 (layers 4–11, 24–31, 32–35) — keeping the stack exactly 3:1 so it stays expressible in GGUF with zero inference-code changes. Block choice was priced empirically with a streaming logit-lens over per-layer FP8 shards on an 8GB GPU.
2. Width: every remaining MLP sliced from 17408 → 13056 neurons (uniform 25% prune, all 44 layers including layer 0 and the tail), keeping the strongest 75% by ‖down_column‖ × activation_std measured over real corpus tokens.

Methodology (compressed — full research log in this repo's research/)

• Instrument first: a streaming prober loads one FP8 layer shard at a time onto an 8GB GPU, records each layer's identity cosine and a logit-lens sounding at every boundary. Note this model's head is untied — lens through lm_head, not the embedding.
• Operators measured on identical layer groups: weight-space merging (LaCo-style) collapses on this GDN hybrid (stream cos 0.27); activation-space parallel composition holds 0.93; plain drop 0.75. Weight-space neuron twins are absent (max cosine 0.58); functional twins are pairwise-abundant. Redundancy in this trained hybrid is functional and pairwise, not literal or blocky.
• Depth ladder: block-drop damage is non-additive — deleting both cheap 8-layer bands costs no more than either alone. 48L and 44L variants both survive at task level; 40L breaks structurally. The tested wall for zero-training depth removal sits between −7.7B and −9.2B.
• Width ladder: 25% width prune rides nearly free end-to-end; 50% breaks. Not because per-layer cost is high (halving one MLP costs 0.5–1.5% of the stream) but because ~35 boundaries compound.
• Expert-sliceability (negative result): activation mass is uniformly diffuse across any balanced clustering at every depth tested — top-2-of-8 oracle routing captures 27% of mass where chance is 25%. Zero-training MoEfication has no raw material here; sparsity would have to be trained in.
• Evaluation at every rung: greedy task batteries (facts / code / arithmetic / commonsense) + chat probes, served through llama.cpp — not just stream cosine. Stream metrics predicted where damage lands but only task testing revealed which capability pays (e.g. layers 32–35 turned out to hold arithmetic/code precision, not knowledge).

Evaluation (39-prompt greedy battery + chat)

Limitations (read before using)

• Corrupted long-tail facts. Width pruning removes rarely-firing neurons, which is where long-tail knowledge lives. Recognition probes show ~⅔ of the lost facts are rewired to plausible-wrong neighbors, not merely attenuated: this model will confidently tell you Buzz Aldrin was first on the moon, water boils at 90°C, and the solar system has nine planets. Do not use for factual reference.
• Think-budget rumination on open-ended/creative prompts (asked for a haiku, it may deliberate about haiku rules until the token limit). Concrete instructions work better.
• Evaluated with a 39-prompt battery and short greedy completions — no MMLU/GSM8K yet; treat all numbers as field measurements, not benchmarks.
• Single model, single run of everything; the intact 27B was never run on the same battery (it does not fit the reference hardware), so the true ceiling is unknown.

Running it

Reproduction

Support this work

Acknowledgements