Every number below comes from live inference runs (same harness, same
prompts, same flags, temperature=0, averaged over multiple runs). Three
models were tested on the identical benchmark so the deltas are apples
to apples:
label
model
normal
Qwen3.8-27B (MTP) — base, no abliteration, no cap
abliterated
Qwen3.8-27B-abliterated (MTP) — refusal removed
ablit+cap
this model (...-thinkingcap-abliterated)
Two suites were run:
Easy — 52 single-shot items (multiple-choice, short math, puzzles),
averaged over 5 runs (260 scored items per model).
Hard — 10 multi-step algebra problems (same distribution as the
ThinkingCap SFT data), averaged over 3 runs (30 items per model).
This is where the models are pushed to think long enough to "run away."
Easy suite (5-run average, 260 items each)
metric
normal
abliterated
ablit+cap
Multiple choice
94.4% (85/90)
93.3% (84/90)
95.6% (86/90)
Math
93.8% (75/80)
93.8% (75/80)
93.8% (75/80)
Think
94.4% (85/90)
94.4% (85/90)
94.4% (85/90)
Overall
94.2% (245/260)
93.8% (244/260)
94.6% (246/260)
think chars (mean ± std)
271 ± 8
324 ± 15
329 ± 31
think tokens (mean ± std)
87 ± 3
101 ± 4
101 ± 8
refusal rate
>80%
0%
0%
Reading it: on short, easy prompts the thinking length is already small
(~87–101 tokens) and all three land within each other's noise band.
Accuracy is flat (≈94%). The cap is essentially free here — it costs
nothing, which is exactly what you want from an efficiency layer.
Hard suite (3-run average, 30 items each)
metric
normal
abliterated
ablit+cap
accuracy
70.0% (21/30)
70.0% (21/30)
70.0% (21/30)
think chars (mean ± std)
636 ± 61
235 ± 6
358 ± 20
think chars (median)
170
170
159
think tokens (mean ± std)
235 ± 20
99 ± 2
141 ± 8
Reading it: on hard multi-step problems the base model starts to run
away (mean 636 chars, pulled up by one 4,868-char outlier). Abliteration is
the biggest reducer (mean 636 → 235, −63%) and it holds accuracy. The
cap sits between the two (358) — see the per-item breakdown below for why.
Per-item thinking length — Hard suite (chars, 3-run avg)
problem
normal
abliterated
ablit+cap
worker earns $750 (runaway)
4868
852
1843
a number × 7, − 3 = 25
214
218
111
cyclist 42 km / 6 h → 12 h
199
192
113
two inlet pipes fill a pool
193
193
197
train 60 km/h, caught by 80 km/h
170
172
166
Sylvia 3× Amy 2 yrs ago
169
168
349
rectangle perimeter 36, area?
157
163
392
sum of two numbers is 198
137
138
152
ratio boys:girls 3:5
130
131
139
x+y=10, x−y=4 → x·y
121
121
119
The pattern is the whole story:
On the runaway item the cap does its job — it cuts the 4,868-char
base spiral down to 1,843 (−62%), and abliteration cuts it further to 852.
This is the real-world win: the cap stops the model from over-thinking.
On short items (120–220 chars) all three are within a few percent —
there is simply little to cut.
On a couple of items (Sylvia, rectangle) the cap runs longer than
abliteration. Abliteration removed the model's shortcuts, so it reasons
more; the cap trims the runaway but leaves that extra reasoning in.
Verdict
ThinkingCap is runaway insurance, not a universal shrinker.
It does not make every answer shorter — on short prompts it is a wash.
What it reliably does is stop the model from spiraling on hard,
multi-step problems (−62% on the runaway item) while keeping accuracy
identical (70% on hard, ≈94% on easy, refusal still 0%). Abliteration is
the bigger reducer; the cap is the guardrail on top. Stacked together
(abliterated + cap = this model) you get the shortest typical thinking
and the smallest worst-case thinking, at no accuracy cost.
How to reproduce
bash
1# bench.py is a self-contained stdlib-only harness (no pip install).2# Run the same command against each model on its own llama.cpp endpoint,3# then compare the JSON summaries. Easy suite:4python3 bench.py --api http://<ip>:8081 --suite all \5 --output base_normal --workers 4 --temp 0 --timeout 1806# ... x5 per model7# Hard (algebra) suite:8python3 bench.py --api http://<ip>:8081 --suite thinkhard \9 --output nh_cap --workers 4 --temp 0 --timeout 24010# ... x3 per model11# A/B two outputs:12python3 compare.py base_normal.json nh_cap.json
Flags kept identical across all runs: --workers 4 --temp 0,
--timeout 180 (easy) / 240 (hard). Thinking length is reported as
reasoning_content character count and completion_tokens from the API.
📊 Benchmarks
Tested with inspect_ai 0.3.260 on local llama.cpp, reasoning_effort=low,
temperature=1, top_k=20, top_p=0.95. All runs are single-epoch (no multi-epoch averaging).
Benchmark
Q6_K (this repo)
Q4_K_M (this repo)
Qwen3.8-27B¹
Qwen3.6-27B
Opus 4.6 Max
GPT-5.5
Kimi K3
Gemini 3.1 Pro
GPQA Diamond (198)
87.1%
79.5%
89.2%¹
87.8%¹
91.3%¹
93.6%
93.5%
94.3%⁸
IFEval (541, strict)²
88.2%
88.4%
~91.5%⁷
~90.4%⁷
95.0%⁷
94.2%
92.8%
93.5%⁸
SWE-bench Verified (50)⁴
69.4% (34/49)¹⁰
63.3% (31/49)⁹
~82%⁶
~77.2%⁶
~80.8%⁶
88.7%
76.8%
80.6%⁸
Benchmark comparison — this repo vs frontier models
¹ Official vendor-reported scores from the Qwen3.8-27B model card, reasoning_effort=xhigh.
² Standard Google IFEval via inspect_evals/ifeval (541 prompts, prompt-level strict accuracy).
⁴ inspect_evals/swe_bench_verified_mini: 50 issues (25 django + 25 sphinx), message_limit=100 steps,
max_tokens=16384/turn, tool_timeout=210s, no internet, Docker sandbox.
⁶ External/independent evaluations (not the official model card). SWE-bench Verified (500) is a different, harder split than our 50-task mini. Treat as directional, not exact.
⁷ External comparison via serenitiesai.com (F16 reference values). Treat as directional.
⁸ From SmartScope Gemini 3.1 Pro Benchmark Analysis (2026).
GPT-5.5 and Kimi K3 SWE scores from MorphLLM SWE-bench Pro Leaderboard.
⁹ 31/49 of-scored (1 out: sphinx-doc__sphinx-9229, step-limited at 117 msgs). Run status error (APITimeoutError at scheduler level) but all 50 samples completed in summaries.
¹⁰ 34/49 of-scored (1 out: sphinx-doc__sphinx-9229, timeout at 162 msgs / 190 min). Run status cancelled — last sample hit the 3600s timeout.
Reading it: Q6_K scores within 2.1 pp of the full BF16 Qwen3.8-27B on GPQA
(87.1 vs 89.2) — with a quantized, abliterated, LoRA-merged GGUF running on
consumer GPUs. On IFEval, Q6_K (88.2%) and Q4_K_M (88.4%) are near-identical and sit
within ~3 pp of the F16 reference (~91.5% for Qwen3.8-27B, 95.0% for Opus 4.6). On
SWE-bench, Q6_K leads at 69.4% (34/49) vs Q4_K_M 63.3% (31/49) — both timed out
on the same hard issue (sphinx-9229); external runs of the full 500-task SWE-bench
Verified put Qwen3.8-27B at ~82% (footnote ⁶).
⚠️ Hardware limitation: Two-box setup — Box A: 1× RTX 3090 24GB + 1× RTX 3060 12GB (36 GB),
Box B: 1× RTX 4060 Ti 16GB + 1× RTX 3060 12GB (28 GB). Total 64 GB VRAM across
both boxes. F16 (55 GB) fits on Box A with tight KV-cache; Q6_K (21 GB) and
Q4_K_M (16 GB) are the practical daily-drivers. F16 / Q8_K benchmark runs are
pending — this table will be updated when they land.
⚠️ Token & step limits: SWE-bench is the most constrained benchmark:
message_limit=100 (agent turns) × max_tokens=16384/turn ≈ 1.6M output-token
ceiling. The model is typically step-limited, not token-limited (agent runs
out of turns before exhausting the token budget). GPQA is single-shot with
max_tokens=65536, so the token ceiling is rarely hit (avg ~4K tokens/question).
Smoke test (F16 GGUF)
27x43 = 1161 (think 163 chars)
bat-and-ball = 0.05 (think 161 chars)
snail wall = day 5 (think 359 chars, base was 4000+)
