Hemlock-Qwen3.8-27B
A 27B vision-language model that writes
Hemlock.
HemBench: 33/38 (86.8%), pass@1 0.816 — the highest score recorded on the suite.
What it is
The name says Qwen3.8, and the lineage is more mixed than that: WichtelHui is a 50/50 SLERP of
Wichtel-Qwen3.6-27B (Qwen3.6-line) and Huihui-Qwen3.8-27B-abliterated (Qwen3.8-line), so this
model is roughly half of each pretrain. The adapter was trained on Qwen3.8-27B.
HemBench
38 tasks, generated code executed against expected output. Run with thinking disabled — as a
reasoning model it otherwise spends the whole token budget thinking and emits nothing.
| model | HemBench | notes |
|---|
| Huihui-Qwen3.8-27B-abliterated | 4/38 | Qwen3.8 has no Hemlock exposure |
| Wichtel-Qwen3.6-27B | 29/38 | trained on Hemlock, 3.6-line |
| WichtelHui (this model's base) | 22/38 | no adapter |
| Hemlock-Qwen3.8-27B | 33/38 | pass@1 0.816 |
On precision: HemBench is 38 items sampled at temperature 0.2, so single runs move by a
couple of tasks. The 33/38 above is from 3 samples per task (114 generations); a single-sample
run of the same weights scored 29/38. Treat differences of 2–3 tasks between models as noise
unless they were measured the same way.
Failures are 4 wrong-output and 1 runtime error. The un-adapted Qwen3.8 parent fails 32 of 38
at runtime, because it writes Rust- and C-shaped code when asked for Hemlock; that is the
failure mode the adapter removes.
General capability
The adapter does not trade general ability for Hemlock. ARC-Challenge, 299 tasks, scored by
summed logprob — deterministic, no sampling:
| ARC |
|---|
| Qwen3.8-27B | 52.84% |
| WichtelHui (no adapter) | 60.54% |
| Hemlock-Qwen3.8-27B (this model) | 66.22% |
A separate configuration — WichtelHui with an earlier adapter checkpoint applied at runtime
rather than merged — measured 68.56% on the same 299 tasks. The 66.22% above is this model's
own score; the other figure is recorded here only so the difference is not mistaken for a
regression.
Safety and truthfulness
Inherited from WichtelHui and not re-measured after the merge:
| |
|---|
| ccp_truth | 18/18 |
| safety_control | 2/2 (refuses both probes) |
| capability | 3/3 |
| ccp_truth_neutral | 3/4 |
The abliterated parent scores 0/2 on safety_control on its own; the SLERP with a
safety-intact parent restores both refusals while keeping the full 18/18 on ccp_truth.
The adapter
Trained on
hemlang/Hemlock-SFT-combined
— 5,852 rows from the three Hemlock SFT sources, with the 150 rows flagged as overlapping
HemBench excluded. Training on those would make the score above a measure of recall.
LoRA r=32, α=64, 2 epochs, lr 2e-4, max_length 2048, on Qwen3.8-27B.
An earlier checkpoint at 40% of training scored within noise of the final one (pass@1 0.737
vs 0.816 at 3 samples; 31/38 vs 29/38 at one sample, in the opposite order). The fully trained
adapter ships because it is the complete artefact, not because it measurably won.
Vision and MTP
Both intact: 333 visual.* and 15 mtp.* tensors, 1199 total. The adapter targets only
attention and MLP projections; everything else is copied unchanged from the base rather than
reconstructed, and the merged weights are asserted to differ from the base before shipping.
Usage
1from transformers import AutoModelForImageTextToText, AutoProcessor
2model = AutoModelForImageTextToText.from_pretrained("hemlang/Hemlock-Qwen3.8-27B", dtype="bfloat16")
Serve with thinking disabled for code generation.
Limitations
- HemBench is 38 tasks. It measures whether generated Hemlock runs and produces the expected
output, not whether it is idiomatic, efficient, or safe.
- Safety numbers are inherited from the base and were not re-run after merging the adapter.
- Half the weights come from a Qwen3.6-line parent, so behaviour will not always match a pure
Qwen3.8 derivative.