Schierling-Qwen3.6-27B-LoRA
Schierling: German for hemlock.
A LoRA that teaches
Hemlock — a systems scripting language
with manual memory management — to Qwen3.6-27B. Supervised fine-tuning on 5,562 examples.
hembench: 10.1% → 65.0%. Every generated program is executed by the real interpreter and its
stdout compared against expected output, so these are programs that actually run, not code that
looks plausible.
Results
| level | | TIES | + Schierling |
|---|
| L1 | syntax | 1/9 | 8/9 |
| L2 | stdlib | 0/5 | 1/5 |
| L3 | algorithms | 0/7 | 5/7 |
| L4 | systems | 0/7 | 5/7 |
| L5 | translation | 0/5 | 3/5 |
| L6 | debugging | 3/5 | 4/5 |
| overall | weighted | 10.1% | 65.0% |
| tasks passed | 4/38 | 26/38 |
22 tasks fixed, 0 broken. Nothing the base could already do was lost.
The sharpest single number: generated code that fails to execute drops from 30 tasks to 8.
The base does not write runnable Hemlock at all — its 10.1% comes almost entirely from L6, where
the task supplies existing code to repair rather than asking for a program from scratch.
L2 (stdlib) is the weak spot at 1/5. Standard-library work needs exact function names and
signatures — memorised API knowledge rather than transferable syntax. The stdlib portion of the
training data (the formulary rows folded into codex3) is the smallest slice, and the per-task
cap trimmed it further. More stdlib coverage is the obvious next improvement.
Why SFT and not DPO
An earlier attempt taught Hemlock with 400 ORPO preference pairs and did not work. Inspecting
that dataset showed why: only 186 pairs differed in code — the other 214 differed in which tool
was called, so the objective could be satisfied by fixing tool selection without emitting a single
correct line of Hemlock. Preference optimisation sharpens choices among behaviours a model already
has; it does not install a grammar it has never seen.
Same language, same base, two instruments: 400 preference pairs moved nothing, 5,562 supervised
examples moved 55 points.
Training
| |
|---|
| base | nbeerbower/Qwen3.6-27B-TIES |
| method | SFT, LoRA r=32 α=64 dropout 0.05 |
| data | 5,562 rows |
| schedule | 2 epochs, 682 steps, lr 2e-4 cosine, warmup 0.05 |
| sequence | max_length 1536, max_prompt_length 768 |
| batch | 2 × grad-accum 8 = effective 16 |
| source | rows | contributes |
|---|
hemlang/Hemlock-SFT | 2,673 | docs, semantics, compiler/parity tests |
hemlang/hemlock-codex3-SFT | 923 | execution-verified generation/translation/stdlib |
hemlang/hemlock-transmutation | 1,966 | verified C/Python → Hemlock, templated |
Two filters the dataset cards do not apply:
- 150
bench_overlap rows dropped. Their task names collide with hembench scenarios. The
codex3 card suggests keeping them for SFT — fine if you never benchmark on hembench, and
invalidating if you do. Dropping them is what makes the 65.0% a held-out number.
- Per-task cap of 4. Tasks recur ~6× (one row per source language, identical target), so
uncapped, 270 programs would be seen far more often than everything else.
codex-SFT, codex2-SFT and formulary-SFT are not used separately — codex3 supersedes and
merges all three.
A note on the loss curve
Training loss fell 0.863 → 0.231 across epoch 1, then stepped to 0.068 at the epoch boundary and
stayed flat for the remaining 230+ steps. That is the shape of memorisation rather than learning,
and it suggested the second epoch was wasted.
The benchmark says otherwise — 26/38 on held-out tasks is not what a memorised model produces.
Still, 1 epoch is worth trying: it would halve a 14-hour run. No epoch-1 checkpoint was kept
here to compare against, since Merlina's epoch-end save overwrites; set save_steps explicitly
if you want intermediates.
Limitations
- stdlib is weak (1/5). Expect to check API calls against the docs.
- Not a tool-calling model. The training data is 5,562 rows of direct code generation with
zero tool calls. If you stack this with an agent adapter, measure whether delegation survives.
- Trained and evaluated only at r=32 on the TIES base; behaviour on other Qwen3.6 descendants is
untested.
- 38 benchmark tasks is a small sample — level scores of 5/7 carry wide error bars even where the
overall difference is unambiguous.
