Agentic SFT on Wichtel, trained on a
6,000-sample slice of Lehrling-SFT
— real Claude Code session transcripts, 69% of which contain a tool call.
This adapter suppresses delegation and teaches investigation over action. Read the results
before using it.
Measured
No capability damage. ARC-Challenge (299 tasks, paired McNemar against the base) is flat at
every scale, despite this being the largest adapter in the family by effective delta:
scale
ARC
delta
p
0.25
64.88
+0.00
1.000
0.50
64.55
−0.33
1.000
0.75
65.22
+0.33
1.000
1.00
64.55
−0.33
1.000
Worth noting against the family: ‖B·A‖ = 18.95 here, versus 7.81 for a sibling adapter that
cost 4 ARC points at scale 1.0 (p = 0.036). Delta magnitude does not predict damage — what the
adapter changes matters more than how much. This one moves tool-call behaviour, which ARC never
exercises.
Agent behaviour changed, mostly for the worse. Ten held-out tasks from a repository that
contributed no training data, graded by running the repo's own test suite:
outcome
Wichtel
+Lehrling
passed
9/10
7/10
self, passed
7
7
escalated, passed
2
0
self, failed
0
3
escalated, failed
1
0
Every task the base delegated, this attempts alone. Two of those turned out fine — including one
the base had escalated and still failed. Three did not, and they failed in the same way.
Thinking mode is untouched. An 8-prompt probe at temperature 0, matched 12k token cap:
Wichtel
+Lehrling
unclosed think blocks
0/8
0/8
total reasoning
6,872 words
6,914 (1.01x)
This was not the expected result. The training targets contain no <think> blocks at all —
Claude Code does not export thinking — so 372 steps of never deliberating looked likely to
suppress it. It did not, and the probable reason is format_type: raw: the prompts are
pre-rendered conversations that never contain the <think> opener, so the model was never shown a
context where it sat inside a think block and produced nothing. The reasoning pathway was simply
never in the gradient.
Trained through a chat template with enable_thinking: true instead, the same data would likely
suppress reasoning — the model would see the opener followed by an immediate close, 372 times. The
raw format isolated the two behaviours by accident.
Practically: this composes with a reasoning adapter rather than competing with one.
The failure mode
The three regressions are not scattered. All three explored and never edited:
metrics_words: 10 tools — read, glob, execute x5 ... no write, no edit
"[Agent reached maximum turns without producing a final response]"
metrics_content_words: 6 tools — read, glob, read, read, git_log, read
(empty response)
The function under test was left holding raise NotImplementedError, untouched.
This is a data composition problem rather than a training failure. Claude Code explores heavily
before committing to an edit, and with one sample per assistant turn that exploration dominates
the dataset — Read, Glob, Grep, Bash vastly outnumber Write, Edit and Agent. The
adapter learned the reconnaissance half of the behaviour and not the decisive half.
The obvious next experiment: filter the source dataset to samples whose target is a mutating
action — Write, Edit, or Agent — and retrain. That is a filter on the existing 24k, not new
data, and it is testable on the same ten tasks.
Honesty about the statistics
Three lost, two won on ten tasks is p = 1.000. Nothing here is statistically significant, and
the direction should not be over-read. What raises it above noise is that the three failures share
one mechanism and one file family rather than scattering — a pattern that usually survives more
data, though that remains to be shown.
Overfitting
Not strongly indicated, with one caveat: no eval loss was recorded despite eval_steps: 0.5,
so the standard check was unavailable.
No format bleed. Across ten agent runs, zero responses contained Claude Code tool syntax and all
ten used the host harness's native tool names.
Verbatim memorisation probe on twelve training prompts at temperature 0: mean similarity 0.496,
with 2/12 above 0.9. Both high scores are tool-call structures, where the syntax is rigid and
much of the similarity is template rather than recall.
Training loss fell smoothly, 1.508 → 0.987, no divergence.
Training
base
nbeerbower/Wichtel-Qwen3.6-27B
method
SFT, LoRA r=32 α=64 dropout 0.05
data
6,000 shuffled samples of 24,395
schedule
1 epoch, 372 steps, lr 2e-4 cosine, warmup 0.05
batch
1 × grad-accum 16
sequence
max_length 4096, max_prompt_length 3600
format
raw — the prompts are pre-rendered Qwen3 conversations
The learning rate was too high. Loss bottomed at 0.947 by step 16, climbed to 1.212 by step 56
as the LR sat near its 2e-4 peak, and only recovered as cosine decay took hold — roughly the first
120 steps were spent undoing the first 40. 5e-5 would likely reach a lower final loss in the same
budget. The 2e-4 was inherited from a sibling run that was installing a capability the base
lacked entirely, where aggressive updates are appropriate; nudging behaviour a model already has
is a different problem.
Subsampled to 6,000 because the full 24,395 samples at ~3,300 tokens each is ~79M tokens per
epoch, which on the training hardware is 15 days. At 6,000 it is 3.3 days.
