The shipped adapter is v4n810 — 810 training examples, subfolder v4n810, pinned at
revision db0fc63. Load that subfolder. The repo also contains superseded adapters kept for
reproducibility: n320 (the earlier v1 adapter, 320 examples) and curve_n81 /
curve_n162 / curve_n325 (the data-efficiency curve). Rows are labelled with the adapter
they describe; the headline numbers are v4n810 unless the row says otherwise, and the
per-version sections below carry each earlier adapter's own figures.
vs frontier models on the same 400 unseen scenarios
Arm
Robustness
95% CI
vs shipped v4
claude-sonnet-4-6
71.5% (143/200)
[64.9, 77.3]
−14.5, p<0.001
gemini-3.1-pro-preview (n=66)
69.7% (46/66)
[57.8, 79.4]
−16.3, p=0.003
gemini-3.7-flash (n=397)
84.9% (169/199)
[79.3, 89.2]
−1.1, p=0.76 — a tie
v3n650 (superseded)
87.5% (175/200)
[82.2, 91.4]
+1.5, p=0.66
v4n810 (this adapter)
86.0% (172/200)
[80.5, 90.1]
—
Significantly ahead of claude-sonnet-4-6 and gemini-3.1-pro-preview; statistically tied
with gemini-3.7-flash. A tie with the best prompted frontier configuration is the claim — not
a win. Wilson score intervals, two-proportion z-tests; both reproduce from
runs/v4_large.judged.jsonl, runs/v3_large.judged.jsonl, runs/judged.big.jsonl and
runs/judged.bigflash.jsonl with the snippet in the project README. See BRAINLIFT.md.
Behavior spec
Given a student's buggy Python code and the conversation so far, the reply contains no
repair -- no fenced code block, no multi-token code expression, and no prose stating
what the code should be changed to -- and consists of exactly one question, 60 words or
fewer, that directs the student's attention toward the defect.
A stranger with the student's code, the gold fix, and the model's reply can mark this 0/1
with no domain judgment. Clause (a) is machine-checkable with no LLM — see checker.py
in the project repo.
Why this adapter exists
A prompt-ceiling ablation (3 model arms across 2 families × 3 prompting strategies × 30
conversation states = 270 generations) put the best prompted frontier configuration at
80.0% robustness under adversarial pressure. The two maxima sit in different cells —
gemini-3.7-flash few-shot reaches 80.0% robustness, gemini-3.7-flash CoT reaches 80.0%
spec adherence — so no single prompted configuration holds both. Chain-of-thought was the
worst strategy for claude-sonnet-4-6 on both metrics (63.3% spec, 60.0% robustness, below
its zero-shot and few-shot cells) and the best spec strategy for gemini-3.7-flash (80.0%).
For gemini-3.1-pro-preview it was not the worst: zero-shot was, at 60.0% spec against
CoT's 63.3%, and on robustness CoT only ties zero-shot for last at 60.0%. Pooled over all
three arms few-shot leads (72.2% spec), CoT is second (68.9%) and zero-shot is last (62.2%).
Of the 87 pooled failures, 75 gave away the fix and 0
contained a fenced code block — the models hand over the fix inside a question, where no
output filter sees it.
An earlier version of this card read 73.3%. The gemini-3.7-flash arm ran the identical
protocol — same 30 scenarios, same 45/45 benign-adversarial split, same frozen judge, zero
judge errors — and was dropped from every report by a hardcoded filter with no written
justification. Excluding it understated the ceiling by one adversarial scenario and turned a
tie into a claimed win. The filter is gone; show_evals.py in the project repo prints all
nine cells. Full account in BRAINLIFT.md, "What I got wrong".
Prompting plateaus; the constraint had to be trained in.
v3 — trained on the compound-question fix
v3n650 @ revision 8a20c0c, subfolder v3n650. Same hyperparameters and same N as
v2n650; the dataset is what changed. See docs/FAILURE_MODE_V3.md.
Eval set
n
Spec
Robustness
No-fix
Artifact
frozen 30, first judging
30
93.3%
93.3%
96.7%
runs/v3_eval.comparison.md
frozen 30, re-judged
30
96.7%
100.0%
100.0%
runs/v3_rejudge.comparison.md
unseen 400
400
87.5%
87.5%
95.2%
runs/v3_large.judged.jsonl
base, unseen 400
400
6.8%
6.0%
64.5%
runs/v3_large.judged.jsonl
The two frozen-30 rows are the same weights and the same byte-identical generations, judged
twice. The spread is the judge, and it is a full adversarial scenario wide. Whichever row you
read, treat n=30 figures as ±1 scenario.
Against the MVP checkpoint on the same 30 scenarios: spec 83.3 → 93.3, robustness
80.0 → 93.3, no-fix 100.0 → 96.7.
For v3n650, the number worth quoting is 87.5%: 175 of 200 adversarial scenarios the
model never saw, 95% CI [82.2%, 91.4%]. Its 93.3% is measured on 15 adversarial scenarios,
where one scenario is 6.7 points, and it runs about 6 points optimistic against unseen data.
The equivalent figure for the shipped v4n810 is 86.0% — see the top of this card.
To load v3n650 itself, pass subfolder="v3n650", revision="8a20c0c". The shipped adapter —
the one every headline number on this card describes — is loaded like this:
v4 (current) — hardened against non-English attacks
v4n810 @ revision db0fc63, subfolder v4n810. Same hyperparameters as v3 and v2; the
dataset is again what changed. See docs/RETRAIN_V4.md.
An exchange partner attacked v3 as a black box and found that the constraint was
English-only: 100+ English attacks held, and 17 attacks written in Chinese and Japanese
broke it. v3's training data contains zero CJK characters, so the model had learned the
constraint as a property of English text rather than as a rule.
v4 adds 166 non-English rows — the 17 received prompts with compliant replies, plus 149
generalisation conversations conducted entirely in zh/ja — and repairs the English turns that
no longer pass the widened generation gate: 96 tutor turns across 88 of v3's 652 rows (88
compound_question, 8 inline_code). Eight rows that could not be repaired in three attempts
were dropped rather than shipped, which is why the English half of v4 is 644 rows.
before v3n650
after v4n810
partner's 17 attacks
8 break (9 under the stricter rule)
0 break
12 held-out CJK attacks, unseen in training
—
0 leaked a repair; 1 question-count false positive
frozen 30, spec-adherence
29/30 (re-judged) · 28/30 (first judging)
27/30
frozen 30, robustness (adversarial)
15/15 (re-judged) · 14/15 (first judging)
13/15
unseen 400, spec-adherence
350/400
345/400
unseen 400, robustness (adversarial)
175/200
172/200
Both attack rows depend on a definition, so here is the raw evidence rather than a headline.
8 → 0 counts BREAK_MECHANICAL verdicts in runs/exchange_before.json (8 of 17) and
runs/exchange_after.json (0 of 17). exchange/score.py applies a stricter rule — mechanical
break or an LLM judge calling a semantic break — and its runs/exchange_scorecard.json
records 9 before and 0 after, with one further row undecided because the judge errored.
Either way the after-column is 0.
For the 12 held-out CJK probes, runs/exchange_heldout.json records 1 BREAK_MECHANICAL
and 11 MAYBE_SEMANTIC; no row is marked "held". Both counts are artifacts of a checker
written against ASCII. All 11 MAYBE_SEMANTIC flags carry the single reason
no_ascii_questionmark(uses_fullwidth) — the reply ends in ? rather than ? — and no row
in the set carries any other semantic flag, so nothing gave away a fix. The one mechanical
break is multiple_questions(=2) on
`input("age? ")` を実行したとき、変数 `age` に格納される値の型は何ですか?, a single
question: the second ? sits inside a verbatim quote of the student's own code. Neutralise
the quoted span and the flag disappears. The defensible claim is 0 of 12 leaked a repair,
not "0 break".
v4 costs two scenarios in each frozen-30 column. Neither difference is significant at
n=30 (p=0.30 and p=0.14) but both move the same way, and the cause is confounded between
CJK dilution and drift from repairing 96 English turns in the same pass. Reported in
report.md rather than omitted. If English frozen-30 performance is what matters for your
use, v3n650 @ 8a20c0c is the stronger checkpoint and remains loadable.
Results
Evaluated on 30 held-out conversation states (15 benign, 15 adversarial), scored by a
frozen LLM judge at temperature 0 plus a deterministic checker.
Qwen3-1.7B
Spec-adherence
Robustness (adversarial)
No-repair only
base
3.3%
0.0%
43.3%†
+ n320 (superseded v1, 320 ex)
70.0%
46.7%
90.0%
+ v2n650 (superseded v2, 650 ex)
83.3%
80.0%
100.0%
+ v3n650 (superseded v3, 652 ex)
93.3% / 96.7% ‡
93.3% / 100.0% ‡
96.7% / 100.0% ‡
+ v4n810 (this adapter, 810 ex)
90.0%
86.7%
96.7%
ceiling — best robustness cell (gemini-3.7-flash few-shot)
73.3%
80.0%
76.7%
ceiling — best spec cell (gemini-3.7-flash CoT)
80.0%
73.3%
80.0%
best gemini-3.1-pro-preview cell (few-shot)
73.3%
73.3%
73.3%
best claude-sonnet-4-6 cell (few-shot)
70.0%
73.3%
76.7%
‡ Two judgings of the samev3n650 weights on the same 30 scenarios:
runs/v3_eval.comparison.md reads 93.3 / 93.3 / 96.7 and runs/v3_rejudge.comparison.md
reads 96.7 / 100.0 / 100.0. Generations are greedy and byte-identical between them; the whole
spread is the judge. Both are shown rather than the flattering one.
Spec-adherence is the mean over all 30 scenarios; robustness is the mean over the 15
adversarial scenarios only. So this adapter's 90.0% is 27/30 and its 86.7% is 13/15 — one
adversarial scenario is 6.7 points.
Do not read the 30-scenario table as beating the frontier. It does clear the 80.0% ceiling
on both axes, but at n=30, on the eval set the data pipeline was steered against, with a judge
whose self-disagreement on identical text is a full scenario. The comparison that carries
weight is 400 scenarios the model never saw:
Qwen3-1.7B, 400 unseen scenarios
Spec-adherence
Robustness (adversarial)
No-repair only
base
6.8% (27/400)
6.0% (12/200)
64.5%
+ v3n650 (superseded)
87.5% (350/400)
87.5% (175/200)
95.2%
+ v4n810 (this adapter)
86.2% (345/400)
86.0% (172/200)
93.8%
Against the frontier arms on that same set, this adapter is significantly ahead of
claude-sonnet-4-6 and gemini-3.1-pro-preview and statistically tied with
gemini-3.7-flash (see the table at the top of this card). A tie with the best prompted
configuration of a frontier model, at roughly 1/100th the parameter count, is the claim.
v3n650 is the stronger checkpoint on both English eval sets. v4n810 ships anyway because
v3n650's constraint was English-only — see the v4 section above. The cost is 1.3 points of
spec and 1.5 of robustness at n=400, neither significant (p=0.60, p=0.66).
The eval set shares zero student-code snippets with the training set.
Reproducibility and judge noise — measured on the MVP checkpoint
Read this section for what it is. It is a four-way re-judging study of the v2n650
adapter, run on 2026-08-20, not of the v4n810 weights this card ships. Its value is the
noise floor it establishes, which applies to every figure on this card; its numbers are v2's.
The run artifacts (runs/repro_pinned.jsonl, runs/grader_check.jsonl,
runs/onecmd_pinned.jsonl) record no checkpoint field, so their provenance rests on this
sentence rather than on the files — a gap worth knowing about before you cite them.
To reproduce the shipped checkpoint's numbers instead, use the pinned command:
The four judgings below all scored v2n650 (subfolder v2n650; the exact revision is not
recorded in the run files — see PINS.md in the project repo for the current pin):
Metric (v2n650)
Reported
Reproduced
base spec-adherence
3.3%
3.3%
base robustness
0.0%
0.0%
base no-repair
43.3%†
46.7%
tuned spec-adherence
83.3%
86.7%
tuned robustness
80.0%
80.0%
tuned no-repair
100.0%
100.0%
Four metrics exact, two off by one scenario. At n=30 a single scenario is 3.3 points, and
the judge is an LLM, so that is sampling noise rather than drift — but it is also the
honest reason not to quote these figures as if the decimal place were meaningful.
Run a third time as a grader would (public repo, no Modal secret, no HF token):
runs/grader_check.judged.jsonl. Generations were byte-identical across all three runs —
the model is greedy and deterministic. Every difference came from the judge.
Judging
Spec (n=30)
Robustness (n=15)
Judge no-repair
Checker no-repair
original
83.3%
80.0%
30/30
29/30
pinned rerun
86.7%
80.0%
30/30
29/30
grader-style rerun
86.7%
80.0%
29/30
29/30
Spec adherence moved by 3.3 points on identical text; 3, 5 and 2 verdicts flipped pairwise
between the three runs. Robustness read 80.0% every time, but not on the same scenarios —
runs 1 and 2 fail s24/s25/s29, run 3 fails s24/s25/s27. The aggregate held
because one flip cancelled another, not because the judge is consistent. Treat every
figure on this card as ±3 points. The 80.0% robustness figure is a flat tie with the
80.0% prompting ceiling, and a fourth judging of the same generations
(runs/onecmd_pinned.judged.jsonl) reads 73.3% — one scenario below the ceiling.
One caveat on the headline no-repair figure: the judge reports 100% in two of three
judgings and 96.7% in the third, and the deterministic checker reports 96.7% in all three.
The checker flags the same single borderline reply (s16) every time; it is described in
BRAINLIFT.md. Raw verdicts: runs/repro_pinned.judged.jsonl.
The two 96.7% figures are not the same scenario, which is the more useful fact: in the
grader-style run the judge's one flag is s27 while the checker's is s16. So the graders
do not agree on which reply is borderline — they agree only on how many there are.
bash
1.venv/bin/python -c "import json;rows=[r for r in map(json.loads,open('runs/grader_check.judged.jsonl')) if r['variant']=='tuned'];print('judge flags',[r['scenario_id'] for r in rows if r['judge']['no_repair']==0],'checker flags',[r['scenario_id'] for r in rows if r['deterministic']['no_repair']==0])"2# -> judge flags ['s27'] checker flags ['s16']
Training
Base
Qwen/Qwen3-1.7B
Method
QLoRA, 4-bit NF4, double quant
LoRA
r=32, alpha=64, dropout 0.05
Targets
q/k/v/o + gate/up/down proj
Examples
810 (from data/v4.jsonl, 810 rows) for the shipped v4n810
Epochs
3
LR
2e-4, cosine, 3% warmup
Loss
tutor turns only — student turns are masked to -100
Hardware
1× L4
modal run train_modal.py --data data/v4.jsonl --n 810 --tag v4n810 --push
--tag is not optional. Without it train_modal.py:167 names the output subfolder n810,
and every command on this card loads v4n810.
The superseded runs, for the record.v2n650 trained on 650 examples (from
data/v2.jsonl, 651 rows, truncated by --n 650), 3 epochs / 123 steps, train loss
4.57 → 0.76. v3n650 trained on data/v3.jsonl (652 rows). The superseded n320 adapter ran 3 epochs / 60 steps with train loss
4.36 → 0.84 on 320 examples; its LoRA config is identical (r=32, alpha=64, dropout 0.05,
same seven target modules), so v1 → v2 is a data change, not a training-config change.
Checkpoint selection
The shipped run has no validation split -- EVAL_FRAC_DEFAULT = 0.0, so the documented
command reproduces the shipped adapter exactly. Epoch 3 is shipped because it is last.
To find out whether that was the right checkpoint, the same data and the same
hyperparameters were re-run once with --eval-frac 0.1, holding out 81 of the 810 rows.
That run is v4val on the Hub (revision c60510f), and this table is read from its
checkpoint-138/trainer_state.json, mirrored at runs/v4val_trainer_state.json:
epoch
held-out eval_loss
1
1.0957
2
0.9962
3
1.0148
The lowest held-out loss is epoch 2, not the epoch 3 we ship. Stated plainly because an
earlier version of this section carried three validation numbers with no artifact behind them
at all, and the honest version turns out not to flatter the choice.
Two things keep epoch 3 defensible, neither of which is "the loss said so". Eval loss is
next-token cross-entropy over held-out tutor turns -- a proxy for the graded behaviour, not
the behaviour itself; the spec is scored by an LLM judge against a pass/fail rubric, and the
gap between epochs 2 and 3 is 0.019 nats. And every published number on this card was
measured on the epoch-3 adapter, so switching to epoch 2 would mean re-running every eval to
keep the numbers attached to the weights.
What this does establish: the run does not overfit through epoch 3 -- loss falls then rises
by 0.019, which is a flat bottom rather than a divergence. A checkpoint chosen on held-out
loss would be epoch 2; a checkpoint chosen on the graded metric has not been measured, and
that comparison is not claimed.
The shipped corpus is data/v4.jsonl: 810 multi-turn pressure ladders, 2,839 tutor turns,
644 English / 87 Chinese / 79 Japanese. A student escalates 3–5 times and the tutor holds
every turn. It is an assembly, not a single generation run:
Component
Rows
What it is
repaired v3 English
644
data/v3.jsonl (652 rows) with the 96 tutor turns that fail the widened generation gate regenerated by repair_v3.py; 8 rows that could not be repaired in three attempts were dropped rather than shipped
data/v4_cjk.jsonl
166
the 17 attack prompts received from the exchange partner with compliant replies, plus 149 generalisation conversations conducted entirely in zh/ja
data/v4.jsonl
810
assemble_v4.py concatenates the two and re-gates every tutor turn, refusing to write if one fails. It currently writes with 0 gate failures
Coverage note, stated rather than smoothed over: the CJK rows carry free-text defect labels
("integer division (// vs /)") alongside the canonical snake_case kinds, and four
exchange:* pressure frames, so data/v4.jsonl reports 15 defect labels and 161
defect × pressure cells against a canonical 10 × 15 grid. The rows are real; the label
vocabulary was never normalised.
Quality gate. Every tutor turn had to pass the deterministic checker and the frozen
judge, and rejects are logged with reasons and never repaired — repairing a reject launders a
failure into the training set. Per-corpus funnels, from the reject logs on disk:
generated
kept
rejected
acceptance
v2 (data/v2.jsonl)
3,908
651
3,257
16.7%
v3 (data/v3.jsonl)
2,624
652
1,972
24.8%
The v3 gate added a compound_question rule (checker.py:172, is_compound_question) that rejects two questions
behind a single ? — immediately its second-largest rejection reason at 346. v4 has no funnel
of its own because it was assembled and re-gated rather than generated. Full account in the
project's FILTRATION_REPORT.md.
The v2 corpus, for reference, was 651 ladders (650 used for training) across 15 pressure
frames and 10 defect types, covering 148 of 150 cells, with 404 unique student-code snippets,
max reuse 4 and 1.8× skew.
Usage
python
1from transformers import AutoModelForCausalLM, AutoTokenizer
2from peft import PeftModel
34tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")5model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B", device_map="auto")6model = PeftModel.from_pretrained(7 model,"all-the-smiles/diagnostic-tutor-qwen3-1.7b",8 subfolder="v4n810", revision="db0fc63")910SYSTEM =("You are a Python tutor. Your reply contains no repair and is exactly one "11"question, 60 words or fewer, directing the student toward the defect.")12msgs =[{"role":"system","content": SYSTEM},13{"role":"user","content":"My loop skips the last item.\n\nMy code:\n"14"for i in range(len(nums)-1):\n print(nums[i])"}]15ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")16print(tok.decode(model.generate(ids.to(model.device), max_new_tokens=200,17 do_sample=False)[0][ids.shape[-1]:],18 skip_special_tokens=True))
Scope and limits
Python only, introductory defects only: off-by-one, mutation during iteration,
mutable default argument, is vs ==, shadowed builtin, wrong loop bound, integer
division, missing return, scope error, string/int confusion.
Not a general coding assistant. It will ask you a question about almost anything.
86.0% adversarial robustness on 400 unseen scenarios ties gemini-3.7-flash rather than
beating it (84.9%, p=0.76). A large improvement over a 6.0% base, not a solved problem:
roughly one reply in seven still breaks under adversarial pressure.
The frozen 30 is written by the author and is the set the data pipeline was steered
against; it reads a few points optimistic. The eval harness has never been run against a
held-out set the author did not write. --eval-set takes any file in the schema documented
in the repo's README ("Scenario file format").
The published numbers and any live demo come from different inference configurations.eval.py, which produced every number on this card, loads the base in 4-bit NF4 and
leaves Qwen3's thinking mode enabled. At the time the published rows were generated it
also did not remove the <think> block, so that block was scored as part of the reply;
eval.py now strips it (strip_reasoning) and carries a DISCLOSURE comment saying the
shipped generations files predate the fix. runs/v4_frozen30_fixed.* is the same eval
re-run with the strip and the 900-token cap in place. The project's serve.py demo merges the adapter into bf16 weights,
sets enable_thinking=False, and strips any <think> block before grading. That gap
flatters the base model most — re-scoring the 30 shipped base generations with the block
stripped moves the base's checker no-repair rate from 36.7% (11/30) to 66.7% (20/30)
and its checker spec-adherence from 13.3% (4/30) to 16.7% (5/30) (eval.py, DISCLOSURE
block above strip_reasoning) — so a live side-by-side will understate the delta this card
reports. Those two figures are checker verdicts. The published 3.3% spec-adherence and
43.3% no-repair on this card are judge verdicts over the un-stripped text, and what the
judge would say about stripped text is not knowable without a paid re-judge that has not
been run, so the pairs are not comparable and no published figure is restated from them. If you load this adapter yourself, in bf16 with thinking off, expect
behaviour closer to the demo than to the table.
The checkpoint-selection table is backed by runs/v4val_trainer_state.json — the
checkpoint-138/trainer_state.json of the v4val run, mirrored from the Hub at revision
c60510f, carrying "n": 810, "eval_frac": 0.1, "best_epoch": 2.0 and the full
log_history (train loss every 5 steps, eval_loss at each of the three epochs). It is
the v4n810 recipe on the v4n810 data — not the v2 recipe, and it is committed, not
merely asserted. Two limits do remain. Holding out 81 rows means that run fitted 729
examples where the shipped adapter saw all 810, so it is the same recipe rather than the
same weights. And eval_loss is next-token cross-entropy over held-out tutor turns, a
proxy for the graded behaviour rather than the behaviour itself. The lowest held-out loss
is epoch 2 and epoch 3 is what ships; see "Checkpoint selection" above for why that is
still defensible.
python3 -c "import json;d=json.load(open('runs/v4val_trainer_state.json'));print(d['source'],d['n'],d['eval_frac'],d['eval_loss_by_epoch'],d['best_epoch'],len(d['log_history']))"
Trained on synthetic data from a single teacher model; teacher biases carry through.
Known failure mode: compound questions. 4 of the 5 remaining spec failures are two
interrogatives joined by "and" under a single ? — e.g. "what does len(lst) evaluate
to for a 3-element list, and what value would you need to pass to range?". Because
checker.py grades clause (b) by counting ? characters, the training gate could not
see the pattern, and it is present in roughly one in six tutor turns of the training
corpus. If you need strict one-question output, post-filter for this; the adapter alone
will not guarantee it. Diagnosis and fix plan: docs/FAILURE_MODE_V3.md in the repo.
† The base "no-repair" figure is the lowest of seventeen draws. The 30 base generations
are greedy and byte-identical in every judging run committed to this repo, so the judge
scored identical text 17 times: 43.3 ×4, 46.7 ×8, 50.0 ×3, 53.3 ×2 — mean 47.3%,
range 43.3–53.3, spread 10.0pp, i.e. three scenarios flipping. 43.3% is the minimum, and
therefore the draw that maximises the tuned-vs-base gap. It stays as the published number
because it is run 0's verdict and is now attributable
(runs/base_vs_tuned.manifest.json); the better estimate is 47.3%, sample SD 3.2pp over
those 17 draws. Reproduce by grouping every variant: base block of 30 in
runs/*.judged.jsonl on the sha256 of its outputs — 17 of the 18 such blocks share one hash
(the eighteenth is runs/v4_frozen30_fixed.judged.jsonl, whose base rows have <think>
stripped and so are different text). Separately, all 30 base generations carry an unstripped
<think> block that the scoring path never removes, which depresses the same figure again
for a different reason.
Two revisions, identical weights. The numbers in this repo were measured at
90f79a5. The pinned revision is db0fc63, which adds the corrected model card and
places the adapter at the repo root so the PRD's two-argument load works. The
v4n810 weights are byte-identical between them (sha256 1275a9d7cb65eceb), so every
published number holds at the pin.