LoRA adapter teaching Qwen3-1.7B padel rule fidelity: answers are rule-accurate per the
FIP regulations and import nothing from tennis/squash.
45.2% spec adherence against 19.4% for the untuned base — and 55.6% control pass, the
first checkpoint to beat the base model on control rather than trading it away.
Trap scenarios are ones where the tennis/squash prior gives a wrong answer. Control
scenarios are ones where the prior-consistent answer is correct — they exist to catch
paranoid overcorrection, so both failure directions are scored.
Checkpoint
Data
Spec adherence
Trap pass
Control pass
Strict
Flagged
Qwen3-1.7B base
—
19.4%
9.1%
44.4%
19.4%
4
v1 (1200 it)
953
25.8%
36.4%
0.0%
22.6%
3
v2
953
41.9%
45.5%
33.3%
35.5%
6
v3
1181
38.7%
40.9%
33.3%
38.7%
0
v4 (this one)
1496
45.2%
40.9%
55.6%
45.2%
1
The control story
Fine-tuning for trap performance initially destroyed control performance. v1 drove trap
pass from 9.1% to 36.4% and control pass from 44.4% to 0.0% — the model learned to
contradict the user reflexively, "correcting" rules that are genuinely shared with tennis.
v2 and v3 recovered control to 33.3%, still below the untuned base. A dedicated
"sameness" slice (228 examples) added in v3 did not move it at all.
v4 adds a "binding" slice (315 examples) on top of sameness, and control pass reaches
55.6% — above the 44.4% base — with trap pass unchanged at 40.9%. The overcorrection
failure is closed without giving back trap performance.
Per control item, base → v4: pd-net-touch, pd-netcord, pd-scoring and
pd-two-serves move fail → PASS; pd-body-touch, pd-rally-volley and pd-tiebreak
move PASS → fail. It is a net gain over a partly different set of items, not a superset.
With only 9 control items, one item is 11 points — treat the control column as coarse.
"Strict" — the deterministic second check
strict = judge verdict AND zero deterministic contamination flags. The flags are regex
rules (sentence-level co-occurrence for two diagnosed rule errors, plus a tennis-import
lexicon) run with no API calls.
These checks are now negation-aware (checks v2). The first version was negation-blind:
it matched a term regardless of polarity, so correct answers that named the concept were
flagged. Both of v4's judge-passed-but-flagged answers were false positives of that kind —
"The padel court has no tramlines or doubles alleys at all" tripped the tennis lexicon, and
a correct statement of the serve-let rule tripped the let/fence rule because any correct
statement of it must mention both.
v2 clears a term when a denial precedes it, and clears the let/fence rule when the sentence
rejects the let or negates the fence contact. Effect on this checkpoint:
strict (checks v1)
strict (checks v2)
flags v1
flags v2
v3
38.7%
38.7%
2
0
v4
38.7%
45.2%
4
1
The apparent v3/v4 strict tie was an artifact of the checks and is gone. v4's strict now
equals its judge score. Judge-side numbers (spec, trap, control) were unaffected — only the
deterministic column moved.
Two regressions found while building v2 are pinned in a self-test that runs in CI
(padel/test_checks_v2.py): an unrelated negation elsewhere in a sentence must not clear a
genuine contamination. Every flag that cleared was audited by hand against its source
sentence before the change was accepted.
Adversarial-hardened variant (v5) — not the default
A second checkpoint lives in this repo under adversarial-v5/. The root files remain v4
and remain the default; v5 is published for transparency, not as a replacement.
It was trained for an Adversarial Exchange: a partner red-teamed the pinned v4 checkpoint
and delivered 25 reproducible spec breaks. v5 adds 71 corrected, paraphrased versions of
those attacks to the v4 data (paraphrases, not the literal attack strings — max similarity
to a source prompt 0.687).
MLX users point --adapter-path at the adversarial-v5/ folder instead (the root of that
subfolder holds the MLX-format weights).
The honest result
v4 (default)
v5
Partner's 25 attacks — breaks
24/25
15/25
Held-out 21 attacks — breaks
14/21
16/21
Core 31 — spec / trap / control
35.5 / 31.8 / 44.4
35.5 / 31.8 / 44.4
The hardening did not generalise. v5 fixes 9 of the partner's rules and regresses none
of them — but those attacks' paraphrases were in its training data. On a held-out set of 21
attacks it had never seen, it got worse, not better.
Three strategies were tried and measured: corrected-attack data (v5), grounding-scale
capability data (v6), and 3x upweighting of the corrected attacks (v7). Held-out breaks
across all four checkpoints: 14, 16, 14, 14 — the number does not move. Both v6 and v7
also damaged core control pass (44.4% → 11.1% and 22.2%), so robustness was being bought
with core performance; neither is published.
Core behaviour is unchanged between v4 and v5 in aggregate, though 12 of 31 individual
verdicts differ, so read it as a wash rather than an identical model.
Full analysis, per-item tables and the failed strategies: padel/adversarial/RESULTS.md in
the project repo.
Data efficiency
Nested subsets, ~2.44 epochs held constant at every point so the curve varies dataset
size only:
N
147
295
590
1181
1496
Trap pass
13.6%
22.7%
31.8%
40.9%
40.9%
Control pass
22.2%
11.1%
22.2%
33.3%
55.6%
Trap pass is monotonic through N=1181 and then flattens — the first saturation signal.
Control pass is still climbing steeply. More data of this kind no longer buys trap
performance, but it is still buying control.
Cross-family judge check
The same completions re-judged by gpt-5.5 instead of claude-opus-5 (v3 checkpoint):
base 19.4% → 22.6%, tuned 38.7% → 32.3%. The second family is kinder to the base model and
harsher to the tuned one, so it narrows the measured gain — the improvement is not an
artifact of a friendly judge. Per-item agreement was 96.8% on base and 80.6% on tuned.
Training
Data: 1496 judge-gated examples distilled from claude-opus-5 grounded in the official
FIP rulebook. Single-paragraph rule-first answers, median 68 words.
Dataset: https://huggingface.co/datasets/vevag/padel-rules-sft
Split: 1347 train / 149 valid.
MLX QLoRA, 1645 iters, batch 2, lr 1e-4. 616 s on Apple M2 Max, 5.43 GB peak.
Generation runs with enable_thinking=False. Qwen3's chat template defaults to thinking
mode; at max_tokens=400 the base model spends the whole budget inside <think> and never
answers, flooring the metric for a harness reason rather than a knowledge one. Training
targets contain no <think> blocks, so base and tuned must both run with it off or the
comparison is invalid.
--eval-set <path> accepts any JSONL with the schema
id, kind, cat, q, truth, contamination, so a held-out set drops straight in. rescore
needs no API keys and no third-party packages.
Judge variance: judge models reject a pinned temperature, so generations are
byte-reproducible but expect ±1 judged item on re-judging (±3.2 pts spec,
±11.1 control). See REVIEW.md finding 10 in the project repo.
Date: 2026-08-19. Part of a Gauntlet AI SLM-week submission; full harness, raw judge
transcripts, training logs, and every kept/rejected decision in the project repo.
