SocraTeach vanilla SFT (run 2 of 2) — LoRA adapter for OLMo-2-1B-Instruct
Plain supervised fine-tuning on Socratic tutoring dialogues — no KL reweighting. This is the
second of two runs of the same recipe, uploaded specifically so the pair can be used to
measure the pedagogy judge's noise floor against training noise.
- Step 923 (1 epoch) · LoRA r=16, alpha=32, dropout=0.05, 7 projections
- Base:
allenai/OLMo-2-0425-1B-Instruct
- Data:
meric533/socrateach-sft (revision v1) — a later v2 revision drops 225 dialogues where the tutor affirms wrong student arithmetic; this adapter was trained before that fix, so use revision="v1" to reproduce it. See the dataset's MATH_ERRATA.md.
The calibration pair
| run 1 | run 2 (this repo) |
|---|
| repo | socrateach-sft-olmo2-1b-lora | socrateach-sft-olmo2-1b-lora-run2 |
| judge OVERALL (no SI, deduped) | 0.685 | 0.527 |
| GSM8K bare | 0.468 | 0.456 |
| GSM8K hinted | 0.208 | 0.212 |
Same recipe, independently trained (run 1 comes from the earlier POC). They are genuinely
different weights, not a re-upload of one checkpoint. Note how tightly the math numbers
agree — 0.012 and 0.004 apart — compared to the judge gap below.
Why this pair matters
Two noise floors, both paired bootstraps over the same held-out contexts:
- Judge round-to-round — re-judging the same generations in a fresh round:
+0.002, 95% CI [-0.027, +0.045]. Indistinguishable from zero.
- Retraining — this pair, one judging round: +0.157, 95% CI [+0.018, +0.310].
Excludes zero.
The judge is reproducible; the training is not. A pedagogy gap between two methods that is
smaller than roughly 0.15 at one seed per arm is inside retraining noise, and re-judging will
not resolve it — only more seeds will.
Per-response, the judge is noisier than the aggregate suggests. Because duplicate contexts
produced byte-identical generations, 36 of them got scored more than once within a single
round: mean spread 0.080, exactly reproduced in only 10 of 36 cases, worst case 0.188. That
per-response noise averages out over contexts, which is why the aggregate round-to-round
delta is ~0.002.
Sample sizes this implies
Contexts needed for 80% power at fixed weights (paired-difference sd 0.273):
| gap to detect | contexts |
|---|
| 0.15 | ~27 |
| 0.10 | ~59 |
| 0.05 | ~235 |
| 0.034 | ~507 |
And across seeds, with a between-run sd of ~0.111, the smallest method gap resolvable at
1 seed per arm is ~0.44; at 5 seeds ~0.20; at 10 seeds ~0.14.
Caveats on the numbers above
Every judge number here was produced with no system instruction in context —
gen_pedagogy.py's --si flag defaults to none. The POC's headline SFT score (0.84) is
the +SI condition and is not comparable. A +SI re-judge is pending.
The sample is 12 distinct problems, not the 40 rows the files appear to contain: the held-out
set stores several dialogues per problem and greedy decoding makes the copies byte-identical.
All values here are deduped to the 12.
Usage
1from peft import PeftModel
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4base = AutoModelForCausalLM.from_pretrained("allenai/OLMo-2-0425-1B-Instruct")
5tok = AutoTokenizer.from_pretrained("allenai/OLMo-2-0425-1B-Instruct")
6model = PeftModel.from_pretrained(base, "meric533/socrateach-sft-olmo2-1b-lora-run2")
Provenance
Trained on 1x H200 (MIT ORCD). Code:
edu-llm/OLMo-core, branch
p7/impl3.
