A 125.8M-parameter grounded legal/financial Q&A model, preference-tuned with DPO (Direct Preference Optimization) on AI-generated feedback (RLAIF).

Honest headline

This model is, if anything, slightly worse than the SFT model it was trained from. Use slm-125m-legal-sft instead.


win-rate vs its SFT baseline	0.455
tally	37 wins / 46 losses / 17 ties (n=100)
two-sided sign test on the 83 decisive pairs	p = 0.380
95% CI on the overall win-rate	[0.361, 0.552]

The interval straddles 0.5, so the apparent degradation is not statistically established either — the honest reading is "no measurable gain, with the point estimate on the wrong side." It is published as a reproducible RLAIF artifact and as one point in a scaling comparison, not as an upgrade.

Independent absolute score (2026-08-15)

Scored by Claude Sonnet on four axes out of 10 — a different model family from the Gemini judge behind the win-rate above, and one that had no hand in writing this project's training data. 300 held-out prompts, the same prompts and the same scale used for all twelve checkpoints, so this number is comparable across models in a way no win-rate here is.


mean score	3.52 / 10
95% CI (bootstrap over per-prompt scores)	[3.23, 3.82]
paired step	-0.52 against its own SFT (4.05), CI [-0.86, -0.18] - a real regression on this scale too.

The full twelve-checkpoint table, with every stage-to-stage interval, is in MODEL_INDEX.md in the project repository.

Why publish a negative result

Because it is the smallest point in a controlled 3-model comparison run through an identical pipeline — same prompt pool, same Gemini 2.5 Flash judge, same preference-mining thresholds, same DPO recipe:

model	params	DPO win-rate vs its own SFT	p
125M (this)	126M	0.455	0.380
500M	518M	0.480	0.704
Gemma-2-2B	2,614M	0.530	0.545

DPO's win-rate rises monotonically with model scale, crossing from harmful to parity to slightly positive. No individual point is statistically distinguishable from 0.5 at n=100 — the trend is the finding, not any single number. Distinguishing a 0.53 effect from parity would need roughly 400–600 eval prompts.

Why it does not help at 125M

The objectives optimized correctly — DPO preference accuracy reached 0.96 on held-out pairs, and the reward model trained from the same data reached 0.837 pairwise accuracy. This is a capacity finding, not a bug.

SFT-greedy decoding is already close to this model's ceiling. Distilling toward a Gemini-class target pushes it into a fluent prose register it does not have the capacity to fill with substance: the answers get smoother, not more correct. A variant trained on Gemini-authored chosen responses instead of on-policy ones was worse still (0.335, 29 wins / 62 losses), which is the same effect seen more sharply.

Preference data

888 triplets (788 train / 100 held-out eval), built on-policy:

Sample 4 candidate answers from the frozen SFT model (temp 0.9, top-p 0.95)
Gemini 2.5 Flash scores each 1–10 on correctness and grounding
Keep the best/worst pair only if the score gap is ≥ 2
An independent pairwise judge re-checks that chosen really beats rejected, with A/B order randomized to cancel position bias
Embedding dedup on prompts (cosine ≥ 0.97), then split

Prompt format

Use the chat template shipped with the tokenizer. The assistant turn ends on <|eos|>.

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("abhishekai/slm-125m-legal-dpo")
model = AutoModelForCausalLM.from_pretrained("abhishekai/slm-125m-legal-dpo")

msgs = [
    {"role": "system", "content": "You are a precise legal and financial assistant. Answer only from the provided context."},
    {"role": "user", "content": "Context: In a civil negligence action the plaintiff must prove duty, breach, causation, and damages by a preponderance of the evidence.\n\nQuestion: What standard of proof applies to the plaintiff?"},
]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors="pt", add_special_tokens=False),
                     max_new_tokens=128, do_sample=False)

Limitations

No arithmetic reliability. At 125M the model will state a revenue delta and a percentage that do not follow from the two figures it just produced. Do not trust generated numbers.
Grounded QA only. It is trained to answer from a supplied passage. Asked open-ended questions with no context, it confabulates.
Not legal or financial advice. Fluent, register-correct legal prose is not the same as correct legal reasoning, and this model produces the former.
The judge that created its preference data is also the judge that evaluated it (Gemini 2.5 Flash). Win-rates against an independent judge may differ.

Training

DPO, β=0.1, effective batch 16, on 1×H100. Full pipeline, configuration, and the negative results are documented in the project repository.