slm-125m-legal-sft-v2
A 125M-parameter Llama-architecture model, supervised fine-tuned from
abhishekai/slm-125m-legal-base
for
grounded question answering over legal and financial passages.
v2 replaces
slm-125m-legal-sft.
The dataset was rebuilt: 17,170 training pairs instead of 9,096, balanced so no single source
document dominates, with a deliberately harder held-out set.
The honest result: v2 did not measurably improve this model
| score /10 |
|---|
| v1 SFT | 4.05 |
| v2 SFT | 4.13 |
Paired difference +0.09, 95% CI [−0.22, +0.39] — the interval contains zero, so no
improvement is established. The same dataset moved the 500M by +0.39 and Gemma-2-2B by +0.92.
That is a capacity signal, and an unusually clean one: all three models saw identical data
through identical code, and only the smallest failed to move. It also early-stopped after less
than a third of one pass, where the 500M ran more than twice as far. Roughly 4.3% of its
output collapses into repetition, so part of the ceiling is generation quality rather than
comprehension.
v2 is published because it is the checkpoint the rest of the pipeline builds on, and because a
negative result measured properly is worth more than an unmeasured positive one.
How it was trained
Q&A pairs written by Gemini 2.5 Flash over a legal/financial corpus, then filtered by a
grounding self-check, embedding dedup, length and format gates, per-document balancing, and
decontamination against the eval split.
3 epochs configured; early-stopped at step 1,000 of 3,183, best weights restored before saving.
Validation loss 1.1415. 1×H100, $0.58.
The validation split is carved from the training data. Earlier runs validated on the
judge's eval set, which meant selecting the checkpoint by looking at the exam — their published
eval losses are optimistic by an unknown amount. This one is not.
Prompt format
It expects a chat template with a system prompt, and a user turn containing the question
followed by Context: and the passage. Use tokenizer.apply_chat_template.
How it was evaluated
Claude Sonnet as an independent judge — a different model family from the Gemini 2.5 Flash
that wrote the training data, which breaks the circularity that makes a teacher-judged score
partly self-congratulation. Four axes out of 10 (question answering, instruction following,
grounding in the supplied passage, appropriate refusal), scored against the prompt, the answer,
a golden answer and the corpus evidence. The same 300 held-out prompts score every model in this
project, so the numbers are comparable across models — which win-rates against each model's own
baseline never were.
The judge was itself gated before use: it had to reproduce a known ordering (base Gemma-2-2B
above a broken fine-tune of it) before any of its scores were trusted. It did, at p = 0.0022.
Limitations
- It answers from a passage you supply. Remove the context and it confabulates fluently.
This is a reading-comprehension model for a retrieval pipeline, not a knowledge store.
- ~4.3% of outputs degenerate into repetition — the highest rate of any model in this project.
- No arithmetic reliability. It will state figures and a percentage change that do not follow
from each other. Expected at this scale.
- It invents citations. It reproduces the register of judicial opinions convincingly, so
invented case names and holdings look plausible.
- Not legal or financial advice. It is a demonstration of method.
- One judge, one eval set, n=300. The eval supplies the passage, which favours this style of
model. No labelled benchmark (CaseHOLD, LexGLUE) has been run, so nothing here says the model
is good in absolute terms — only how it compares to its own previous version.
