slm-500m-legal-sft-v2
A 518M-parameter model, supervised fine-tuned from
thesreedath/slm-500m-base for
grounded
question answering over legal and financial passages.
v2 replaces
slm-500m-legal-sft,
retrained on a rebuilt dataset: 17,170 pairs instead of 9,178, document-balanced, with a harder
held-out set.
Result
| score /10 |
|---|
| v1 SFT | 4.65 |
| v2 SFT | 5.04 |
Paired difference +0.39, bootstrap 95% CI [+0.05, +0.72] — excludes zero, so the
improvement is established. But the lower bound is +0.05 and the gain is small. A sign test
over the same pairs gives p = 0.15 and does not reject; the two disagree because the sign test
discards ties. Read the effect as real and modest.
This is the middle point of a clean scale trend — the same dataset moved the 125M by +0.09 (not
significant) and Gemma-2-2B by +0.92. Better data helps in proportion to model size.
How it was trained
Same curated Q&A set as the 125M, re-tokenized against this model's own 32,768-entry vocabulary.
The curated data is stored as raw text precisely so it re-tokenizes cleanly against whatever
tokenizer a base model ships.
3 epochs configured; early-stopped at step 2,100 of 6,417, best weights restored before saving.
Validation loss 1.0660. 1×H100, $0.54.
The validation split is carved from the training data. Earlier runs validated on the judge's
eval set — selecting the checkpoint by looking at the exam — so their published eval losses are
optimistic by an unknown amount. This one is not.
A note on the upstream base
thesreedath/slm-500m-base ships no chat template, and its config.json claims
eos_token_id=2 where its tokenizer resolves <|pad|>=2. Left alone, generation stops on the
padding token and never on the real one — silently. Both are fixed in this repo: a chat template
is attached and the bos/eos/pad ids are rewritten from the token strings.
How it was evaluated
Claude Sonnet as an independent judge — a different model family from the Gemini 2.5 Flash
that wrote the training data, which breaks the circularity that makes a teacher-judged score
partly self-congratulation. Four axes out of 10 (question answering, instruction following,
grounding in the supplied passage, appropriate refusal), scored against the prompt, the answer,
a golden answer and the corpus evidence. The same 300 held-out prompts score every model in this
project, so the numbers are comparable across models.
The judge was itself gated before use: it had to reproduce a known ordering (base Gemma-2-2B
above a broken fine-tune of it) before any of its scores were trusted. It did, at p = 0.0022.
Limitations
- It answers from a passage you supply. Remove the context and it confabulates fluently.
This is a reading-comprehension model for a retrieval pipeline, not a knowledge store.
- ~3% of outputs degenerate into repetition.
- No arithmetic reliability. Expected at this scale.
- It invents citations convincingly enough to be dangerous.
- Not legal or financial advice. A demonstration of method.
- One judge, one eval set, n=300. The eval supplies the passage, which favours this style of
model. No labelled benchmark (CaseHOLD, LexGLUE) has been run, so nothing here says the model
is good in absolute terms.
