Answers legal/financial questions only from a context passage you supply, and
refuses when the answer isn't in it. QLoRA fine-tune on grounded data where 20% of
training examples had no answer in the context — that is what teaches "I don't know".
This is the deployable one. Its closed-book sibling
(Kotichitturu/gemma-2-2b-legal-qa)
answers from memory and invents facts; this model reads.
Measured on 485 held-out rows (greedy)
Refusal — measured as a confusion matrix, because refusal rate alone is gameable
(a model that refuses everything scores 100% recall and is useless):
answer ABSENT → refused
100.0% (91/91)
answer PRESENT → refused (over-refusal)
0.3%
overall refusal accuracy
99.8%
Extraction — does it get the answer RIGHT? (a separate question from refusing well)
Metric
Result
Single-figure accuracy (one figure asked, one given)
79.6% (95% CI 67.1–88.2, n=54)
Numeric match (all gold numbers reproduced)
81.5%
Token F1 vs gold
73.2%
Exact match (normalised)
14.2%
Emitted no number at all
9.8%
The finding: capacity, not data
Our from-scratch 125M RAFT model, trained on the same dataset with the same
scorers and the same val split, gets a single figure right 37.0% of the time.
This model gets it right 79.6% — and the 95% confidence interval (67.1–88.2)
sits entirely above 37%.
125M RAFT
This model
Single-figure extraction
37.0%
79.6%
Token F1
46.3%
73.2%
Emitted no number
24.3%
9.8%
So the 125M's 37% was a capacity limit, not a data limit — the dataset could support
a good reader all along; the small model simply couldn't extract. Note what this does
not mean: the same architecture answering closed-book is still only 14.2% correct.
Scale buys reading comprehension, not stored facts.Skills in the weights, facts in
the retriever.
⚠️ Not a clean "scale alone" experiment. Gemma-2B-it is already instruction-tuned
by Google on vastly more data than our 125M base ever saw, so this measures "can this
dataset produce a good reader?" (yes) rather than "what do parameters alone buy?"
Gold answers are teacher-generated and judge-filtered — a proxy for truth, not truth.
Limits
It still misses ~1 figure in 5. Check figures that matter against the passage.
Retrieval is your job. This is the reader, not the retriever.
Context + question must fit comfortably inside Gemma 2's 8,192-token window.
Use greedy decoding — every number above was measured with do_sample=False; sampling
degrades refusal behaviour. attn_implementation="eager" is required (Gemma 2 soft-capping).
When the answer is absent it emits:
The answer is not available in the provided context.
Architecture
Base google/gemma-2-2b-it (unchanged shape) with LoRA adapters merged in.
QLoRA (4-bit), lr 2e-4 cosine, effective batch 16 (micro 2 × accum 8), eager
attention (Gemma 2 soft-capping), 3 epochs with early-stopping on val loss.
Data: 8,000 legal/financial passages (US case law + SEC filings), teacher-generated
and LLM-judge-filtered (the same plain-text dataset used for our 125M models —
tokenizer-agnostic by design).
Loss masked to the model's answer turn only; the boundary is found by searching the
<start_of_turn>model token ids (verified 200/200).
