google/gemma-2-2b-it fine-tuned with 4-bit QLoRA for grounded question answering
over US case law and SEC filings. Merged FP16 weights, usable directly with transformers.
Read this first
This model answers only from a passage you supply. It is not a
closed-book legal model — ask it a question without source text and it will
invent an answer.
Results
Held-out validation, 1,003 examples, greedy decoding. Every row uses the
same metrics, so they are directly comparable:
Model
Token F1
Exact
F1 ≥ 0.5
F1 < 0.2
Numeric fidelity
125M trained from scratch, full FT
0.590
11.4%
60.1%
10.5%
98.1%
gemma-2-2b-it zero-shot
0.720
12.4%
82.2%
2.9%
97.3%
This model (QLoRA 4-bit)
0.856
36.0%
94.9%
0.4%
99.8%
On what the fine-tune actually bought. Token F1 is scored against reference
answers written by the teacher model, so some of that gain is the model
learning the teacher's phrasing rather than getting more answers right — the
exact-match jump in particular. The honest signal is numeric fidelity,
which checks every number the model emits against the passage rather than
the reference and so cannot be gamed by style: it rose
97.3% → 99.8%, and clearly-wrong
answers fell 2.9% → 0.4%. That part is real.
Measured at identical batch size, 4-bit peaked at 25.3 GB against bf16's
31.0 GB — an 18% saving, not the 8× the "4-bit" label suggests. The reason
is that bitsandbytes quantizes nn.Linear but not nn.Embedding, and Gemma 2's
256,128-token vocabulary makes the embedding table 22.6% of all
parameters, which stays in bf16:
Training used bitsandbytes NF4. It is CUDA-only and cannot run on CPU.
Serving uses GGUF Q4_K_M via llama.cpp, a separate quantization of the
merged weights. That is what the GGUF repo contains.
An adapter trained against a 4-bit base does not merge losslessly into FP16 —
it partly learned to correct quantization error that is absent after merging.
Gotchas if you fine-tune Gemma 2 yourself
Use attn_implementation="eager". FlashAttention-2 does not implement
Gemma 2's logit soft-capping and fails silently — you get a plausible loss
curve on a model that is not Gemma 2.
Watch the logits tensor, not the weights. With a 256,128 vocab, cross-entropy
at batch 32 × seq 1024 is 33.6 GB of logits alone and OOMs an 80 GB card before
any weight loads. Quantization does not help; a smaller micro-batch does.
Gemma 2's chat template has no system role and raises if given one. Fold
the system prompt into the first user turn — in training and inference alike.
Limitations
Requires the source passage; no useful closed-book recall.
~0% of answers are still clearly wrong.
Training data is synthetic, from a single teacher, and inherits its biases.
