You can deploy this model and start earning money today!

Kotichitturu/
gemma-2-2b-legal-raft-v2

Name: gemma-2-2b-legal-raft-v2
Availability: InStock

Views

No views yet

Gemma 2 2B — Legal/Financial RAFT (QLoRA, merged, V2)

Fine-tuned for grounded QA over retrieved context, with refusal on US case law, SEC filings, and FineWeb-Edu (≈40/40/20 source mix, matching the pretraining distribution).

Base: google/gemma-2-2b-it
Training data: raft_15-20K (18,614 train / 991 val), ~15% refusal
Method: QLoRA (4-bit) adapter, merged into the bf16 base

Evaluation — frozen `eval_v2`

Scored on a held-out benchmark built from fresh passages (different sampling seed), decontaminated against every training set, and frozen with a sha256 receipt. Judge = claude-sonnet-5 (a different model family from both this model and the data pipeline's teacher), rubric 0–100.

Metric	Value
Judge score	38.0/100 (95% CI 35.7–40.3)
Correct (vs reference)	32.4%
Grounded	34.8%
Refusal recall (answer absent → refused)	2.0% (n=296)
Over-refusal (answer present → refused)	0.0% (n=366)
Numeric match	98.0% (n=148)
Token F1 vs reference	31.6% (n=1030)
Items scored	1326

By task type

Task	Judge /100	Token F1	n
Closed-book QA (no context — recall from weights)	17.6	12.0%	381
Instruction following (no context)	21.0	36.4%	283
RAFT (answer IS in the retrieved context — extraction)	99.0	48.3%	366
Refusal (answer is ABSENT — must decline)	5.2	—	296

Read the refusal row against the RAFT row. A high RAFT score with a low refusal score means the model answers well when the answer is present and fabricates when it is not — the two are the same behaviour measured twice.

Gold answers are model-authored (gemini-3.1-pro) from a source passage. These measure agreement with the reference, not truth.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Kotichitturu/gemma-2-2b-legal-raft-v2")
model = AutoModelForCausalLM.from_pretrained(
    "Kotichitturu/gemma-2-2b-legal-raft-v2", torch_dtype="bfloat16",
    attn_implementation="eager",   # Gemma 2 soft-capping is silently WRONG under sdpa
).eval()

SYSTEM = 'You are a legal and financial expert. Use ONLY the provided context documents to answer the question. If the answer is not in the context, reply exactly: "The context does not contain the answer."'

msgs = [{"role": "user", "content": SYSTEM + "\n\n" + "<your question>"}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt", add_special_tokens=False)
out = model.generate(**ids, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

RAFT behaviour. Give the model retrieved documents followed by the question. If the answer is not in the context it is trained to reply "The context does not contain the answer." rather than guess — that refusal is the point of the format, not a failure.

How the data was built

Teacher-generated and judge-filtered, not scraped:

Chunk the corpus into 250–350-word passages, stratified by jurisdiction / SEC section.
Generate Q&A with gpt-4o-mini, then a regex prefilter (self-containment, anchoring) removes ~20% for free before any paid judging.
Judge every pair with gemini-3.1-flash-lite for faithfulness and correctness — a different model family from the teacher, so it cannot reward its own idiom.
Filter chain of survivors (not rank-top-K): MinHash near-dup removal → per-source quota 40/40/20 → question-type cap → strata round-robin.
Split by document, so a passage's questions never straddle train and validation.

Limitations

Not legal or financial advice. Outputs can be confidently wrong.
Trained on US case law and SEC filings; other jurisdictions and document types are out of distribution.
Gold answers used for both training and evaluation are model-authored, so the reported metrics measure agreement with a reference rather than ground truth.
A 2B model merged from a 4-bit QLoRA adapter; expect degradation vs the bf16 ceiling.

Kotichitturu/gemma-2-2b-legal-raft-v2