
base model	thesreedath/slm-500m-base (Apache-2.0)
base pretraining	2.08B tokens, 5 epochs, 8×B200, val ppl 7.91, bits-per-byte 0.588
base pretraining cost	$115.69, paid by the base's author — not by this fine-tune
what we added	Q&A SFT on 9,178 curated grounded pairs, 3 epochs, 1×H100
our cost	$0.91

The base was pretrained on a legal/financial corpus (~40% US case law, ~40% SEC filings, ~20% educational web text), MinHash-deduplicated and decontaminated against CaseHOLD/LexGLUE.

Independent absolute score (2026-08-15)

Scored by Claude Sonnet on four axes out of 10 — a different model family from the Gemini judge behind the win-rate above, and one that had no hand in writing this project's training data. 300 held-out prompts, the same prompts and the same scale used for all twelve checkpoints, so this number is comparable across models in a way no win-rate here is.


mean score	4.65 / 10
95% CI (bootstrap over per-prompt scores)	[4.33, 4.99]
paired step	+3.02 against the 500M pretrain base (1.63), CI [+2.68, +3.37] - the fine-tune is the largest single effect measured in this project.

The full twelve-checkpoint table, with every stage-to-stage interval, is in MODEL_INDEX.md in the project repository.

Why this exists

The base author already publishes a Q&A fine-tune, thesreedath/slm-500m-qa. On our prompt distribution it did not answer questions — it continued the source passage. Measured, not assumed:

check	`slm-500m-qa`	this model
answers a grounded question	no — continues the passage	yes
fails at greedy decoding too	yes	—
fails under 4 different prompt formats	yes	—
usable preference pairs mined from 4 samples/prompt	54 / 1175 (4.6%)	1048 / 1380 (76%)

That is not a criticism of the base or of their fine-tune — their reported QA-SFT perplexity of 5.41 is a fine number on their own Q&A data. Perplexity does not measure instruction-following on a different task mix. Ours is heavier on summarize / rewrite / list / extract over 700–3,200 character passages.

Training data

9,178 train / 799 val curated grounded Q&A pairs, generated with Gemini 2.5 Flash and filtered by: embedding dedup, teacher grounding self-check, length/format gates, task+difficulty balancing, and n-gram + embedding decontamination against the eval split. Loss is masked to the answer tokens only.

Results

metric	value
answer-token validation perplexity	3.43
training	3 epochs, 3,441 steps, 13.8 min, 1×H100
tokens / epoch	6.82M (389K supervised)

For reference, a 125M model trained on the identical data reaches 3.36 — but on a 16K vocab versus this model's 32K. A larger vocab packs more text per token and makes each prediction harder, so 3.43 here is the better result in tokenizer-invariant terms. Compare bits-per-byte, not perplexity, across different tokenizers.

Usage

The user turn must contain the passage. Use greedy or low temperature.

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

m = "abhishekai/slm-500m-legal-sft"
tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForCausalLM.from_pretrained(m).eval()

question = "What standard of proof applies to the plaintiff?"
context = ("In a civil negligence action the plaintiff must prove duty, breach, "
           "causation, and damages by a preponderance of the evidence.")
messages = [
    {"role": "system", "content": "You are a precise legal and financial assistant. "
     "Answer using only the provided context. If the context does not contain the "
     "answer, say you cannot answer from the context."},
    {"role": "user", "content": f"{question}\n\nContext:\n{context}"},
]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(text, add_special_tokens=False, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=160, do_sample=False,
                     eos_token_id=tok.convert_tokens_to_ids("<|eos|>"),
                     pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Limitations

Grounded only. Bare closed-book questions are out of distribution.
No arithmetic reliability. At this scale it will state a figure and a derived percentage that do not follow from each other. Do not trust generated numbers.
Synthetic training data. Q&A pairs were written by Gemini 2.5 Flash and inherit its biases and errors, filtered but not eliminated.
Not legal or financial advice. Outputs are not a substitute for a qualified professional.
1,024-token context.

License

Apache-2.0, inherited from thesreedath/slm-500m-base.