An instruction-tuned 125M legal/financial model: thesreedath/slm-125m-base fine-tuned on a synthetic, LLM-judged, decontaminated grounded-QA dataset.

The base model is a completer. This one follows instructions: it answers questions grounded in a supplied context, and refuses ("That is not stated in the context.") when the answer is not present.

Results (held-out validation)

	Base	After QA SFT
Loss	2.61	0.69
Perplexity	13.61	2.00

Chat template (required)

Trained on one exact format, shipped in tokenizer_config.json:

<|bos|><|system|>SYSTEM<|user|>USER<|assistant|>ANSWER<|eos|>

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Sudhanshu1985/slm-125m-sft")
model = AutoModelForCausalLM.from_pretrained("Sudhanshu1985/slm-125m-sft")

msgs = [
    {"role": "system", "content": "You are a legal and financial assistant. Answer only from the provided context. If the answer is not in the context, say so."},
    {"role": "user", "content": "Context:\n<your passage>\n\nWhat date was suit filed?"},
]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
enc = tok(text, return_tensors="pt", add_special_tokens=False)
enc = {k: v for k, v in enc.items() if k in ("input_ids", "attention_mask")}
print(tok.decode(model.generate(**enc, max_new_tokens=90)[0][enc["input_ids"].shape[1]:],
                 skip_special_tokens=True))

Training data

24,713 grounded-QA pairs (23,204 train / 474 val after filtering to the 1024-token window), synthesized from a cleaned legal/financial + web corpus and filtered by an LLM judge (1-5 rubric for correctness + grounding; kept >= 4):

Source	Pairs	Share
US case law (`HFforLegal/case-law`)	13,005	52.6%
SEC filings (`PleIAs/SEC`)	9,309	37.7%
Educational web (`HuggingFaceFW/fineweb-edu`)	2,399	9.7%

Task types: lookup, reasoning, and unanswerable (refusals). ~32% of the pairs are unanswerable, which directly trains reliable refusal behavior.

Recipe: 3 epochs, 1xH100, lr 2e-5 cosine, AdamW, bf16, effective batch 32, loss on the assistant span only.

Behaviour

Prompt type	Example output
lookup	"The plaintiff filed suit on March 14, 1994..."
reasoning	"...because DeWitt failed to establish that Northstar had actual or constructive notice of the defect."
unanswerable	"That is not stated in the context."

Limitations

It knows no facts of its own; it only works over context you supply.
Domain-biased toward US legal/financial register.
1024-token context window; longer inputs must be chunked.
Not legal or financial advice.