An instruction-tuned ~125.8M Llama-style model for legal / financial text. Supervised fine-tuned from a from-scratch base (Sarath569/slm-125m-legal-base, further pretrained to 4 epochs over ~2.04B tokens of US case law + SEC filings + educational web; base val perplexity 9.66) on a blend of general and domain instructions.

Chat format

The model uses the reserved chat tokens. Format each turn as:

<|user|>
{your question}
<|assistant|>
{model answer}<|eos|>

SFT data (33,601 examples, 2 epochs)

databricks/databricks-dolly-15k — general instructions (~15k)
gbharti/finance-alpaca — financial Q&A (~15k)
dzunggg/legal-qa-v1 — US legal Q&A (~3.7k)

Loss on assistant-response tokens only; LR 2e-5 cosine. Sensible decoding defaults (repetition_penalty=1.3, no_repeat_ngram_size=3, temperature=0.7) are baked into generation_config.json.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Sarath569/slm-125m-legal-chat")
model = AutoModelForCausalLM.from_pretrained("Sarath569/slm-125m-legal-chat")

prompt = "<|user|>\nExplain in simple terms what a 10-K filing is.\n<|assistant|>\n"
ids = tok(prompt, add_special_tokens=False, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=160)   # decoding defaults come from generation_config
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Limitations (please read)

This is a ~125M model. After SFT it follows the chat format and produces fluent, legal/financial-flavored answers and stops appropriately, but it is not a reliable source of legal or financial facts — it hallucinates, and must not be used for legal, financial, or any real-world advice. It is best on general/common topics and weakest on precise domain facts. It is a base+SFT model with no alignment/RLHF. Intended for research and as a small fine-tuning starting point, not production use.