You can deploy this model and start earning money today!

thesreedath/
slm-125m-base

Name: slm-125m-base
Availability: InStock

Views

No views yet

slm-125m-base

A 125M-parameter Llama-style base language model, pretrained from scratch on a legal/financial corpus (US case law + SEC filings + a web slice, ~2.19B tokens).

Pretraining: 10 epochs, 8xH100. Held-out validation perplexity: 8.36.
This is a base COMPLETER, not a chat model. Prompt it with the start of a sentence and it continues in the legal register. It speaks the register fluently but does not know facts (knowledge is capped at ~2 bits/param at this size).

python

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("thesreedath/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("thesreedath/slm-125m-base")
ids = tok("The plaintiff alleges that the defendant", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=60, do_sample=True,
      temperature=0.8, min_new_tokens=40)[0]))

Full pipeline + fine-tuning code: https://github.com/Vizuara-AI-Lab/slm-125m-from-scratch