You can deploy this model and start earning money today!

srinivasch87/
slm125live-base

Name: slm125live-base
Availability: InStock

text-generation-inference

endpoints_compatible

Views

No views yet

SLM125 — a 125M-parameter legal/financial base language model, trained from scratch

SLM125 is a small (~125.8M parameter), decoder-only transformer pretrained from scratch on a legal- and finance-heavy corpus. It is a base model — it completes text, it is not instruction-tuned or chat-tuned. Give it a sentence to continue ("The plaintiff argued that ..."), not a question.

🔗 Try it live: SLM125 Playground

Model Details

Architecture: Llama-style decoder-only transformer (RoPE, RMSNorm, SwiGLU MLP, tied embeddings)
Parameters: ~125.8M
Hidden size: 768
Layers: 12
Attention heads: 12 (head dim 64, standard multi-head attention — no GQA)
Intermediate (MLP) size: 3,072
Context length: 1,024 tokens
Vocabulary: 16,384 tokens (custom-trained tokenizer)
RoPE theta: 10,000
Precision: trained and served in fp32/bf16 on Modal

Training Data

Pretrained on a cleaned, deduplicated, decontaminated corpus of ~2.19B tokens, mixed "legal-first" from three public, streamed HuggingFace datasets:

Source	Dataset	Share	What it is
case-law	`HFforLegal/case-law` (`us` config)	~39% (~863M tokens)	US court opinions (scanned; some OCR noise)
sec	`PleIAs/SEC`	~39% (~861M tokens)	SEC filings (10-K, etc.), born-digital
fineweb-edu	`HuggingFaceFW/fineweb-edu` (`sample-10BT`)	~21% (~465M tokens)	General educational web text, added as fluency filler

The two legal sources were taken in full (they cap out around 2B tokens combined); a smaller web slice was added on top, landing at roughly a 40/40/20 split — about 78% legal/financial text overall, not the originally-planned 70/20/10.

How to Get Started

python

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "srinivasch87/slm125live-base"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

prompt = "The plaintiff argued that"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
    top_k=50,
)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Intended Use & Limitations

This is a base completion model, not an instruction-following assistant. It will not reliably answer questions, follow chat-style instructions, or refuse unsafe requests — prompt it with the start of a sentence or document.
At 125M parameters and ~2.2B training tokens, factual accuracy, coherence over long spans, and reasoning are limited. Treat all output as unverified autocomplete, not legal, financial, or professional advice.
Training data skews heavily toward US case law and SEC filings, so the model's style and any biases reflect that domain (and the OCR noise present in the scanned case-law source).
License is not specified for this model; the underlying training datasets each carry their own terms — check the dataset cards linked above before redistributing outputs.

Training & Serving Infrastructure

Built end-to-end on Modal: data cleaning, deduplication, tokenizer training, and pretraining all ran as fanned-out CPU/GPU Modal functions against a shared Modal Volume. Inference is served from a Modal web endpoint with token-streaming generation, behind a Next.js frontend deployed on Vercel (link above).