You can deploy this model and start earning money today!

rahulreddyhanu/
slm-125m-legal-financial

Name: slm-125m-legal-financial
Availability: InStock

Views

No views yet

slm-125m-legal-financial

A 125.8M-parameter Llama-architecture language model pretrained from scratch on a legal- and financial-domain corpus. Not fine-tuned from any existing base model — tokenizer and weights are both trained from zero.

Model details

Architecture: Llama (tied embeddings), 12 layers, 768 hidden, 12 heads (MHA), SwiGLU MLP (3072 inner)
Params: 125,847,552 (~125.8M)
Context length: 1024 tokens
Tokenizer: byte-level BPE, 16,384 vocab, trained on the same corpus
Precision: bf16

Training data

Streamed, cleaned, deduplicated, and decontaminated from three public HuggingFace datasets:

Source	HF dataset	Tokens	Share
Case law	`HFforLegal/case-law` (US)	~715M	~35%
SEC filings	`PleIAs/SEC`	~860M	~42%
Web (educational)	`HuggingFaceFW/fineweb-edu`	~464M	~23%
Total		~2.04B

Legal-first mix (not 70/20/10 — the two legal sources only hold ~2B clean tokens total, so all of both were taken and a small web slice added for fluency).

Cleaning: line-level noise/boilerplate filtering, repetition filter, English-language gate, OCR-garble gate (case-law only, dictionary-based). Dedup: MinHash near-dup removal (case-law) + exact-hash dedup (all sources). Decontamination: documents overlapping 13-grams from casehold/LexGLUE case_hold eval sets were stripped, so downstream legal-QA benchmarks (e.g. CaseHOLD) are not contaminated.

Training

1 epoch over the full 2.04B-token corpus (~17 tok/param)
8x H100, DDP via torchrun, bf16, torch.compile, flash-SDPA, fused AdamW
Global batch 524,288 tokens, cosine LR schedule (peak 6e-4), 3,889 steps, ~11.4 min wall-clock
Final train loss: 2.39

Evaluation

Held-out validation perplexity (1% split, never trained on):

Domain	Val perplexity
SEC (financial)	6.32
Case law (legal)	11.21
Web (fineweb-edu)	29.00
Overall	12.71

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("rahulreddyhanu/slm-125m-legal-financial")
model = AutoModelForCausalLM.from_pretrained("rahulreddyhanu/slm-125m-legal-financial")

ids = tok("The plaintiff shall bear the burden of proving", return_tensors="pt")
ids.pop("token_type_ids", None)
out = model.generate(**ids, max_new_tokens=60, do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))

Limitations

Small model (125M) trained for a single epoch — general-purpose fluency and factual reliability are limited; this is a domain-flavored base model, not an instruction-tuned assistant.
English only. Legal text is US case law specifically.
No RLHF/instruction tuning applied.