slm-125m-base
A 125.8M-parameter Llama-architecture language model pretrained from scratch
on a legal/financial corpus. No weights or tokenizer were inherited from any
existing model.
Model
| |
|---|
| params | 125,847,552 (125.8M, tied embeddings) |
| layers / d_model / heads | 12 / 768 / 12 (MHA) |
| context length | 1024 |
| vocab | 16384 (byte-level BPE, trained from scratch) |
| activation / norm | SwiGLU / RMSNorm, RoPE |
Training data (2.04B tokens)
Streamed from HuggingFace, cleaned, deduplicated and decontaminated:
| source | role |
|---|
HFforLegal/case-law | US court opinions |
PleIAs/SEC | SEC filings (10-K etc.) |
HuggingFaceFW/fineweb-edu | general educational web text |
Realized mix is roughly 40/40/20 (case-law / SEC / web), i.e. ~78% legal. The
legal sources together contain only ~2B clean tokens, so a legal-heavy mix at a
larger scale is not achievable with these corpora.
Pipeline: 6-step deterministic cleaning (line filters, boilerplate strip,
repetition, language ID, OCR-garble gate on scanned case law) -> MinHash LSH
near-dedup + exact dedup -> 13-gram decontamination against the CaseHOLD /
LexGLUE case_hold evaluation set -> 16K byte-level BPE -> packed uint16
1024-token windows, 99/1 train/val split.
Training
-
2 epochs over the fixed corpus (~4.08B tokens seen)
-
8x H100, DDP, bf16 autocast, torch.compile, fused AdamW, SDPA attention
-
global batch 524,288 tokens, cosine LR 0.0006 -> 6e-05, 200M warmup tokens
-
measured throughput ~3.19M tok/s (35% MFU)
-
final val loss: 2.2083
-
final val perplexity: 9.10
Intended use and limitations
A small base model for research on legal/financial text. It is not
instruction-tuned, has no safety alignment, and will produce fluent but
unreliable legal or financial statements. Do not use it for legal or financial
advice. Case-law inputs include OCR noise; SEC filings skew to older filings.
The corpus is decontaminated against CaseHOLD, so CaseHOLD evaluation is
meaningful; no other benchmark was held out.
