A 125.8M-parameter Llama-style decoder-only language model pretrained from scratch on a cleaned, deduplicated, decontaminated corpus of US case law, SEC filings, and educational web text (~2.04B unique tokens; legal-first mix).

Model


Architecture	Llama (RoPE, SwiGLU, RMSNorm, tied embeddings)
Parameters	125,848,320
Layers / hidden / heads	12 / 768 / 12
Context length	1024
Vocab (byte-level BPE)	16,384
Precision	bf16

Training


Tokens seen	4.08B (2 epochs over 2.04B)
Optimizer steps	7,778
Optimizer	AdamW (β=0.9/0.95, wd=0.1, grad-clip=1.0)
LR schedule	cosine 6e-4 → 6e-5, 200M-token warmup
Hardware	8× H100 (data-parallel)
Final val loss / ppl	2.3307 / 10.29

Data mix (realized)

case-law (HFforLegal/case-law) — ~722M tokens (35%)
sec (PleIAs/SEC) — ~869M tokens (42%)
fineweb-edu (HuggingFaceFW/fineweb-edu) — ~469M tokens (23%)

Decontaminated against CaseHOLD / LexGLUE (case_hold) with 13-gram overlap filtering.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("Sarath569/slm-125m-legal-base")
model = AutoModelForCausalLM.from_pretrained("Sarath569/slm-125m-legal-base", torch_dtype=torch.bfloat16)

ids = tok("The plaintiff shall bear the burden of", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=60, do_sample=True, top_k=50, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))

Limitations

This is a small (~126M) base model, not instruction-tuned or aligned. It will produce fluent but unreliable text, can hallucinate legal/financial facts, and must not be used for legal or financial advice. Reserved chat tokens (<|user|>, <|assistant|>, <|system|>) are present in the tokenizer for downstream fine-tuning.