slm-125m-base
A 125.8M-parameter Llama-style causal LM pretrained from random
weights on a legal-first corpus. No distillation, no fine-tuning from an existing
checkpoint: the tokenizer, the corpus and the weights were all built from scratch.
Architecture
| Field | Value |
|---|
| parameters | 125,847,552 |
| layers | 12 |
| hidden size | 768 |
| attention heads | 12 (head dim 64), MHA |
| MLP | SwiGLU, inner 3072 |
| norm | RMSNorm, pre-norm |
| positions | RoPE (theta 10000) |
| context length | 1024 |
| vocab | 16,384 (byte-level BPE, trained on this corpus) |
| embeddings | tied (input = output) |
Because embeddings are tied, model.safetensors contains no lm_head.weight — it is
shared with the input embedding. That is expected, not a truncated upload.
Training data
Legal-first mix, cleaned and deduplicated from scratch (~40/40/20):
| Source | Role |
|---|
HFforLegal/case-law (US) | US case law, strict OCR gate |
PleIAs/SEC | SEC filings |
HuggingFaceFW/fineweb-edu | general web, kept as a fluency floor |
Pipeline: quality/language/OCR filtering, MinHash near-dedup, exact dedup, and 13-gram
decontamination against the CaseHOLD benchmark via LexGLUE's case_hold config, which is held
out. That removed 24,002 case-law and 175 SEC documents.
Result: 2.04B training tokens packed into 1024-token windows.
Contamination caveat (measured, please read before benchmarking)
Decontamination used LexGLUE's case_hold config (480,908 13-grams). A later audit found the
standalone casehold/casehold dataset contributes 796,649 additional 13-grams that were
never screened — LexGLUE covers only 38% of them. 5.1% of the retained corpus matches at
least one (case-law 5.69%, SEC 2.35%).
Treat that 5.1% as an upper bound rather than a leakage rate: SEC filings should contain no
CaseHOLD case-law text, so their 2.35% is largely innocent collision on legal boilerplate, and
a 13-gram overlap in a citing context is not the same as leaking CaseHOLD's multiple-choice
answer. Still, do not quote a CaseHOLD score for this model and treat any LexGLUE
case_hold number as optimistic. The honest metric for this model is held-out perplexity on
its own validation split, which is what is reported above.
Training
| Knob | Value |
|---|
| tokens seen | 2.04B (1 epoch) |
| steps | 3,890 |
| hardware | 8x H100, single node, DDP |
| wall clock / cost | ~13.4 min, $7.05 |
| optimizer | AdamW (0.9, 0.95), wd 0.1 |
| LR | 0.0006 -> 6e-05, cosine |
| batch | ~524K tokens/step |
| precision | bf16 |
Held-out validation perplexity: 10.42.
Usage
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3tok = AutoTokenizer.from_pretrained("narendraalluri/slm-125m-base")
4model = AutoModelForCausalLM.from_pretrained("narendraalluri/slm-125m-base")
5
6ids = tok("The plaintiff alleges that the defendant", return_tensors="pt").input_ids
7out = model.generate(ids, max_new_tokens=80, min_new_tokens=40, do_sample=True,
8 temperature=0.8, top_p=0.95, top_k=50, repetition_penalty=1.1)
9print(tok.decode(out[0], skip_special_tokens=True))
Pass min_new_tokens — a base model at this scale will happily emit EOS immediately.
Honest limitations
This is a base completer, not an assistant. It has never seen instruction data or a
chat template (the tokenizer carries <|user|>/<|assistant|> tokens, but training never
used them), so it will not follow instructions or answer questions.
It writes fluent legal and financial prose with correct register and citation form, and
it invents the content: case names, reporter numbers and dollar figures are plausible
fabrications. Perplexity is the honest metric here; factual grounding needs retrieval.
Do not use it for legal advice.
