A 517.8M-parameter Llama-style decoder, pretrained from scratch (fresh weights, fresh tokenizer) on a legal/financial corpus. This is a base model: it completes text, it does not answer questions.

Architecture


parameters	517,795,840
layers	24
hidden size	1280
attention heads	20 (head dim 64, MHA)
MLP	SwiGLU, inner 3456
norm	RMSNorm, pre-norm
positions	RoPE (theta 10000)
context length	1024
vocab	32,768 (byte-level BPE, trained on this corpus)
embeddings	tied
precision	bf16

Training

Data: thesreedath/slm-pretraining-corpus (~2.19B unique tokens; US case law, SEC filings, fineweb-edu), cleaned, MinHash-deduplicated, and decontaminated against CaseHOLD/LexGLUE.
Recipe: AdamW (0.9, 0.95), wd 0.1, LR 0.0003 -> 3e-05 cosine, 200M-token warmup, global batch 524,288 tokens, grad clip 1.0.
Hardware: 8x B200, DDP, torch.compile.
Epochs: n/a | tokens seen: 0.0B

Results

metric	value
validation perplexity	7.912
bits per byte	n/a

Read bits-per-byte, not perplexity, when comparing this to a model with a different tokenizer. Perplexity is per-token: a larger vocabulary packs more text into each token, making each prediction harder and the perplexity higher, for a model that is doing better. We measured exactly that during this run: at step 3000 this model's perplexity (11.51) looked worse than the 125M's (11.33) while its bits-per-byte (0.694) was 3.9% better. Bits-per-byte is tokenizer-invariant and is the honest cross-model number.

What this model is not

A base model is a completer. Give it the start of a passage and it continues in-register. It does not follow instructions, and it does not reliably know facts: at this scale, knowledge capacity is roughly 2 bits per parameter. For question answering, use the fine-tuned variants; for factual reliability, use retrieval.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("thesreedath/slm-500m-base")
model = AutoModelForCausalLM.from_pretrained("thesreedath/slm-500m-base")

ids = tok("The plaintiff alleges that the defendant", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=60, do_sample=True, top_k=50,
                     temperature=0.8, min_new_tokens=40)
print(tok.decode(out[0], skip_special_tokens=True))

min_new_tokens matters: a base model will otherwise sometimes emit EOS immediately.