A 125M-parameter Llama-style base language model pretrained from scratch — tokenizer, data pipeline, and training loop included — on a legal/financial corpus (US case law + SEC filings + a web slice, ~2.19B tokens).

🎮 Try it in your browser (WebGPU, no install): slm-125m-playground.vercel.app — the int8 ONNX export runs fully client-side via 🤗 Transformers.js (~139 MB one-time download, then cached/offline).


Parameters	≈125.8M (tied embeddings)
Training tokens	~2.19B (≈17.5 tokens/param — approximately Chinchilla-optimal)
Held-out validation perplexity	11.01
Context length	1,024 tokens
Precision	Trained in bf16 · int8 (q8) ONNX export for browser

This is a base completer, not a chat model. Prompt it with the start of a sentence and it continues fluently in the legal/financial register. It does not answer questions, follow instructions, or reliably state facts — at this size, world knowledge is capped at roughly ~2 bits/param. Instruction-tuned (SFT) and preference-tuned variants are on the roadmap below.

Quickstart

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("genaiquest/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("genaiquest/slm-125m-base")

ids = tok("The plaintiff alleges that the defendant", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=60, do_sample=True,
      temperature=0.8, min_new_tokens=40, repetition_penalty=1.3)[0]))

Suggested sampling: temperature 0.8 · top_p 0.95 · repetition_penalty 1.3 (small base models loop without a repetition penalty).

Prompts that work well (in-register continuations):

IN THE UNITED STATES DISTRICT COURT
Pursuant to Section 10(b) of the Securities
The Company's revenue for the fiscal year
The court finds that the evidence presented

Architecture

Deliberately vanilla geometry, mapping 1:1 to transformers.LlamaConfig — so it exports to ONNX (and GGUF) with zero custom-converter work. That design choice is exactly what makes the in-browser playground possible.

Component	Spec
Family	Llama-style decoder-only
Hidden size	768
Layers	12
Attention heads	12 (head dim 64)
KV heads	12 — MHA, `num_kv == num_heads` → export-safe
MLP	SwiGLU, intermediate 3072 (4×), SiLU gate
Normalization	RMSNorm (eps 1e-5), pre-norm
Positional encoding	RoPE, θ = 10,000
Vocabulary	16,384
Embeddings	Tied (input embedding == LM head)
Attention bias	False

Parameter budget (tied):

embed      16,384 × 768        = 12.58M
attn/layer 4 × 768²            =  2.36M
mlp/layer  3 × 768 × 3072      =  7.08M
(attn + mlp) × 12 layers       = 113.2M
total                          ≈ 125.8M

Training

Data: US case law + SEC filings + a general web slice, deduplicated; ~2.19B tokens total. Case law and SEC filings are public-domain US government works. [40%-40%-20%]
Tokenizer: 16,384-token BPE trained from scratch on the corpus.
Compute: 1 epoch on 8×H100. [FILL: wall-clock hours, tokens/sec, approx cost — e.g., "X hrs ≈ $Y at market spot rates"]
Recipe: optimizer (AdamW), peak LR, schedule (cosine), 200M token warmup, global batch size 32, 0.5M tokens/step, grad clip, weight decay
Scaling note: 2.19B tokens ÷ 125.8M params ≈ 17.5 tokens/param, close to the Chinchilla compute-optimal ratio (~20) — this run prioritized compute-optimality over deliberate overtraining.

Loss curve

Evaluation

Metric	Value
Validation perplexity (held-out, in-domain)	11.01

Perplexity is measured on a held-out slice of the training distribution (legal/financial register). Expect materially higher perplexity on out-of-domain text; this model is a domain specialist by construction.

Intended use & limitations

Intended: studying small-model pretraining end to end; domain-register text continuation; a base checkpoint for SFT/DPO/RAG experiments; edge/browser inference demos; teaching material for the full train→quantize→deploy pipeline.

Not intended: legal or financial advice, factual question answering, chat, or any production decision-making. Outputs can be fluent and wrong — at 125M parameters that is a certainty, not a risk. Generated text may echo the style of court filings and SEC documents; verify nothing here as fact.

ONNX / browser deployment

The repo includes an int8 (q8) ONNX export consumed by 🤗 Transformers.js in the playground. Because the architecture maps 1:1 to LlamaConfig, the export pipeline is: bf16 safetensors → ONNX → dynamic int8 quantization, with no custom conversion code.

Roadmap

This is checkpoint 1 of a series building the full enterprise-SLM stack in public:

✅ slm-125m-base — pretraining from scratch (this model)
⏳ -sft — supervised fine-tuning for instruction following / QnA
⏳ -dpo/ppo — preference tuning
⏳ RAG-grounded variants for document QnA with citations
⏳ Serving optimization study on this model family (vLLM, CUDA graphs, FP8) — companion to my vLLM H100 serving experiments

Author

Bhabani Nayak — AI engineering leader; agentic AI at consumer scale (Sephora; previously Shopify, Twitter, Apple). IEEE Senior Member. LinkedIn · Substack · Playground

Citation

bibtex

@misc{nayak2026slm125m,
  author = {Nayak, Bhabani},
  title  = {slm-125m-base: a 125M Llama-style language model pretrained from scratch on legal/financial text},
  year   = {2026},
  url    = {https://huggingface.co/genaiquest/slm-125m-base}
}