You can deploy this model and start earning money today!

ankitw497/
slm-125m-base

Name: slm-125m-base
Availability: InStock

Views

No views yet

ankitw497/slm-125m-base

A 125.8M-parameter LLaMA-architecture language model trained from scratch on a legal/financial/web data mix.

Model

12 layers, 768 hidden, 12 heads (MHA), vocab 16,384, context 1024, tied embeddings
Trained with a custom 16K byte-level BPE tokenizer (fit on the training corpus)

Data

Legal-first mix (NOT 70/20/10): case-law ~40%, SEC filings ~40%, fineweb-edu ~20%. Cleaned, deduplicated (exact + MinHash near-dup), and decontaminated against CaseHOLD/LexGLUE eval sets. See the source repo's REPLICATION_GUIDE.md for the full pipeline.

Training

2.04B train tokens, 1 epoch, seq_len 1024
1x H100, AdamW, cosine LR schedule with warmup
Final val loss: 2.3263 (perplexity 10.24)

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("ankitw497/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("ankitw497/slm-125m-base")