slm-125m-base
asinha08/slm-125m-base is a 125M parameter, Llama-compatible, decoder-only causal language model trained from scratch on a legal-first English corpus. It is a base language model, not an instruction-tuned or chat-aligned assistant.
Model Details
| Field | Value |
|---|
| Repository | asinha08/slm-125m-base |
| Project / Modal app | slm125mLIVE |
| Model type | Decoder-only causal LM, Llama-compatible |
| Parameters | 125,848,320 |
| Context length | 1,024 tokens |
| Vocabulary size | 16,384 |
| Tokenizer | Byte-level BPE trained from the final corpus |
| Hidden size | 768 |
| Layers | 12 |
| Attention heads | 12 |
| KV heads | 12 |
| Intermediate size | 3,072 |
| Activation | silu / SwiGLU |
| Positional encoding | RoPE, theta 10,000 |
| Norm | RMSNorm, eps 1e-5 |
| Embeddings | Tied input/output embeddings |
| Attention bias | false |
| Saved format | safetensors |
| Weight dtype | bfloat16 |
Special tokens:
| Token role | Token |
|---|
| BOS | <|bos|> |
| EOS | <|eos|> |
| PAD | <|pad|> |
| UNK | <|unk|> |
| Extra | <|user|>, <|assistant|>, <|system|> |
The extra chat-style tokens are present in the tokenizer, but the model was not instruction tuned.
Intended Use
This model is intended for:
- Research on small language model pretraining.
- Legal-domain language modeling experiments.
- Continued pretraining, supervised fine-tuning, or evaluation pipelines.
- Text continuation/completion tests.
This model is not intended for:
- Legal advice or compliance decisions.
- Production legal automation without task-specific evaluation and guardrails.
- Chatbot use without instruction tuning and safety tuning.
- Factual question answering where accuracy is required.
Training Data
The corpus was built from three public Hugging Face datasets with a legal-first mix:
| Source | HF dataset | Kept docs | Proxy corpus tokens | Tokenized train tokens | Tokenized val tokens | Tokenized mix |
|---|
| Case law | HFforLegal/case-law | 206,684 | 806,244,775 | 714,858,496 | 7,223,296 | 35.06% |
| SEC filings | PleIAs/SEC | 45,035 | 1,091,148,638 | 860,025,856 | 8,688,640 | 42.18% |
| Web/education | HuggingFaceFW/fineweb-edu / sample-10BT | 418,405 | 499,936,147 | 464,188,416 | 4,689,920 | 22.76% |
| Total | | 670,124 | 2,397,329,560 | 2,039,072,768 | 20,601,856 | 100.00% |
Data processing included:
- Basic line/document cleaning.
- OCR/noise filtering for case-law data.
- Exact duplicate removal.
- MinHash near-duplicate removal.
- Decontamination against held-out evaluation sources:
coastalcph/lex_glue and casehold/casehold.
- Byte-level BPE tokenizer training on the cleaned corpus.
- Packed fixed-length token windows with sequence length
1,024.
- Validation routing of every 100th packed window, giving an approximately 99/1 train/validation split.
Deduplication and decontamination summary:
| Source | Exact duplicates removed | Near duplicates removed | Contaminated docs removed |
|---|
| Case law | 0 | 1,606 | 24,002 |
| SEC filings | 1,989 | 0 | 175 |
| FineWeb-Edu | 62 | 0 | 0 |
Training Procedure
Training ran on Modal using 8x H100 GPUs.
| Field | Value |
|---|
| GPUs | 8 x H100 |
| Precision | bfloat16 autocast |
| Optimizer | AdamW |
| Learning rate | 6e-4 |
| Minimum LR | 6e-5 |
| Warmup tokens | 200,000,000 |
| Weight decay | 0.1 |
| Betas | (0.9, 0.95) |
| Gradient clipping | 1.0 |
| Micro batch size | 64 windows per GPU |
| World size | 8 |
| Gradient accumulation | 1 |
| Global batch tokens | 524,288 |
| Steps per full epoch | 3,889 |
| Train windows available | 1,991,282 |
| Train windows consumed per epoch | 1,991,168 |
| Tokens consumed per epoch | 2,038,956,032 |
| Checkpoint interval | Every 500 steps |
| Evaluation interval | Every 1,000 steps and at epoch end |
| Seed | 1337 |
The model has completed 4 full training epochs over the packed train split.
Evaluation
Perplexity is computed as exp(validation_loss).
The reported validation metrics use 4,096 validation windows, or about 4.19M validation tokens, sampled from the internal packed validation split. The full validation split contains 20,119 windows, or 20,601,856 tokens.
| Epoch | Status | Epoch tokens | Total tokens seen | Steps | Validation loss | Validation perplexity |
|---|
| 1 | Completed | 2.039B | 2.039B | 3,889 | 3.3730 | 29.17 |
| 2 | Completed | 2.039B | 4.078B | 3,889 | 2.9986 | 20.06 |
| 3 | Completed | 2.039B | 6.117B | 3,889 | 2.8545 | 17.37 |
| 4 | Completed | 2.039B | 8.156B | 3,889 | 2.7765 | 16.06 |
The latest completed evaluation is:
| Metric | Value |
|---|
| Validation loss | 2.7765 |
| Validation perplexity | 16.06 |
| Total tokens seen | 8,155,824,128 |
No downstream task benchmark results are reported yet.
Approximate Training Cost
Known pretraining cost by epoch:
| Epoch | H100 | CPU | Memory | Total |
|---|
| 1 | ~$7.18 | Not recorded | Not recorded | ~$7.18+ |
| 2 | ~$9.60 | Not recorded | Not recorded | ~$9.60+ |
| 3 | $10.18 | $0.25 | $0.33 | $10.76 |
| 4 | $11.02 | $0.27 | $0.36 | $11.65 |
Approximate known pretraining total: ~$39.19+. This excludes any unrecorded dataset preparation, storage, network, or earlier CPU-only costs.
Usage
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4model_id = "asinha08/slm-125m-base"
5
6tokenizer = AutoTokenizer.from_pretrained(model_id)
7model = AutoModelForCausalLM.from_pretrained(
8 model_id,
9 torch_dtype=torch.bfloat16,
10 device_map="auto",
11)
12
13prompt = "<|bos|>The plaintiff alleged that"
14inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
15
16with torch.no_grad():
17 output_ids = model.generate(
18 **inputs,
19 max_new_tokens=80,
20 do_sample=True,
21 temperature=0.8,
22 top_p=0.95,
23 eos_token_id=tokenizer.eos_token_id,
24 )
25
26print(tokenizer.decode(output_ids[0], skip_special_tokens=False))
Because this is a base model, prompts should be treated as text-completion prefixes. The tokenizer contains <|user|>, <|assistant|>, and <|system|> tokens, but the model has not been trained to follow chat instructions.
Limitations and Risks
- The model is small and undertrained relative to modern production LLMs.
- The model can hallucinate, fabricate citations, and produce legally incorrect text.
- The model is not safe for legal advice, compliance review, contract review, court filings, or any high-stakes use without extensive validation.
- The model was not instruction tuned, RLHF tuned, preference tuned, or safety aligned.
- The training corpus is mostly legal/financial English text plus a smaller web-education slice; performance outside that distribution may be poor.
- The model may reproduce biases, sensitive patterns, or memorized fragments from public data.
- Evaluation currently reports only internal validation perplexity, not downstream legal-task accuracy.
- Source datasets have their own licenses and use constraints; users should verify dataset licenses before commercial or redistributed use.
Reproducibility Notes
Main artifacts were stored on the Modal volume slm125mLIVE:
| Artifact | Path |
|---|
| Cleaned data | /data/clean |
| Deduplicated corpus | /data/corpus |
| Tokenizer | /data/tokenizer |
| Tokenized train/val windows | /data/tokens |
| Checkpoints | /data/checkpoints |
| Final base checkpoint | /data/checkpoints/base |
Latest training summary:
| Field | Value |
|---|
| Status | completed |
| Steps | 3,889 / 3,889 |
| Epoch base tokens | 6,116,868,096 |
| Epoch tokens | 2,038,956,032 |
| Total tokens seen | 8,155,824,128 |
| World size | 8 |
| Gradient accumulation | 1 |
| Final checkpoint | /data/checkpoints/ckpt.pt |
| Final model directory | /data/checkpoints/base |
Citation
No formal citation is available. If you use this model, cite the Hugging Face repository:
