This is a 125,848,320-parameter base causal language model, trained from random weights on a packed English legal, financial, and educational corpus. It is a passage-completion model, not an instruction-following chatbot, legal adviser, or factual authority.

Training result

Metric	Value
Requested horizon	2 epochs
Optimizer steps	7,643
Input tokens seen	4,007,133,184
Final validation loss	2.2201357403
Final validation perplexity	9.2085807565
Effective global batch	524,288 input tokens
Hardware	Single node, 8× NVIDIA H100
Precision	bfloat16
Context length	1,024
Vocabulary	16,384
Actual production GPU charge	USD 23.06837
Actual production total charge	USD 23.41963

Validation perplexity was computed over the complete validation split using token-weighted next-token cross-entropy across 20,210,388 predicted tokens.

Architecture

Llama-style decoder-only transformer
12 layers
Hidden size 768
12 attention heads and 12 KV heads
SwiGLU intermediate size 3,072
RoPE theta 10,000
RMSNorm pre-normalization
Tied input/output embeddings

Optimization

AdamW; betas (0.9, 0.95)
Weight decay 0.1 (matrix parameters; biases and one-dimensional parameters excluded)
Peak learning rate 6e-4; floor 6e-5
Token-based 200M-token warm-up followed by cosine decay
Gradient clipping at 1.0
Micro-batch 32 windows per GPU, 8 ranks, accumulation 2

Dataset

The packed training split contains 2,003,556,352 tokens and the validation split contains 20,230,144 tokens. Every record is an already-packed 1,024-token uint16 window. The source mixture was built from:

HFforLegal/case-law
PleIAs/SEC
HuggingFaceFW/fineweb-edu (sample-10BT)

The data pipeline applied document cleaning, exact/near deduplication, legal-evaluation decontamination, a custom 16,384-token tokenizer, EOS-delimited packing, and a deterministic 99:1 train/validation window split. Review each upstream dataset's terms before downstream commercial or regulated use; this repository does not relicense upstream data.

Validation trajectory

Step	Validation loss	Perplexity
1,000	2.7723	15.9950
2,000	2.5223	12.4573
3,000	2.4149	11.1890
4,000	2.3437	10.4192
5,000	2.2922	9.8963
6,000	2.2537	9.5232
7,000	2.2288	9.2887
7,643	2.2201	9.2086

Two one-H100 smoke runs preceded production. Run 1 reached validation but failed during metric-event logging; run 2 passed after that logging defect was fixed. The production run completed with no error events.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "lekhamenon28/legal-slm-125m-2ep"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "The court held that the contractual provision"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=120, do_sample=True, temperature=0.8)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations and safety

This model is not instruction-tuned and should be prompted with passage prefixes.
It can hallucinate cases, statutes, citations, numbers, and legal conclusions.
It must not be used as legal, financial, compliance, or investment advice.
Training data may contain errors, outdated material, biases, or personal information present in upstream sources.
The two-epoch experiment is shorter than the project's five-epoch reference recipe.