Supervised fine-tunes of Ace-2504/slm-125m-base (125.8M params, 1024 ctx, pretrained from scratch on 2.04B tokens of US case law, SEC filings and educational web text).

This repo is a research artifact, not a useful assistant. It is the output of a ten-round data-scaling study whose headline result is a negative one: scaling SFT data 10× did not improve answer quality (judge flat at ~1.5/5) and tripled catastrophic forgetting.

📊 Full write-up: https://ace-2504.github.io/fine-tuned-125m-slm/ · 💻 Code: https://github.com/Ace-2504/fine-tuned-125m-slm

Two revisions — pick deliberately

Revision	Data	Judge /5	Forgetting vs base	Notes
`main`	2,000 pairs	1.50	+9.5%	The better model. Recommended.
`day-10`	10,000 pairs	1.54	+16.3%	Study endpoint. 5× the data, no quality gain, ~2× the forgetting, and a regressed probe answer.

The two judge scores are identical within the measured ±0.07 noise band, so main (day 2) strictly dominates: same quality, half the damage, one-fifth the data.

python

from transformers import AutoModelForCausalLM, AutoTokenizer

# the better model (day 2)
tok = AutoTokenizer.from_pretrained("Ace-2504/fine-tuned-125m-slm")
model = AutoModelForCausalLM.from_pretrained("Ace-2504/fine-tuned-125m-slm")

# the 10k endpoint, for comparison
tok10 = AutoTokenizer.from_pretrained("Ace-2504/fine-tuned-125m-slm", revision="day-10")
model10 = AutoModelForCausalLM.from_pretrained("Ace-2504/fine-tuned-125m-slm", revision="day-10")

msgs = [
    {"role": "system", "content": "You are a precise legal and financial assistant."},
    {"role": "user", "content": "What is the standard of proof in a civil lawsuit?"},
]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
print(tok.decode(model.generate(ids, max_new_tokens=120)[0][ids.shape[1]:], skip_special_tokens=True))

Honest limitations

Answer quality is poor (~1.5/5). It learned the shape of an assistant turn, not reliable answers. Expect repetition, prompt echoing, and confident wrong answers.
It forgot part of its pretraining. Perplexity on the original pretraining validation set rose from 11.35 → 12.42 (main) / 13.20 (day-10).
Root cause: ~40% of training was closed-book QA — recalling a document fact from a single exposure, which a 125M model cannot do. Common facts land; long-tail specifics never do.
Do not use for legal or financial advice. It is a study artifact.

Training

Full fine-tune (no LoRA), 3 epochs, batch 16, LR 3e-5→3e-6 cosine, AdamW, bf16, assistant-only loss masking, seed 1337, on a single Modal L4. Data: QnA written by gemini-3.1-flash-lite, grounded in passages from the same corpus the base was pretrained on, then filtered, deduplicated (exact + embedding), decontaminated and balanced. Chat template uses <|system|> / <|user|> / <|assistant|>.

generation_config ships with repetition_penalty=1.3 and no_repeat_ngram_size=3 — without them greedy decoding falls into repetition loops.