SFT fine-tune of Probabilistr/slm-125m-base (a 125.8M-parameter from-scratch Llama-architecture model) on a synthetic instruction/response dataset for legal and financial documents.
Fine-tuning data
4,500 examples (grounded QA/RAFT, summarization, extraction-to-JSON, and plain-English rewrite tasks) generated with gpt-4o-mini from case-law and SEC-filing passages, then filtered for format, eval-set decontamination, near-duplicate removal, and LLM-judged grounding. See the project's Instructions_for_finetuning.txt and the dataset guidelines it references for the full pipeline.
Training
1x A10G, 3 epochs, 837 steps, ~8 minutes
Loss masked to only the assistant-response tokens
See metrics.jsonl / summary.json in this repo for the training curve
Known limitations
Grounding discipline is inconsistent on held-out prompts -- in particular the model doesn't reliably say "Not stated in the context" when the context doesn't contain the answer, and can conflate similar figures within a passage. This is a small model fine-tuned on a small dataset; treat outputs accordingly.
Prompt format
<|bos|><|system|>
You are a legal/financial assistant. Answer strictly using the information in the provided context. If the answer is not stated in the context, say "Not stated in the context."<|user|>
Context:
{context}
Instruction: {instruction}<|assistant|>
