A 125M decoder-only language model (chat-aligned via SFT + DPO preference learning). Part of the SLM model family — built entirely from scratch, from raw web data through to a production-ready aligned model.

This is the chat variant — the instruct model further aligned via Direct Preference Optimization (DPO). This is the recommended variant for conversational and assistant use cases. Use tohio/slm-125m-instruct for the SFT-only version. Use tohio/slm-125m for the raw base model.

Model Family

Variant	Hub	Description
Base	tohio/slm-125m	Pretrained only
Instruct	tohio/slm-125m-instruct	Chat + response-control + code SFT
Chat	tohio/slm-125m-chat	SFT + DPO aligned

Architecture

Component	Choice	Rationale
Positional encoding	RoPE	Better length generalisation, relative position awareness
Normalization	RMSNorm	Faster than LayerNorm, modern standard
Activation	SwiGLU	Better gradient flow, used by LLaMA and Mistral
Attention	GQA	Reduces KV cache memory at inference
Bias	None	Simpler, modern standard
Embeddings	Tied	Reduces parameters, effective at small scale
Vocab size	32,000	Custom BPE tokenizer trained on the pretraining corpus
Parameters	125.3M (125,264,640 parameters)

Training

Pretraining corpus — 10B curation target blended across the following sources:

Source	Target Share	Link
`common_crawl`	5.0%	Common Crawl
`fineweb`	10.0%	FineWeb
`fineweb_edu`	31.5%	FineWeb-Edu
`wikipedia`	10.0%	Wikipedia (EN)
`pg19`	2.5%	PG-19 (Project Gutenberg)
`pes2o`	5.0%	peS2o (academic papers)
`nemotron_cc_math`	7.0%	Nemotron CC Math
`stackexchange`	1.0%	StackExchange
`synthetic_arithmetic`	0.1%	Synthetic arithmetic
`synthetic_task_code`	0.4%	Synthetic task code
`educational_qa_mcq_math`	0.1%	Educational QA/MCQ (math)
`educational_qa_mcq_general`	0.2%	Educational QA/MCQ (general)
`factual_restraint`	0.1%	Factual restraint
`nemotron_specialized`	12.0%	Nemotron Specialized
`stack_v1`	12.45%	The Stack v1 dedup
`codesearchnet`	2.25%	CodeSearchNet
`stack_smol`	0.15%	The Stack (smol)
`jupyter`	0.07%	Jupyter notebooks
`conala`	0.07%	CoNaLa

Realized mix may differ from target — supply-bound sources (pes2o, jupyter at this scale) route their deficit to FineWeb.

Fine-tuning and alignment

Stage	Dataset	Size
Chat SFT	OpenHermes-2.5	~1M examples
Response-control SFT	Generated locally by `finetune/data/response_control.py`	5K examples
Code SFT	Magicoder-OSS-Instruct-75K + handcrafted body-only completions	~75K examples + small handcrafted set
DPO alignment	Anthropic/hh-rlhf + Intel/orca_dpo_pairs + argilla/dpo-mix-7k	~60K pairs after filtering

Hardware: NVIDIA H200 (pretraining on 1× H200, fine-tuning on 1× H200)

Evaluation

Evaluated using lm-evaluation-harness.

Benchmark	Few-shot	Metric	Score
HellaSwag	10-shot	acc_norm	0.3099
ARC-Easy	25-shot	acc_norm	0.4895
ARC-Challenge	25-shot	acc_norm	0.2577
MMLU	5-shot	acc	0.2536
TruthfulQA	0-shot	acc	0.4295
HumanEval	0-shot	pass@1	0.0915

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "tohio/slm-125m-chat",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "tohio/slm-125m-chat",
    trust_remote_code=True,
)

messages = [
    {"role": "system", "content": "Answer clearly and concisely."},
    {"role": "user", "content": "Explain what a transformer is."},
]

inputs = tokenizer.apply_chat_template(
    messages,
    return_tensors="pt",
    add_generation_prompt=True,
    return_dict=True,
)

endofturn_id = tokenizer.convert_tokens_to_ids("<|endofturn|>")

output = model.generate(
    **inputs,
    max_new_tokens=120,
    do_sample=False,
    repetition_penalty=1.1,
    pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id,
    eos_token_id=[tokenizer.eos_token_id, endofturn_id],
)

input_len = inputs["input_ids"].shape[1]
print(tokenizer.decode(output[0][input_len:], skip_special_tokens=True))

trust_remote_code=True loads the custom SLM architecture bundled alongside the model weights — no local install of the tohio/slm codebase required.

Limitations

Scale: At 125M parameters this model is significantly smaller than frontier models. It will underperform on complex reasoning, long-context tasks, and domains not well-represented in the pretraining data.
Hallucination: Like all language models, this model can generate plausible-sounding but factually incorrect content. Outputs should not be used as a source of truth without independent verification.
Safety: DPO alignment provides basic harmlessness training but does not guarantee safe outputs in all contexts. This model has not undergone red-teaming or adversarial safety evaluation.
Languages: Training data is predominantly English. Performance on other languages will be significantly degraded.
Code: Code generation is primarily Python-oriented, reflecting the code sub-mix distribution used in pretraining and SFT.

slm — full training pipeline (data curation through serving)
ai-infra — production Kubernetes serving via vLLM

tohio/slm-125m-chat

slm-125m-chat