Metric	Baseline	This Model	Δ
ARC-Easy	54.12%	57.28%	+3.16pp 🏆
GSM8K	0.38%	2.35%	+1.97pp (6.2×) 🏆
PIQA (norm)	66.92%	67.14%	+0.22pp 🏆
ARC-Challenge (norm)	27.73%	29.18%	+1.45pp
HellaSwag (norm)	42.99%	42.95%	-0.04pp (preserved)
BoolQ	43.39%	38.99%	-4.40pp (alignment tax)

Training Recipe

2-stage pipeline on SmolLM2-135M-Instruct:

SFT Stage: All reasoning data (~100K samples), lr=3e-4, 1 epoch, cosine scheduler
DPO Stage: Sigmoid loss, 10K preference pairs, lr=5e-5, constant scheduler (no decay), 1 epoch

Key Discovery: Constant LR Scheduler

The most impactful finding: using a constant learning rate scheduler (instead of cosine) for DPO training produces significantly better benchmark results. This is a free improvement requiring no additional data or compute.

Constant scheduler keeps lr=5e-5 throughout, extracting maximum value from every sample
Cosine decay wastes the final training portion by reducing LR to ~0
More data (20K) with constant scheduler actually hurts — causes overfitting

Study Summary

This model emerged from 200+ systematic experiments exploring:

8 DPO loss types (sigmoid, nca_pair, robust, hinge, IPO, etc.)
5 LR schedulers (cosine, constant, constant_with_warmup, linear, polynomial)
Data scaling (3K to 100K samples)
Architecture variants (LoRA vs full fine-tuning)
Multi-stage pipelines and curriculum strategies

Full experiment report: github.com/lldois/tiny_resoning_llm

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("lldois/SmolLM2-135M-Reasoning-exp146")
tokenizer = AutoTokenizer.from_pretrained("lldois/SmolLM2-135M-Reasoning-exp146")

messages = [{"role": "user", "content": "What is 25 + 37?"}]
input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))