
Examples / tokens	934 / 497,968
Supervised	409,975 / 497,968 = 82.3%
Epochs / steps	1 / 59
lr / schedule	4e-5, cosine, 3% warmup
Runtime	36 min, 1x H100 80GB
r / alpha / dropout	32 / 64 / 0.05
batch x grad-accum	1 x 16
max seq len / packing	3072 / off
Final loss	0.538
Token accuracy	0.878

Reading the loss

0.538 is roughly half the loss of every mixed-corpus arm in this family (0.76-0.95), and token accuracy is ~8 points higher. That is the narrowness of the target, not model quality: NuminaMath solutions follow a highly regular mathematical structure, so next-token prediction is far easier than on open-ended instruction data. Loss converged by about step 15 of 59.

Run	Data	Loss	Token acc
this	NuminaMath only	0.538	0.878
500k maths-weighted	67% maths + mixed	0.878	0.793
500k da20 + maths	20% difficult-advice	0.946	0.776

What is supervised

Everything outside an assistant turn is -100. A supervised span starts immediately after the <|im_start|>assistant\n header and ends after the closing <|im_end|>. Verified before training: zero user or system tokens inside any supervised span.

loss_type: nll is set because TRL's default chunked-CE path patches the LM head and reads forward.__func__, which fails on this checkpoint's functools.partial forward.

Not yet evaluated.

Usage

python

from peft import PeftModel
from transformers import AutoModelForImageTextToText

model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.6-27B", dtype="bfloat16")
model = PeftModel.from_pretrained(model, "LASR-Callum/2026-08-03-qwen36-lora-500k-numina-only")
model = model.merge_and_unload()

Use AutoModelForImageTextToText, not AutoModelForCausalLM — this is a vision-language checkpoint.

LASR-Callum/qwen3.6-27b-lora-500k-numina-only

Qwen3.6-27B — 500k tokens, NuminaMath-CoT only

Reading the loss

What is supervised

Usage

LASR-Callum/
qwen3.6-27b-lora-500k-numina-only