You can deploy this model and start earning money today!

elguantletai002/
checklist-reviewer-qwen3-0.6b-n455

Name: checklist-reviewer-qwen3-0.6b-n455
Availability: InStock

Views

No views yet

checklist-reviewer — Qwen3-0.6B LoRA (N=455)

This is the checkpoint every reported number refers to. Trained on data/sft_v2.jsonl, evaluated on eval/sets/review_v2.jsonl, whose fixtures share no distinctive line with the training seeds (test-enforced).

Diff in, checklist JSON out. Ten frozen rules, no praise, no patches, no naming the symbol to change.

Results

Judged by claude-sonnet-5, frozen rubric, 36 scenarios.

	base	this adapter	prompted frontier ceiling
spec-adherence	0.139	0.611	0.861
robustness	0.208	0.583	0.875
strict (right rule, legal anchor)	0.028	0.278	0.806
false-clean (lower is better)	0.133	0.100	0.000

Data closed roughly half the gap on adherence and a third on strict. It reaches neither the 0.90 bar nor the prompted ceiling.

What it learned: the format completely — markdown fences went from 32 of 36 replies to zero — and the rule vocabulary.

What it did not: where to point. strict reached 0.278 because the model did not learn to count @@ hunk arithmetic. That is the dominant residual failure and it is also the only genuine failure mode the frontier shows.

It did not learn to say nothing. false-clean fell (0.133 → 0.100) while adherence quadrupled. {"findings": []} passes every schema and groundedness check, so a model that collapsed to empty would score well and be worthless.

Training

LoRA — not QLoRA: bf16, no quantisation, since 4-bit saves VRAM that is not scarce for a 0.6B model on a 24GB A10. r=256, alpha=256, target_modules="all-linear", lr 2e-4, 2 epochs, effective batch 16, on 455 teacher-distilled and filtered examples.

Known defect: TRL 0.15 exposes neither assistant_only_loss nor completion_only_loss, so loss covered the ~1.1k-token system prompt too. Most of the gradient went to reproducing the spec, which is likely part of why strict lagged.

Sibling checkpoints n227, n113, n56 form the data-efficiency curve; minimum viable N is 227.

Use

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B")
model = PeftModel.from_pretrained(base, "elguantletai002/checklist-reviewer-qwen3-0.6b-n455").merge_and_unload()
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")

The system prompt is spec/behavior_spec.md from the source repo. Behaviour is undefined without it.

The n414 / n207 / n103 / n51 repos are superseded: they were trained on a dataset that shared fixtures with its own eval set, so their numbers measured recognition of training data. Use this one.