btprop-sft-judge-qwen3-8b
122B-distilled judge. Closed none of the teacher-student gap and became unparseable on 15% of claims.
This is an archived negative result. It is published so the claim it supports can be checked, not because the model is worth deploying -- read the numbers below before using it for anything.
Qwen3-8B distilled from Qwen3.5-122B-A10B on the judging half of the pipeline: scoring
a claim against retrieved evidence. Unlike the generation side, there was a real gap to close here
-- the teacher is genuinely better.
| arm | AUROC | PRAUC | Acc | unparseable |
|---|
| Qwen3-8B (untrained) | 0.8423 | 0.8038 | 78.28 | 0.0% |
| Qwen3.5-122B teacher | 0.8639 | 0.8340 | 79.40 | 1.0% |
| this model | 0.8402 | 0.8082 | 77.69 | 14.6% |
Teacher minus base is +0.0216; this model closes -10% of it. The scoring targets carried full
chain-of-thought (median 9,619 characters) and the filter was format-only, with no length
constraint: the student learned to write long reasoning and then fail to produce a parseable score
on 14.6% of claims, against 0% for the model it started from. The AUROC above is computed on the
claims it could answer, so it already flatters this model.
Archived as a negative result. The route is not closed -- the teacher does have something to teach
-- but the length blowup has to be fixed first.
Evaluation protocol
Metrics are per-statement hallucination detection on the BTProp stop-node test split (6 datasets, n=2,225 shared claims), with retrieval, judging and aggregation held fixed so that only the generator differs. See EXPERIMENTS.md in the code repo for the full provenance table.
Code
