0.4932 (confirmed 0.4921 and 0.4929 on two independent reruns)
Training
Three soft knowledge-distillation stages from the base model, within the challenge budget of 10 H100 GPU hours (measured replay: ~8.9–9.9 H100h including data generation, teacher labeling and all confirmation evaluations; stage-3 training measured 1h42m on an RTX PRO 6000 Blackwell).
For every training question a teacher committee provides its probability distribution over the answer letters at the answer token (stage 1: a 3-teacher committee — Mistral-Small-3.2-24B, Gemma-3-27B, Qwen3-32B, each fitting a single H100 in bf16; stage 2: a public 2-teacher committee — Qwen2.5-72B-Instruct-AWQ + Mistral-Small-3.2-24B; stage 3: Mistral-Small-3.2-24B). The student is trained in fp32 with a KL term on that distribution (temperature 2.0) plus cross-entropy on the gold letter (weight 0.5).
Stage 1 (from base): 21,606 items — pinocchio-derived knowledge MCQs, committee-labeled grammar MCQs, and generated orthography/morphology MCQs for the categories where public data is scarce. → 0.4787
Stage 2 (M2): 22,950 culture-focused committee-labeled items (italic-m2-culture-pool), LR 2e-5, seed 99, best checkpoint at step 2000. → 0.4878/0.4880
Stage 3 (extkd, this model): 28,561 external in-distribution items — the public datasets italic_sft (20,665), italic_sft_ext (3,268), quiz_militare (3,641) and a pinocchio sample — soft-labeled by Mistral-Small-3.2-24B (public, local) and agreement-filtered, best checkpoint at step 1500. → 0.4921–0.4932
Every pool is decontaminated against the ITALIC test set (exact match plus semantic similarity at cosine 0.80; the external datasets show 0 exact leaks and 0 near-duplicates at that threshold; a pinocchio sample with 0.74% near-duplicates was dropped). All pools are answer-position balanced (~25% per letter) — we found that position skew in synthetic MCQ pools causes answer-distribution collapse in the student.
Official ITALIC harness, unmodified: vLLM server, fast mode (letter-only answers), 5-shot, temperature 0, accuracy over all 10,000 questions. The chat template is embedded in the tokenizer config; serve the model with its own template.
vllm serve idealab-cs2/zagreus-0.4B-italic-extkd --chat-template-content-format auto
Per-category accuracy: civic education .56, lexicon .58, synonyms and antonyms .55, current events .53, tourism .51, geography .50, art history .49, history .47, syntax .45, literature .42, orthography .42, morphology .39.
Acknowledgments
Training, data curation and evaluation by ruggsea, using compute resources of the Complex Social & Computational Systems (CS²) group, IDea_Lab, University of Graz.
Intended use
This model is specialized for Italian multiple-choice question answering in the ITALIC format. It is a 0.4B model: outside this format expect base-model-level capabilities.
