Base model : allegrolab/hubble-1b-100b_toks-perturbed-hf @ step48000 Method : SHRED (Self-distillation via High-surprisal-only Retain-set-free Entropy Demotion) Paper : https://arxiv.org/abs/2605.07482 Forget set : allegrolab/biographies_yago dup >= 64 (268 examples) Retain set : NOT USED — SHRED is retain-set-free

Token-level surprisal selection (bottom 15% lowest-probability tokens) followed by KL self-distillation with logit demotion (target prob=0.01). No reference model, no retain set.

lr=1e-05 select_fraction=0.15 demote_target=0.01 effective_batch=8 epochs=5

Completed 5 epochs — final avg loss: 0.0008