tulu3sft-epe-3p-smollm-1p7b-100B-20n-2048sl-960gbsz-bce-tmpl-epe

EPE-aware SFT

• Charter tokens [49153, 49188) are masked to -inf in the logits before cross-entropy so the next-token head does not compete with the constitution BCE head during SFT.
• Chat template used for SFT: epe (default = standard SmolLM2 <|im_start|>assistant\n; epe = angle-bracketed <|im_start|><assistant>\n).
• Base pretraining variant: with constitution BCE head.
• Loss fingerprint: epe_sft_shifted_ntp_charter_mask_v4, loss_scaling=hf_distributed_token_average.

Training

• LR: 1e-5, cosine with min_lr_ratio=0.1, warmup_ratio=0.03
• Epochs: 1
• Seq length: 2048
• Effective batch: 128 (mbs=1, grad_accum=8, 16 GPUs)
• Precision: bf16, flash-attention-2