tulu3sft-epe-1p-smollm-1p7b-100B-20n-2048sl-960gbsz-nochartersoft-default-v4

EPE-aware SFT

• Charter tokens [49153, 49188) are masked to -inf in the logits before cross-entropy so the next-token head does not compete with the constitution BCE head during SFT.
• Chat template used for SFT: default (default = standard SmolLM2 <|im_start|>assistant\n; epe = angle-bracketed <|im_start|><assistant>\n).
• Base pretraining variant: nocharter-soft variant (v4).
• Loss fingerprint: epe_sft_shifted_ntp_charter_mask_v4, loss_scaling=hf_distributed_token_average.

Training

• LR: 1e-5, cosine with min_lr_ratio=0.1, warmup_ratio=0.03
• Epochs: 1
• Seq length: 2048
• Effective batch: 128 (mbs=1, grad_accum=8, 16 GPUs)
• Precision: bf16, flash-attention-2