DPO checkpoint from a masking-vs-long-context diagnostic re-run of Stage-1 SFT→DPO on the MV-16k reference model, trained on top of the corresponding masked SFT checkpoint (laion/1.7b-MixtureVitae-300BT-v1-decontaminated-16k-SFT-Tulu3-decontaminated-masked). This variant uses assistant-only loss masking ({% generation %} chat-template tags + assistant_only_loss=true) at the SFT stage, i.e. loss is only computed on assistant turns, not the full sequence.

It is one of a pair of runs (masked vs unmasked) used to test whether a reported post-training MATH500 collapse (35% → 3%) on MV-16k was caused by instruction masking or by long-context handling in the Tulu3 SFT/DPO recipe. (MV-4k was unaffected by this issue, so only MV-16k was re-run.)

Base/starting policy: masked SFT checkpoint above (itself fine-tuned from ali-elganzory/1.7b-MixtureVitae-300BT-v1-decontaminated-16k)
DPO data: ali-elganzory/llama-3.1-tulu-3-8b-preference-mixture-decontaminated (272,585 rows)
Counterpart run: laion/1.7b-MixtureVitae-300BT-v1-decontaminated-16k-DPO-Tulu3-decontaminated-unmasked
Only two things differ from the originally published recipe: attn_implementation is sdpa instead of flash_attention_2 (no aarch64 flash-attn wheel available, held constant across both masked/unmasked runs), and the masking configuration applied at the SFT stage (the variable under test).

Training results

metric	value
train_loss	2.6546
train_samples	272,585
epochs	1

Trained with DPO (Rafailov et al., 2023).

Framework versions

TRL: 0.27.1
Transformers: 4.57.6
Pytorch: 2.6.0+cu126
Datasets: 4.8.4
Tokenizers: 0.22.2

Trained with TRL.

laion/1.7b-MixtureVitae-300BT-v1-decontaminated-16k-DPO-Tulu3-decontaminated-masked

1.7b-MixtureVitae-300BT-v1-decontaminated-16k-DPO-Tulu3-decontaminated-masked

Training results

Framework versions

laion/
1.7b-MixtureVitae-300BT-v1-decontaminated-16k-DPO-Tulu3-decontaminated-masked