SFT checkpoint from a masking-vs-long-context diagnostic re-run of Stage-1 SFT on the MV-16k reference model. This variant uses assistant-only loss masking ({% generation %} chat-template tags + assistant_only_loss=true), i.e. the loss is only computed on assistant turns, not the full sequence (including user turns/prompts).

It is one of a pair of runs (masked vs unmasked) used to test whether a reported post-training MATH500 collapse (35% → 3%) on MV-16k was caused by instruction masking or by long-context handling in the Tulu3 SFT/DPO recipe. (MV-4k was unaffected by this issue, so only MV-16k was re-run.)

Base model: ali-elganzory/1.7b-MixtureVitae-300BT-v1-decontaminated-16k
SFT data: ali-elganzory/tulu-3-sft-mixture-decontaminated (936,509 rows)
Counterpart run: laion/1.7b-MixtureVitae-300BT-v1-decontaminated-16k-SFT-Tulu3-decontaminated-unmasked
Only two things differ from the originally published recipe: attn_implementation is sdpa instead of flash_attention_2 (no aarch64 flash-attn wheel available, held constant across both masked/unmasked runs), and the masking configuration itself (the variable under test).

Training results

metric	value
train_loss	0.0323
train_samples	936,509
epochs	1

Framework versions

TRL: 0.27.1
Transformers: 4.57.6
Pytorch: 2.6.0+cu126
Datasets: 4.8.4
Tokenizers: 0.22.2

Trained with TRL.

laion/1.7b-MixtureVitae-300BT-v1-decontaminated-16k-SFT-Tulu3-decontaminated-masked

1.7b-MixtureVitae-300BT-v1-decontaminated-16k-SFT-Tulu3-decontaminated-masked

Training results

Framework versions

laion/
1.7b-MixtureVitae-300BT-v1-decontaminated-16k-SFT-Tulu3-decontaminated-masked