Linear weight-merge of two SmolLM2-1.7B pretraining checkpoints (each trained on 100B tokens).
merged = 0.9 * EPE + 0.1 * NORMAL
Component
Weight
Source checkpoint
EPE (no_bce)
0.9
epe-1p-smollm-1p7b-100B-20n-2048sl-960gbsz-no_bce
NORMAL
0.1
normal-smollm-1p7b-100B-20n-2048sl-960gbsz
Merge details
Method: element-wise linear interpolation of weights (computed in fp32, stored in bf16).
Architecture: LlamaForCausalLM, identical for both parents (hidden 2048, 24 layers, 32 heads, tied embeddings).
Vocabulary: 49280. The two parents share 49152 tokens (interpolated); the 128 extra EPE
"charter" tokens are carried over from the EPE parent unchanged. The EPE tokenizer is bundled.
