HuggingFaceTB/

smollm3-configs

[IMPORTANT NOTE]: for the latest configs go to this repo: https://github.com/huggingface/smollm/tree/main/text/pretraining/smollm3 Here you can find the training configs for SmoLLM3-3B-Base using nanotron with exact training details and data mixtures. The model was trained on 11.2T tokens in 3 stages on 4k context:

stage 1 config stage 2 config stage 3 config

And then we trained on an additional 2 stages to extend the contetx length to 64k:

stage 4… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/smollm3-configs.