cortexso/

smollm2

SmolLM2 is a family of compact language models available in three sizes: 135M, 360M, and 1.7B parameters. These models are designed to solve a wide range of tasks while being lightweight enough for on-device deployment. More details can be found in the SmolLM2 paper.

The 1.7B variant demonstrates significant improvements over its predecessor, SmolLM1-1.7B, especially in instruction following, knowledge retention, reasoning, and mathematical problem-solving. It was trained on 11 trillion tokens using a diverse dataset combination, including FineWeb-Edu, DCLM, The Stack, and newly curated mathematics and coding datasets that will be released soon.

The instruct version of SmolLM2 was developed through supervised fine-tuning (SFT) using a mix of public datasets and curated proprietary datasets. It further benefits from Direct Preference Optimization (DPO) using UltraFeedback.

Additionally, the instruct model supports tasks such as text rewriting, summarization, and function calling, enabled by datasets from Argilla, including Synth-APIGen-v0.1. The SFT dataset is available at: SmolTalk SFT Dataset.

For further details, visit the SmolLM2 GitHub repository, where you will find resources for pre-training, post-training, evaluation, and local inference.