This model was developed for the paper Grow Up and Merge: Scaling Strategies for Efficient Language Adaptation.

It is based on the SmolLM2 architecture, but instead of the original English tokenizer, it uses the multilingual Llama 3.3 tokenizer with a 128K vocabulary. This results in a 180M-parameter base model.

The base models were trained on:

There exist 2 English base models: A 180M base model, and a 572M base model that is the 180M base model HyperCloned after 80% of the training steps, and trained on the remaining 20% of the data afterwards.

Models were adapted to each target language using the corresponding FineWeb-2 subset. The mutilingual models were adapted the same way, but on the FineWeb-2 subsets for all target languages at once.

We evaluate three adaptation setups:

For more details on our setups, please refer to our paper.

The model can be loaded like this. But note that this is a base model; it is not instruction-tuned.

Users should apply the model responsibly, particularly in high-stakes or user-facing applications. The model is not guaranteed to provide factual, neutral, or safe outputs. Additional safety filtering, human oversight, or RLHF-style alignment may be required depending on the deployment context. Models of such sm all sizes are expected to produce hallucinations, especially for topics underrepresented or inconsistently represented in the training data.

While the base model’s pre-training corpus presumably excludes overtly unethical or harmful material due to its educational focus, the continued pre-training phase introduces additional considerations. Large-scale corpora for our target languages are comparatively limited in availability and curation quality. As a result, the datasets used for domain or language adaptation may contain inappropriate, biased, or otherwise undesirable content.

If you use this model, please cite: