OpenELM 270M for Basque trained from-scratch on
ZelaHandi-v1 for 25 epochs with a native 32K llama3 tokenizer.
The creation of this dataset has been partially funded by the Basque Government (ICL4LANG project, grant no. KK-2023/00094) and the European Union (EFA 104/01-LINGUATEC IA project, INTERREG POCTEFA 2021-2027 program).
Pre-training and fine-tuning of SLMs were conducted using the Hyperion system at the Donostia International Physics Center (DIPC).
Finally, we thank Idoia Davila Uzkudun for her contributions to manual data curation and evaluation.
1@inproceedings{urbizu2025sub,
2 title={Sub-1B Language Models for Low-Resource Languages: Training Strategies and Insights for {B}asque},
3 author={Urbizu, Gorka and Corral, Ander and Saralegi, Xabier and San Vicente, I{\~n}aki},
4 booktitle={Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025)},
5 pages={519--530},
6 year={2025}
7}
8
