Llama-3.2 1B for Basque continually pretrained on
ZelaHandi-v1 for 5 epochs.
The creation of this dataset has been partially funded by the Basque Government (ICL4LANG project, grant no. KK-2023/00094) and the European Union (EFA 104/01-LINGUATEC IA project, INTERREG POCTEFA 2021-2027 program).
Pre-training and fine-tuning of SLMs were conducted using the Hyperion system at the Donostia International Physics Center (DIPC).
Finally, we thank Idoia Davila Uzkudun for her contributions to manual data curation and evaluation.
1@inproceedings{urbizu2025sub,
2 title={Sub-1B Language Models for Low-Resource Languages: Training Strategies and Insights for {B}asque},
3 author={Urbizu, Gorka and Corral, Ander and Saralegi, Xabier and San Vicente, I{\~n}aki},
4 booktitle={Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025)},
5 pages={519--530},
6 year={2025}
7}
8
