HiTZ/gl_Qwen3-8B-Base
This is a Galician (gl) language-specific base language model trained by the HiTZ Research Center, starting from Qwen3-8B-Base and further pretrained on curated Galician data.
This model is released as a base model, intended for further fine-tuning or adaptation (e.g., instruction tuning, domain adaptation).
Training Data
To train language-specific base LLMs, we followed the methodology proposed by
Etxaniz et al. (2024), originally developed for Basque, and extended it to other low-resource languages. To enable fair comparisons across languages, we limited the corpus size for each language to roughly the same number of tokens. We also included a small English subset to mitigate catastrophic forgetting.
Corpus composition
| Language | Documents | Tokens (Qwen3) |
|---|
| Galician (gl) | 8.9M | ~3.5B |
| English (en) | 0.5M | ~0.3B |
Token counts vary slightly depending on the tokenizer, but remain comparable in overall size.
Data sources
Galician data was obtained from the CorpusNÓS corpus, which comprises large-scale web crawls and texts from public administrations, among other sources.
The English subset was sampled from the FineWeb corpus.
Model Training
- Sequence length: 8,196 tokens
- Effective batch size: 256 sequences
- Tokens per optimization step: ~2M
- Learning rate schedule: cosine decay with 10% warm-up
- Peak learning rate: 1e-5
Training was conducted on the CINECA Leonardo high-performance computing cluster using Fully Sharded Data Parallel (FSDP) across 32 nodes, each equipped with 4 NVIDIA A100 GPUs (64 GB).
Getting Started
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model_id = "HiTZ/gl_Qwen3-8B-Base"
4
5tokenizer = AutoTokenizer.from_pretrained(model_id)
6model = AutoModelForCausalLM.from_pretrained(model_id)
7
8inputs = tokenizer("Ola!", return_tensors="pt")
9outputs = model.generate(**inputs, max_new_tokens=50)
10
11print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Acknowledgements
This work has been partially supported by the Basque Government (Research group funding IT1570-22 and IKER-GAITU project), the Spanish Ministry for Digital Transformation and of Civil Service, and the EU-funded NextGenerationEU Recovery, Transformation and Resilience Plan (ILENIA project, 2022/TL22/00215335; and ALIA project).
