You can deploy this model and start earning money today!

proxectonos/
deberta-xsmall-gl

Name: deberta-xsmall-gl
Availability: InStock

microsoft/deberta-v3-xsmall

finetune

mit

endpoints_compatible

Views

No views yet

DeBERTa-xsmall-gl

DeBERTa-xsmall-gl is a continued pretraining checkpoint based on microsoft/deberta-v3-xsmall, adapted to Galician through large-scale masked-language modeling. It is intended as a strong general-purpose encoder for downstream NLP tasks in Galician.

Training

Base model: microsoft/deberta-v3-xsmall
Epochs: 3
Learning rate: 6e-4
MLM probability: 0.15
Max sequence length: 512
Total batch size: 1024
Training examples: 6,139,791

Intended uses

Masked language modeling (fill-mask)
Encoder for classification, NER, QA, and general Galician NLP tasks
Further domain adaptation via fine-tuning

How to use

python

from transformers import AutoModelForMaskedLM, AutoTokenizer, pipeline

model_id = "proxectonos/deberta-xsmall-gl"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)

fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer)

fill_mask("O Parlamento de Galicia aprobou a <mask> hoxe.")

Funding

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Desarrollo de Modelos ALIA

Citation

Please reference this model as: mdeberta-gl (Proxecto Nós Team, 2025).