This checkpoint performs poorly in absolute terms. Exact Match of
2.39 is near zero. TIGQA's test split contains only 67 questions, so the metric is
both very hard and very high-variance. It is published for completeness and
reproducibility of the paper's evaluation, not as a usable Tigrinya QA system.
For Tigrinya extractive QA, see Hailay/VEXMLM-TiQuAD.
Five-seed benchmark evaluation
Fine-tuned independently under seeds 42–46 with one configuration (hash
ce27cc194946) on an A100-PCIE-40GB. Reported as mean ± standard deviation over
the five runs, on the dataset's test split.
Metric
Score
Exact Match
2.39 ± 0.82
F1
9.76 ± 0.97
These are the paper's verified results. They come from the five-seed evaluation
described above — not from interactive use.
Interactive inference vs. benchmark
Benchmark evaluation is the five-seed measurement on the held-out test split,
shown in the table above.
Interactive inference is what the usage example below performs: Supply a Tigrinya context and question; the model returns an extracted span.
Predictions on arbitrary user input are demonstrations only and do not produce or
reproduce the benchmark score.
Repository layout
Five independently fine-tuned checkpoints, one per seed. The reported benchmark
score is the mean ± standard deviation over all five; no single seed is the
"five-seed model."
seed-42/ seed-43/ seed-44/ seed-45/ seed-46/
Load a specific seed with the subfolder argument, as in the example below.
Fine-tuning
Fine-tuned from Hailay/VEXMLM, a
vocabulary-extended XLM-R (280,002 subwords, 30,000 Ge'ez tokens merged into the
SentencePiece model) after continued MLM pretraining.
Hyperparameter
Value
Max sequence length
256
Batch size
32
Epochs
4
Learning rate
2e-5
LR schedule
Linear decay, 10% warmup
Weight decay
0.01
Gradient clipping
1.0
Optimizer
AdamW (β₁ 0.9, β₂ 0.999, ε 1e-8)
Precision
bf16
Trainable parameters
All
Hardware
1× NVIDIA A100
Runs are bit-reproducible: enable_full_determinism,
CUBLAS_WORKSPACE_CONFIG=:4096:8, dataloader_num_workers=0.
1@inproceedings{teklehaymanot2026vexmlm,
2 title = {Expanding the Lexicon of Ge'ez Based African Languages:
3 A Comparative Study of Amharic and Tigrinya},
4 author = {Teklehaymanot, Hailay Kidu and Yadeta, Gebregziabihier and
5 Nejdl, Wolfgang},
6 booktitle = {Proceedings of the Workshop on Language Models for
7 Underserved Communities (LM4UC) at IJCAI},
8 year = {2026}
9}
