TiQuAD is a supplementary task in the VEXMLM paper — a diagnostic
evaluation, not a reported paper benchmark. Its 926 test questions make it a more
stable measurement than TIGQA's 67, and it is the stronger Tigrinya QA checkpoint
of the two.
Five-seed benchmark evaluation
Fine-tuned independently under seeds 42–46 with one configuration (hash
ce27cc194946) on an A100-PCIE-40GB. Reported as mean ± standard deviation over
the five runs, on the dataset's test split.
Metric
Score
Exact Match
50.24 ± 0.48
F1
58.90 ± 0.66
These are the paper's verified results. They come from the five-seed evaluation
described above — not from interactive use.
Interactive inference vs. benchmark
Benchmark evaluation is the five-seed measurement on the held-out test split,
shown in the table above.
Interactive inference is what the usage example below performs: Supply a Tigrinya context and question; the model returns an extracted span.
Predictions on arbitrary user input are demonstrations only and do not produce or
reproduce the benchmark score.
Repository layout
Five independently fine-tuned checkpoints, one per seed. The reported benchmark
score is the mean ± standard deviation over all five; no single seed is the
"five-seed model."
seed-42/ seed-43/ seed-44/ seed-45/ seed-46/
Load a specific seed with the subfolder argument, as in the example below.
Fine-tuning
Fine-tuned from Hailay/VEXMLM, a
vocabulary-extended XLM-R (280,002 subwords, 30,000 Ge'ez tokens merged into the
SentencePiece model) after continued MLM pretraining.
Hyperparameter
Value
Max sequence length
256
Batch size
32
Epochs
4
Learning rate
2e-5
LR schedule
Linear decay, 10% warmup
Weight decay
0.01
Gradient clipping
1.0
Optimizer
AdamW (β₁ 0.9, β₂ 0.999, ε 1e-8)
Precision
bf16
Trainable parameters
All
Hardware
1× NVIDIA A100
Runs are bit-reproducible: enable_full_determinism,
CUBLAS_WORKSPACE_CONFIG=:4096:8, dataloader_num_workers=0.
1@inproceedings{teklehaymanot2026vexmlm,
2 title = {Expanding the Lexicon of Ge'ez Based African Languages:
3 A Comparative Study of Amharic and Tigrinya},
4 author = {Teklehaymanot, Hailay Kidu and Yadeta, Gebregziabihier and
5 Nejdl, Wolfgang},
6 booktitle = {Proceedings of the Workshop on Language Models for
7 Underserved Communities (LM4UC) at IJCAI},
8 year = {2026}
9}
