KinyCOMET is a neural translation quality estimation model for Kinyarwanda-English translation pairs. The model addresses the poor correlation between BLEU scores and human judgment in Kinyarwanda translation evaluation, achieving 0.75 Pearson correlation with human assessments

The model was trained on 4,323 human-annotated translation pairs collected from 15 linguistics students using Direct Assessment scoring aligned with WMT evaluation standards.

Model Variants & Performance

Variant	Base Model	Pearson	Spearman	Kendall's τ	MAE
KinyCOMET-Unbabel	Unbabel/wmt22-comet-da	0.75	0.59	0.42	0.07
KinyCOMET-XLM	XLM-RoBERTa-large	0.73	0.50	0.35	0.07
Unbabel (baseline)	wmt22-comet-da	0.54	0.55	0.39	0.17
AfriCOMET STL 1.1	AfriCOMET base	0.52	0.35	0.24	0.18
BLEU	N/A	0.30	0.34	0.23	0.62
chrF	N/A	0.38	0.30	0.21	0.34

Both KinyCOMET variants outperform existing baselines. KinyCOMET-Unbabel shows the strongest overall correlation, while performance varies by translation direction:

Performance Highlights

Comprehensive Evaluation Results

Overall Performance (Both Directions)

Pearson Correlation: 0.75 (KinyCOMET-Unbabel) vs 0.30 (BLEU) - 2.5x improvement
Spearman Correlation: 0.59 vs 0.34 (BLEU) - 73% improvement
Mean Absolute Error: 0.07 vs 0.62 (BLEU) - 89% reduction

Directional Analysis

Direction	Model	Pearson	Spearman	Kendall's τ
English → Kinyarwanda	KinyCOMET-XLM	0.76	0.52	0.37
English → Kinyarwanda	KinyCOMET-Unbabel	0.75	0.56	0.40
Kinyarwanda → English	KinyCOMET-Unbabel	0.63	0.47	0.33
Kinyarwanda → English	KinyCOMET-XLM	0.37	0.29	0.21

Key Insights:

English→Kinyarwanda consistently outperforms Kinyarwanda→English across all metrics
Both KinyCOMET variants significantly outperform AfriCOMET baselines despite including Kinyarwanda
Surprising finding: Unbabel baseline (not trained on Kinyarwanda) outperforms AfriCOMET variants

Installation

Make sure you have Python ≥ 3.8 and install COMET via pip:

pip install unbabel-comet

You can verify the CLI tool is installed:

bash

which comet-score
# should print something like: /usr/local/bin/comet-score

For more details on COMET, see the official documentation.

Usage

Load and Use the Model in Python

Here's a simple example to score translations directly in Python:

python

from comet import load_from_checkpoint

# Load the public KinyCOMET model
model = load_from_checkpoint("chrismazii/kinycomet_unbabel")

# Example translations
samples = [
    {
        "src": "Umugabo ararya.",
        "mt": "The man is eating.",
        "ref": "The man is eating."
    },
    {
        "src": "Umwana arasinzira.",
        "mt": "A dog sleeps.",
        "ref": "The child is sleeping."
    }
]

# Predict scores
pred = model.predict(samples, gpus=0)
print(pred)

Output Example:

python

Prediction({
  'scores': [0.9899, 0.8813],
  'system_score': 0.9356
})

Using the Command Line Interface (CLI)

You can also evaluate translations directly using the terminal.

Step 1: Create the text files

bash

cat > source.txt <<'SRC'
Umugabo ararya.
Umwana arasinzira.
Uyu mwanya neza cyane.
SRC

cat > reference.txt <<'REF'
The man is eating.
The child is sleeping.
This place is very nice.
REF

cat > hypothesis.txt <<'HYP'
The man is eating.
A dog sleeps.
This place is very nice.
HYP

Step 2: Run KinyCOMET

bash

comet-score -s source.txt -r reference.txt -t hypothesis.txt \
  --model chrismazii/kinycomet_unbabel --gpus 0 --to_json results.json

Step 3: View the results

cat results.json

Score Interpretation

Scores range from 0 to 1: Higher scores indicate better translation quality
System score: Average quality across all translations
Segment scores: Individual quality scores for each translation pair
Threshold guidance: Scores above 0.8 typically indicate high-quality translations

Training Details

Data

4,323 human-annotated Kinyarwanda-English translation pairs
Annotations collected from 15 linguistics students
Direct Assessment scoring following WMT standards
Split: 80% train (3,497) / 10% validation (404) / 10% test (422)
Domains: education and tourism

Model Architecture

Base Models: XLM-RoBERTa-large and Unbabel/wmt22-comet-da
Framework: COMET quality estimation framework
Evaluation metrics: Kendall's τ and Spearman ρ correlation with human DA scores

Training Configuration

Methodology: COMET framework with Direct Assessment supervision
Evaluation Metrics: Kendall's τ and Spearman ρ correlation with human DA scores
Data Split: 80% train (3,497) / 10% validation (404) / 10% test (422)

MT System Benchmarking Results

We evaluated several production MT systems using KinyCOMET:

MT System	Kinyarwanda→English	English→Kinyarwanda	Overall
GPT-4o	93.10% ± 7.77	87.83% ± 11.15	90.69% ± 9.82
GPT-4.1	93.08% ± 6.62	87.92% ± 10.38	90.75% ± 8.90
Gemini Flash 2.0	91.46% ± 11.39	90.02% ± 8.92	90.80% ± 10.35
Claude 3.7	92.48% ± 8.32	85.75% ± 11.28	89.43% ± 10.33
NLLB-1.3B	89.42% ± 12.04	83.96% ± 16.31	86.78% ± 14.52
NLLB-600M	88.87% ± 12.11	75.46% ± 28.49	82.71% ± 22.27

Key Findings:

LLM-based systems significantly outperform traditional neural MT
All systems perform better on Kinyarwanda→English than English→Kinyarwanda

Dataset Access

The training dataset is available separately. See the KinyCOMET Dataset Card for details on accessing the human-annotated quality estimation data.

Citation & Research

If you use KinyCOMET in your research, please cite:

bibtex

@misc{kinycomet2025,
    title={KinyCOMET: Translation Quality Estimation for Kinyarwanda-English},
    author={Prince Chris Mazimpaka and Jan Nehring},
    year={2025},
    publisher={Hugging Face},
    howpublished={\url{https://huggingface.co/chrismazii/kinycomet_unbabel}}
}

License

This model is released under the Apache 2.0 License.

Acknowledgments

COMET Framework: Built on the excellent COMET quality estimation framework
Base Models: Leverages XLM-RoBERTa and Unbabel's WMT22 COMET-DA models
African NLP Community: Inspired by ongoing efforts to advance African language technologies
Contributors: Thanks to the 15 linguistics students and all researchers who made this work possible

Resources: