SSA-COMET-MTL, a robust, unified automatic metric for both MTE and QE, built based on SSA-MTE: It receives a triplet with (source sentence, translation, reference translation) for MTE, or a pair with (source sentence, translation) for QE, and returns a score that reflects the quality of the translation.
This model is based on an improved African enhanced encoder,
afro-xlmr-large-76L.
Li S., Wang J., Ali F., Cherry C., Deutsch D., Briakou E., Sousa-Silva R., Cardoso H.L, Stenetorp P., and Adelani D.I.
SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?. In EMNLP 2025
1pip install --upgrade pip # ensures that pip is current
2pip install unbabel-comet
1from comet import download_model, load_from_checkpoint
2model_path = download_model("McGill-NLP/ssa-comet-mtl")
3model = load_from_checkpoint(model_path)
4data = [
5 {
6 "src": "Nadal sàkọọ́lẹ̀ ìforígbárí o ní àmì méje sóódo pẹ̀lú ilẹ̀ Canada.",
7 "mt": "Nadal's head to head record against the Canadian is 7–2.",
8 "ref": "Nadal scored seven unanswered points against Canada."
9 },
10 {
11 "src": "Laipe yi o padanu si Raoniki ni ere Sisi Brisbeni.",
12 "mt": "He recently lost against Raonic in the Brisbane Open.",
13 "ref": "He recently lost to Raoniki in the game Sisi Brisbeni."
14 }
15]
16model_output = model.predict(data, batch_size=8, gpus=1)
17print (model_output)
Given a triplet with (source sentence, translation, reference translation), or a pair with (source sentence, translation), it outputs a single score between 0 and 1, where 1 represents a perfect translation.
