This is an improved version of
AfriCOMET-STL (single task) evaluation model: It receives a triplet with (source sentence, translation, reference translation) and returns a score that reflects the quality of the translation compared to both source and reference.
Different from the original AfriCOMET-STL, this MT evaluation model is based on an improved African enhanced encoder,
afro-xlmr-large-76L, which leads better performance on African-related machine translation evaluation, verified in WMT 2024 Metrics Shared Task.
1pip install --upgrade pip # ensures that pip is current
2pip install unbabel-comet
1from comet import download_model, load_from_checkpoint
2
3model_path = download_model("masakhane/africomet-stl-1.1")
4model = load_from_checkpoint(model_path)
5data = [
6 {
7 "src": "Nadal sàkọọ́lẹ̀ ìforígbárí o ní àmì méje sóódo pẹ̀lú ilẹ̀ Canada.",
8 "mt": "Nadal's head to head record against the Canadian is 7–2.",
9 "ref": "Nadal scored seven unanswered points against Canada."
10 },
11 {
12 "src": "Laipe yi o padanu si Raoniki ni ere Sisi Brisbeni.",
13 "mt": "He recently lost against Raonic in the Brisbane Open.",
14 "ref": "He recently lost to Raoniki in the game Sisi Brisbeni."
15 }
16]
17model_output = model.predict(data, batch_size=8, gpus=1)
18print (model_output)
Given a triplet with (source sentence, translation, reference translation), it outputs a single score between 0 and 1 where 1 represents a perfect translation.
