A pruned version of Unbabel/wmt22-comet-da for faster, smaller machine-translation quality estimation with minimal accuracy loss.
Variant
Disk size
Params
Pearson vs full
MAE vs full
Original wmt22-comet-da
~2200 MB
580.9 M
1.0000
0.0000
This model
2122 MB
530.5 M
0.9827
0.0467
What was changed
An internal compression pipeline was applied to the base model to reduce size and inference cost while preserving evaluation quality. The pipeline preserves the regressor head and embeddings; only the encoder backbone is modified.
The bundled load.py handles base-model download, layer pruning, layerwise-attention reshaping, and (for the int8 variant) quantization automatically. First call downloads the original wmt22-comet-da (~2.2 GB cached); subsequent calls are instant.
Real-world benchmark — correlation with human DA scores
Evaluated on 1,200 segments from the WMT17 DA human-evaluation set (RicardoRei/wmt-da-human-evaluation), stratified across 12 language pairs (en↔cs, de, fi, ru, tr, zh). This is the standard way to measure COMET quality: how well do model scores correlate with human judgments?
Model
Disk
Pearson (human)
Spearman (human)
Agreement with full
Original wmt22-comet-da
2200 MB
0.6415
0.6724
1.000
This model
2122 MB
0.6181
0.6356
0.8904
Interpretation: this variant loses +0.0234 Pearson points of human correlation vs. the original. Good-vs-bad translation separation is preserved; fine-grained segment-level ranking is slightly degraded.
Tested on 22 src/mt/ref triples across 11 languages (fr, de, es, it, pt, ja, zh, ko, ar, ru, hi), each language contributing one good and one bad translation.
Metric
Value
Pearson vs full model
0.9827
Spearman vs full model
0.9827
Mean absolute score error
0.0467
Good > Bad discrimination
11/11 languages correct
Good-MT mean score (full → this model)
0.951 → 0.831
Bad-MT mean score (full → this model)
0.404 → 0.354
The model retains the full model's ability to separate good from bad translations across every language tested. See per_case_results.json for per-pair scores.
Notes
No fine-tuning was performed; weights are derived directly from the base model.
Tested on Apple M-series and x86 Linux.
Limitations
Internal evaluation set is small (22 multilingual pairs). On larger WMT22 test sets, expect quality close to the calibration correlations reported above.
Behavior outside the languages listed in the metadata is not guaranteed.
1@inproceedings{rei-etal-2022-comet,
2 title={{COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task}},
3 author={Rei, Ricardo and others},
4 booktitle={Proceedings of the Seventh Conference on Machine Translation},
5 year={2022}
6}
