This is a source-only COMET model used for efficient evaluation subset selection. Specifically this model predicts consistency of the system ordering based on a single segment being the same as the system ordering on the whole test-set. The higher the scores, the better it is for evaluation because then fewer samples will be needed to arrive at the same system ordering. It is not compatible with the original Unbabel's COMET and to run it you have to install github.com/zouharvi/PreCOMET:

You can then use it in Python:

The primary use of this model is from the subset2evaluate package:

Random selection gives us only one cluster and system-level Spearman correlation of 0.71 when we have a budget for only 100 segments. However, by using this model:

we get higher correlation. You can expect a bigger effect on a larger scale, as described in the paper.

This work is described in How to Select Datapoints for Efficient Human Evaluation of NLG Models?. Cite as:

@misc{zouhar2025selectdatapointsefficienthuman,
    title={How to Select Datapoints for Efficient Human Evaluation of NLG Models?}, 
    author={Vilém Zouhar and Peng Cui and Mrinmaya Sachan},
    year={2025},
    eprint={2501.18251},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2501.18251}, 
}