This repository contains the MetricX-23 models, a family of models for automatic evaluation of translations that were proposed in the WMT'23 Metrics Shared Task submission MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task. The models were trained in T5X and then converted for use in PyTorch.

There are 6 models available on HuggingFace that vary in the number of parameters and whether or not the model is reference-based or reference-free (also known as quality estimation, or QE):

We recommend using the XXL model versions for the best agreement with human judgments of translation quality, the Large versions for best speed, and the XL for an intermediate use case.

These models available here are most similar to the primary submission to the WMT'23 Metrics Shared Task. They are initialized with mT5 then fine-tuned on a combination of direct assessment and MQM data. However, we made some changes that make these models different from the WMT'23 submissions.

First, the models are trained to regress the actual MQM score rather than a normalized score between 0 and 1. That means the output from the MetricX-23 models is a score in the range [0, 25] where lower is better (i.e., it predicts an error score).

Second, these models were trained with a larger variety of synthetic data that makes them more robust to translation edge cases like over- and undertranslation, described in more detail in the following section.

In order for our MetricX models to learn to identify certain types of bad translations that are not sufficiently (or at all) represented in the regular training data, we created synthetic examples and mixed them in during training. The synthetic training data was generated from the DA datasets ranging from WMT15 to WMT21 (~ 43 language pairs). In most cases, the synthetic examples have the candidate translation manipulated so as to turn it into a bad translation with a specific issue commonly unrecognized by learned metrics.

The table below provides an overview of the various failure modes that we considered, including brief descriptions of how we prepared the synthetic data to address them.

Examples from the first 4 categories were assigned a label corresponding to the worst score on the given rating scale (e.g., 25 when mixed with MQM training data), whereas the reference-matching translation examples are assigned the best score (e.g., 0 when used with MQM data). The missing/incorrect punctuation examples were labeled with a score slightly worse than perfect.

Note that some of the synthetic datasets are only meaningful in the reference-based scenario, and we thus excluded them when training a QE variant of MetricX. These are the Latin-vs-special punctuation and the reference-matching translation examples.

Most of the synthetic training sets were created using stratified sampling across target languages, taking 500 examples per target language. One exception is the missing punctuation set, which used a stratified sample across different punctuation symbols instead.

When training MetricX, a small proportion of the synthetic examples was mixed with the regular training examples. During the first-stage fine-tuning on DA data, each synthetic training set constituted between 0.1% and 1% of all training examples, whereas in the second-stage fine-tuning on MQM data we used an even smaller proportion, around 0.05%.

As for evaluating the effect of the synthetic training data on the model's performance, the DEMETR challenge set - which we originally used to evaluate the models submitted to the WMT23 Metrics Shared Task - was not adequate anymore. We therefore created a new DEMETR-style test set based on the WMT22 DA data, with examples constructed analogically to the synthetic training examples, as described above. This test set helped us determine the right proportions of synthetic data for fine-tuning in order to make MetricX robust for the failure modes in consideration, without sacrificing the system- and segment-level correlations with human ratings.

The code for using MetricX models can be found at https://github.com/google-research/metricx. The repository contains example prediction scripts, described below.

The metricx23/predict.py script contains an example for how to run inference on the models.

Example usage for a reference-based model:

input.jsonl is expected to have 1 serialized JSON object per line with "reference" and "hypothesis" fields. The output jsonl will be parallel to input.jsonl but additionally contain a "prediction" field with the predicted score.

Note that the model was trained with a maximum input length of 1024 tokens, so significantly increasing that value may lead to unpredictable behavior.

Example usage for a reference-free model:

input.jsonl is expected to have 1 serialized JSON object per line with "source" and "hypothesis" fields. The output jsonl will be parallel to input.jsonl but additionally contain a "prediction" field with the predicted score.

The metricx23/evaluate.py script contains code to calculate various correlations between the MetricX-23 scores and MQM ratings of translation quality using the MT Metrics Eval library.

Example usage:

input.jsonl is expected to have one JSON object serialized per line. Each JSON object is expected to contain 4 fields:

The script will calculate the 4 agreement/correlations that were used in the WMT'23 Shared Task. Below are the results for the MetricX-23 models on the WMT'22 Metrics Shared Task data:

English-German:

English-Russian:

Chinese-English:

The metricx23/evaluate_wmt23.py script re-calculates the average correlation score that was used to rank submissions from the WMT'23 Shared Task.

Example usage:

Each of the 3 input files is expected to be in the same format as described above. Each file should correspond to running inference on each of the language pairs from the WMT'23 dataset.

The results for each of the models is the following:

If you use MetricX-23 in your research, please cite the following publication: