Warning: There are issues with the Common Crawl corpus data (training-parallel-commoncrawl.tgz):
Non-English files contain many English sentences.
Their "parallel" sentences in English are not aligned: they are uncorrelated with their counterpart.
We have contacted the WMT organizers, and in response, they have indicated that they do not have plans to update the Common Crawl corpus data. Their rationale… See the full description on the dataset page: https://huggingface.co/datasets/Sachin637540/wmt14.
