Views
No views yet
We use years 2017 to 2019 of the WMT Metrics Shared Task, to-English language pairs. For each year, we used the of- ficial WMT test set, which include several thou- sand pairs of sentences with human ratings from the news domain. The training sets contain 5,360, 9,492, and 147,691 records for each year.
1@inproceedings{sellam2020bleurt,
2 title = {BLEURT: Learning Robust Metrics for Text Generation},
3 author = {Thibault Sellam and Dipanjan Das and Ankur P Parikh},
4 year = {2020},
5 booktitle = {Proceedings of ACL}
6}1from transformers import AutoModelForSequenceClassification, AutoTokenizer
2import torch
3
4tokenizer = AutoTokenizer.from_pretrained("Elron/bleurt-tiny-512")
5model = AutoModelForSequenceClassification.from_pretrained("Elron/bleurt-tiny-512")
6model.eval()
7
8references = ["hello world", "hello world"]
9candidates = ["hi universe", "bye world"]
10
11with torch.no_grad():
12 scores = model(**tokenizer(references, candidates, return_tensors='pt'))[0].squeeze()
13
14print(scores) # tensor([-0.9414, -0.5678])