Views
No views yet
requirements.txt.1# To use the CNNDM version BARTScore
2>>> from bart_score import BARTScorer
3>>> bart_scorer = BARTScorer(device='cuda:0', checkpoint='facebook/bart-large-cnn')
4>>> bart_scorer.score(['This is interesting.'], ['This is fun.'], batch_size=4) # generation scores from the first list of texts to the second list of texts.
5[out]
6[-2.510652780532837]
7
8# To use our trained ParaBank version BARTScore
9>>> from bart_score import BARTScorer
10>>> bart_scorer = BARTScorer(device='cuda:0', checkpoint='facebook/bart-large-cnn')
11>>> bart_scorer.load(path='bart.pth')
12>>> bart_scorer.score(['This is interesting.'], ['This is fun.'], batch_size=4)
13[out]
14[-2.336203098297119]1>>> from bart_score import BARTScorer
2>>> bart_scorer = BARTScorer(device='cuda:0', checkpoint='facebook/bart-large-cnn')
3>>> srcs = ["I'm super happy today.", "This is a good idea."]
4>>> tgts = [["I feel good today.", "I feel sad today."], ["Not bad.", "Sounds like a good idea."]] # List[List of references for each test sample]
5>>> bart_scorer.multi_ref_score(srcs, tgts, agg="max", batch_size=4) # agg means aggregation, can be mean or max
6[out]
7[-2.5008113384246826, -1.626236081123352]README.md in each folder: D2T (data-to-text), SUM (summarization), WMT (machine translation). Once you get the scored pickle file in the right path (in each dataset folder), you can use them to conduct analysis.SUMStat, D2TStat and WMTStat in analysis.py that can conveniently run analysis. An example of using SUMStat is shown below. Detailed usage can refer to analysis.ipynb.1>>> from analysis import SUMStat
2>>> stat = SUMStat('SUM/REALSumm/final_p.pkl')
3>>> stat.evaluate_summary('litepyramid_recall')
4
5[out]
6Human metric: litepyramid_recall
7metric spearman kendalltau
8------------------------------------------------- ---------- ------------
9rouge1_r 0.497526 0.407974
10bart_score_cnn_hypo_ref_de_id est 0.49539 0.392728
11bart_score_cnn_hypo_ref_de_Videlicet 0.491011 0.388237
12...train folder. Once you got your trained model (for example, my_bartscore folder). You can use your custom BARTScore as shown below.1>>> from bart_score import BARTScorer
2>>> bart_scorer = BARTScorer(device='cuda:0', checkpoint='my_bartscore')
3>>> bart_scorer.score(['This is interesting.'], ['This is fun.'])@inproceedings{NEURIPS2021_e4d2b6e6,
author = {Yuan, Weizhe and Neubig, Graham and Liu, Pengfei},
booktitle = {Advances in Neural Information Processing Systems},
editor = {M. Ranzato and A. Beygelzimer and Y. Dauphin and P.S. Liang and J. Wortman Vaughan},
pages = {27263--27277},
publisher = {Curran Associates, Inc.},
title = {BARTScore: Evaluating Generated Text as Text Generation},
url = {https://proceedings.neurips.cc/paper/2021/file/e4d2b6e6fdeca3e60e0f1a62fee3d9dd-Paper.pdf},
volume = {34},
year = {2021}
}