SQuAD2.0 combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.
The model was trained on a Tesla P100 GPU and 25GB of RAM.
The script for fine tuning can be found
here1{
2 "exact": 78.80064010780762,
3 "f1": 82.22801347271162,
4 "total": 11873,
5 "HasAns_exact": 78.74493927125506,
6 "HasAns_f1": 85.60951483831069,
7 "HasAns_total": 5928,
8 "NoAns_exact": 78.85618166526493,
9 "NoAns_f1": 78.85618166526493,
10 "NoAns_total": 5945,
11 "best_exact": 78.80064010780762,
12 "best_exact_thresh": 0.0,
13 "best_f1": 82.2280134727116,
14 "best_f1_thresh": 0.0
15}
1from transformers import pipeline
2
3qa_pipeline = pipeline(
4 "question-answering",
5 model="mrm8488/spanbert-finetuned-squadv2",
6 tokenizer="mrm8488/spanbert-finetuned-squadv2"
7)
8
9qa_pipeline({
10 'context': "Manuel Romero has been working hardly in the repository hugginface/transformers lately",
11 'question': "Who has been working hard for hugginface/transformers lately?"
12
13})
14
15# Output: {'answer': 'Manuel Romero','end': 13,'score': 6.836378586818937e-09, 'start': 0}
