Few things to keep in mind while training longformer for QA task,
by default longformer uses sliding-window local attention on all tokens. But For QA, all question tokens should have global attention. For more details on this please refer the paper. The LongformerForQuestionAnswering model automatically does that for you. To allow it to do that
The input sequence must have three sep tokens, i.e the sequence should be encoded like this
<s> question</s></s> context</s>. If you encode the question and answer as a input pair, then the tokenizer already takes care of that, you shouldn't worry about it.
input_ids should always be a batch of examples.
Results
Metric
# Value
Exact Match
85.1466
F1
91.5415
Model in Action 🚀
python
1import torch
2from transformers import AutoTokenizer, AutoModelForQuestionAnswering,34tokenizer = AutoTokenizer.from_pretrained("valhalla/longformer-base-4096-finetuned-squadv1")5model = AutoModelForQuestionAnswering.from_pretrained("valhalla/longformer-base-4096-finetuned-squadv1")67text ="Huggingface has democratized NLP. Huge thanks to Huggingface for this."8question ="What has Huggingface done ?"9encoding = tokenizer(question, text, return_tensors="pt")10input_ids = encoding["input_ids"]1112# default is local attention everywhere13# the forward method will automatically set global attention on question tokens14attention_mask = encoding["attention_mask"]1516start_scores, end_scores = model(input_ids, attention_mask=attention_mask)17all_tokens = tokenizer.convert_ids_to_tokens(input_ids[0].tolist())1819answer_tokens = all_tokens[torch.argmax(start_scores):torch.argmax(end_scores)+1]20answer = tokenizer.decode(tokenizer.convert_tokens_to_ids(answer_tokens))21# output => democratized NLP
The LongformerForQuestionAnswering isn't yet supported in pipeline . I'll update this card once the support has been added.
