Longformer

Model description

• Transformer-based models are unable to pro- cess long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer’s attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task moti- vated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language mod- eling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently out- performs RoBERTa on long document tasks and sets new state-of-the-art results on Wiki- Hop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Long- former variant for supporting long document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv sum- marization dataset.
• The original Transformer model has a self-attention component with O(n^2) time and memory complexity where n is the input sequence length. To address this challenge, we sparsify the full self-attention matrix according to an “attention pattern” specifying pairs of input locations attending to one another. Unlike the full self-attention, our proposed attention pattern scales linearly with the input sequence, making it efficient for longer sequences. This section discusses the design and implementation of this attention pattern.

Dataset and Task

Tokenizer with Vocabulary size

Computational Resources

Pretraining Objective

Training Setup

1. We train two model sizes, a base model and a large model. Both models are trained for 65K gradient updates with sequences length 4,096, batch size 64 (2 18 tokens), maximum learning rate of 3e-5, linear warmup of 500 steps, followed by a power 3 polynomial decay. The rest of the hyperparameters are the same as RoBERTa.[For MLM Pretraining]
2. Hyperparameters for the best performing model for character-level language modeling
3. Hyperparameters of the QA models. All mod- els use a similar scheduler with linear warmup and de- cay.
4. [For coreference resolution] The maximum se- quence length was 384 for RoBERTa-base, chosen after three trials from [256, 384, 512] using the default hyperparameters in the original implemen- tation.16 For Longformer-base the sequence length was 4,096.....
5. [For coreference resolution] .... Hyperparameter searches were minimal and con- sisted of grid searches of RoBERTa LR in [1e-5, 2e-5, 3e-5] and task LR in [1e-4, 2e-4, 3e-4] for both RoBERTa and Longformer for a fair compari- son. The best configuration for Longformer-base was RoBERTa lr=1e-5, task lr=1e-4. All other hy- perparameters were the same as in the original im- plementation.
6. [For text classification]

Training procedure

Preprocessing

Experiment

1. Character level langyage modeling: a) To compare to prior work we focus on character- level LM (text8 and enwik8; Mahoney, 2009).

1. Pretraining: a) We pretrain Longformer with masked language modeling (MLM), where the goal is to recover randomly masked tokens in a sequence.

1. WikiHop: Instances in WikiHop consist of: a question, answer candidates (ranging from two candidates to 79 candidates), supporting contexts (ranging from three paragraphs to 63 paragraphs), and the correct answer. The dataset does not pro- vide any intermediate annotation for the multihop reasoning chains, requiring models to instead infer them from the indirect answer supervision.
2. TriviaQA: TriviaQA has more than 100K ques- tion, answer, document triplets for training. Doc- uments are Wikipedia articles, and answers are named entities mentioned in the article. The span that answers the question is not annotated, but it is found using simple text matching.
3. HotpotQA: HotpotQA dataset involves answer- ing questions from a set of 10 paragraphs from 10 different Wikipedia articles where 2 paragraphs are relevant to the question and the rest are dis- tractors. It includes 2 tasks of answer span ex- traction and evidence sentence identification. Our model for HotpotQA combines both answer span extraction and evidence extraction in one joint model.
4. Coreference model: The coreference model is a straightforward adaptation of the coarse-to-fine BERT based model from Joshi et al. (2019).
5. Text classification: For classification, following BERT, we used a simple binary cross entropy loss on top of a first [CLS] token with addition of global attention to [CLS].
6. Evaluation metric for finetuned tasks: Summary of finetuning results on QA, coreference resolution, and document classification. Results are on the development sets comparing our Longformer-base with RoBERTa-base. TriviaQA, Hyperpartisan metrics are F1, WikiHop and IMDB use accuracy, HotpotQA is joint F1, OntoNotes is average F1.
7. Summarization: a) We evaluate LED on the summarization task us- ing the arXiv summarization dataset (Cohan et al.) which focuses on long document summariza- tion in the scientific domain.

Ablation

BibTeX entry and citation info