Encoder-decoder models are trained typically by maximizing the likelihood of the target output given an input. To improve the capacities of both the encoder and decoder, we propose to train LongLM with two pretraining tasks including text inﬁlling (Raffel et al., 2020) and conditional continuation (Radford et al., 2019). For the ﬁrst task, the input is a text where a number of spans are sampled and replaced by special tokens with unique IDs, while the output is the spans delimited by the special tokens used in the input. The lengths of masked spans are drawn from a Poisson distribution with λ=3 and all masked tokens compress 15% of the original texts. As for the second task, the input and output are respectively the front and back half of a text, which is split into two parts randomly.

3. Pretraining Data

We collect 120G novels as the pretraining data for LongLM.

4. Checkpoints

Model Loading:

python\
1from transformers import T5Tokenizer, T5ForConditionalGeneration 2tokenizer = T5Tokenizer.from_pretrained('LongLM-large') 3tokenizer.add_special_tokens({"additional_special_tokens": ["<extra_id_%d>"%d for d in range(100)]}) 4model = T5ForConditionalGeneration.from_pretrained('LongLM-large')
Generation:

python
1input_ids = tokenizer("小咕噜对，<extra_id_1>",return_tensors="pt", padding=True, truncation=True, max_length=512).input_ids.to(device) 2 3gen = model.generate(input_ids, do_sample=True, decoder_start_token_id=1, top_p=0.9, max_length=512)

5. Dependencies

datasets                1.6.2
deepspeed               0.3.16
huggingface-hub         0.0.8
jieba                   0.42.1
jsonlines               2.0.0
nltk                    3.5
numpy                   1.19.5
pytorch-lightning       1.2.0
regex                   2020.11.13
rouge                   1.0.1
rouge-score             0.0.4
sacrebleu               1.5.0
scipy                   1.5.4
sentencepiece           0.1.95
tokenizers              0.10.1
torch                   1.8.1
torchaudio              0.8.0
torchmetrics            0.2.0
torchvision             0.9.0
transformers            4.6.1

6. Contributers

Jian Guan at thu-coai

Citation

txt

@misc{guan2021lot,
      title={LOT: A Benchmark for Evaluating Chinese Long Text Understanding and Generation}, 
      author={Jian Guan and Zhuoer Feng and Yamei Chen and Ruilin He and Xiaoxi Mao and Changjie Fan and Minlie Huang},
      year={2021},
      eprint={2108.12960},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

thu-coai/LongLM-base

LongLM

1. Parameters

2. Pretraining Tasks

3. Pretraining Data

4. Checkpoints

5. Dependencies

6. Contributers

Citation

thu-coai/
LongLM-base