This model is a fine-tuned version of allenai/led-large-16384 on the BookSum dataset (kmfoda/booksum). It aims to generalize well and be useful in summarizing lengthy text for both academic and everyday purposes.
Handles up to 16,384 tokens input
See the Colab demo linked above or try the demo on Spaces
Note: Due to inference API timeout constraints, outputs may be truncated before the fully summary is returned (try python or the demo)
Basic Usage
To improve summary quality, use encoder_no_repeat_ngram_size=3 when calling the pipeline object. This setting encourages the model to utilize new vocabulary and construct an abstractive summary.
Important: For optimal summary quality, use the global attention mask when decoding, as demonstrated in this community notebook, see the definition of generate_answer(batch).
The model was fine-tuned on the booksum dataset. During training, the chapterwas the input col, while the summary_text was the output.
Procedure
Fine-tuning was run on the BookSum dataset across 13+ epochs. Notably, the final four epochs combined the training and validation sets as 'train' to enhance generalization.
Hyperparameters
The training process involved different settings across stages:
Initial Three Epochs: Low learning rate (5e-05), batch size of 1, 4 gradient accumulation steps, and a linear learning rate scheduler.
In-between Epochs: Learning rate reduced to 4e-05, increased batch size to 2, 16 gradient accumulation steps, and switched to a cosine learning rate scheduler with a 0.05 warmup ratio.
Final Two Epochs: Further reduced learning rate (2e-05), batch size reverted to 1, maintained gradient accumulation steps at 16, and continued with a cosine learning rate scheduler, albeit with a lower warmup ratio (0.03).
Versions
Transformers 4.19.2
Pytorch 1.11.0+cu113
Datasets 2.2.2
Tokenizers 0.12.1
Simplified Usage with TextSum
To streamline the process of using this and other models, I've developed a Python package utility named textsum. This package offers simple interfaces for applying summarization models to text documents of arbitrary length.
Install TextSum:
pip install textsum
Then use it in Python with this model:
python
1from textsum.summarize import Summarizer
23model_name ="pszemraj/led-large-book-summary"4summarizer = Summarizer(5 model_name_or_path=model_name,# you can use any Seq2Seq model on the Hub6 token_batch_length=4096,# tokens to batch summarize at a time, up to 163847)8long_string ="This is a long string of text that will be summarized."9out_str = summarizer.summarize_string(long_string)10print(f"summary: {out_str}")
Currently implemented interfaces include a Python API, a Command-Line Interface (CLI), and a demo/web UI.
For detailed explanations and documentation, check the README or the wiki
Related Models
Check out these other related models, also trained on the BookSum dataset:
