Quick human evaluation of summarization quality: the results are generally good, after visual inspection of the summaries generated on test set conversations. However it seems some entities/attributions are incorrect (saw an example where model confuses peoples' roles in multi-person chat)

Model description

PEGASUS doc can be found here: https://huggingface.co/docs/transformers/model_doc/pegasus

Intended uses & limitations

This model was trained while studying the NLP With Transformers book; it is not intended to be used for any real applications.

Training and evaluation data

The finetuning data is the SAMSum dataset only.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 1
eval_batch_size: 1
seed: 42
gradient_accumulation_steps: 16
total_train_batch_size: 16
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 500
num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
1.6592	0.54	500	1.4839

Framework versions

Transformers 4.39.3
Pytorch 2.1.2
Datasets 2.18.0
Tokenizers 0.15.2

benjaminzwhite/pegasus-samsum-nlp-with-transformers-ch06

pegasus-samsum-nlp-with-transformers-ch06