Model Card: T5-base-summarization-claim-extractor
Model Description
Model Name: T5-base-summarization-claim-extractor
Authors: Alessandro Scirè, Karim Ghonim, and Roberto Navigli
Contact: scire@diag.uniroma1.it,
scire@babelscape.com
Language: English
Primary Use: Extraction of atomic claims from a summary
Overview
The T5-base-summarization-claim-extractor is a model developed for the task of extracting atomic claims from summaries. The model is based on the T5 architecture which is then fine-tuned specifically for claim extraction.
This model was introduced as part of the research presented in the paper
"FENICE: Factuality Evaluation of summarization based on Natural Language Inference and Claim Extraction" by Alessandro Scirè, Karim Ghonim, and Roberto Navigli. FENICE leverages Natural Language Inference (NLI) and Claim Extraction to evaluate the factuality of summaries.
ArXiv version.
Intended Use
This model is designed to:
- Extract atomic claims from summaries.
- Serve as a component in pipelines for factuality evaluation of summaries.
Example Code
1from transformers import T5ForConditionalGeneration, T5Tokenizer
2
3tokenizer = T5Tokenizer.from_pretrained("Babelscape/t5-base-summarization-claim-extractor")
4model = T5ForConditionalGeneration.from_pretrained("Babelscape/t5-base-summarization-claim-extractor")
5summary = 'Simone Biles made a triumphant return to the Olympic stage at the Paris 2024 Games, competing in the women’s gymnastics qualifications. Overcoming a previous struggle with the “twisties” that led to her withdrawal from events at the Tokyo 2020 Olympics, Biles dazzled with strong performances on all apparatus, helping the U.S. team secure a commanding lead in the qualifications. Her routines showcased her resilience and skill, drawing enthusiastic support from a star-studded audience'
6
7tok_input = tokenizer.batch_encode_plus([summary], return_tensors="pt", padding=True)
8claims = model.generate(**tok_input)
9claims = tokenizer.batch_decode(claims, skip_special_tokens=True)
10
Note: The model outputs the claims in a single string. Kindly remember to split the string into sentences in order to retrieve the singular claims.
Training
For details regarding the training process, please checkout our paper(
https://aclanthology.org/2024.findings-acl.841.pdf) (section 4.1).
Performance
Model | easinessP | easinessR | easinessF1 |
|---|
| GPT-3.5 | 80.1 | 70.9 | 74.9 |
| t5-base-summarization-claim-extractor | 79.2 | 68.8 | 73.4 |
Table 1: Easiness Precision (easiness
P), Recall (easiness
R), and F1 score (easiness
F1) results for the LLM-based claim extractor, namely
GPT-3.5, and
t5-base-summarization-claim-extractor, assessed on
ROSE (Liu et al., 2023b).
Further details on the model's performance and the metrics used can be found in the
paper (section 4.1).
Main Repository
For more details about FENICE, check out the GitHub repository:
Babelscape/FENICECitation
If you use this model in your work, please cite the following paper:
1
2@inproceedings{scire-etal-2024-fenice,
3 title = "{FENICE}: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction",
4 author = "Scir{\`e}, Alessandro and Ghonim, Karim and Navigli, Roberto",
5 editor = "Ku, Lun-Wei and Martins, Andre and Srikumar, Vivek",
6 booktitle = "Findings of the Association for Computational Linguistics ACL 2024",
7 month = aug,
8 year = "2024",
9 address = "Bangkok, Thailand and virtual meeting",
10 publisher = "Association for Computational Linguistics",
11 url = "https://aclanthology.org/2024.findings-acl.841",
12 pages = "14148--14161",
13}
Limitations
- The model is specifically designed for extracting claims from summaries and may not perform well on other types of texts.
- The model is currently available only in English and may not generalize well to other languages.
Ethical Considerations
Users should be aware that while this model extracts claims that can be evaluated for factuality, it does not determine the truthfulness of those claims. Therefore, it should be used in conjunction with other tools or human judgment when evaluating the reliability of summaries.
Acknowledgments
This work was made possible thanks to the support of Babelscape and Sapienza NLP.
