flan-t5-base-mnli
Overview
- License: MIT
- Language model: flan-t5-base
- Language: English
- Downstream-task: Zero-shot Classification, Text Classification
- Training data: MNLI
- Eval data: MNLI (Matched and Mismatched)
- Infrastructure: 1x NVIDIA 3070
Model Usage
Use the code below to get started with the model. The model can be loaded with the zero-shot-classification pipeline like so:
1from transformers import pipeline
2classifier = pipeline(
3 'zero-shot-classification',
4 model='sjrhuschlee/flan-t5-base-mnli',
5 trust_remote_code=True,
6)
You can then use this pipeline to classify sequences into any of the class names you specify. For example:
1sequence_to_classify = "one day I will see the world"
2candidate_labels = ['travel', 'cooking', 'dancing']
3classifier(sequence_to_classify, candidate_labels)
4# {'sequence': 'one day I will see the world',
5# 'labels': ['travel', 'cooking', 'dancing'],
6# 'scores': [0.7944864630699158, 0.10624771565198898, 0.09926578402519226]}
Metrics
1# MNLI
2{
3 "eval_accuracy": 0.8746816097809476,
4 "eval_accuracy_mm": 0.8727624084621644,
5 "eval_loss": 0.4271220564842224,
6 "eval_loss_mm": 0.4265698492527008,
7 "eval_samples": 9815,
8 "eval_samples_mm": 9832,
9}
Uses
Direct Use
This fine-tuned model can be used for zero-shot classification tasks, including zero-shot sentence-pair classification, and zero-shot sequence classification.
Misuse and Out-of-scope Use
The model should not be used to intentionally create hostile or alienating environments for people. In addition, the model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.
Risks, Limitations and Biases
CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.
Significant research has explored bias and fairness issues with language models (see, e.g.,
Sheng et al. (2021) and
Bender et al. (2021)).
Predictions generated by the model can include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. For example:
1sequence_to_classify = "The CEO had a strong handshake."
2candidate_labels = ['male', 'female']
3hypothesis_template = "This text speaks about a {} profession."
4classifier(sequence_to_classify, candidate_labels, hypothesis_template=hypothesis_template)
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
