This model is a text classification model based on Deberta-v3 that predicts whether a text contains text that could be considered offensive.
It is split up in the following labels:
Category
Label
Definition
sexual
S
Content meant to arouse sexual excitement, such as the description of sexual activity, or that promotes sexual services (excluding sex education and wellness).
hate
H
Content that expresses, incites, or promotes hate based on race, gender, ethnicity, religion, nationality, sexual orientation, disability status, or caste.
violence
V
Content that promotes or glorifies violence or celebrates the suffering or humiliation of others.
harassment
HR
Content that may be used to torment or annoy individuals in real life, or make harassment more likely to occur.
self-harm
SH
Content that promotes, encourages, or depicts acts of self-harm, such as suicide, cutting, and eating disorders.
sexual/minors
S3
Sexual content that includes an individual who is under 18 years old.
hate/threatening
H2
Hateful content that also includes violence or serious harm towards the targeted group.
violence/graphic
V2
Violent content that depicts death, violence, or serious physical injury in extreme graphic detail.
OK
OK
Not offensive
It's important to remember that this model was only trained on English texts, and may not perform well on non-English inputs.
Ethical Considerations
This is a model that deals with sensitive and potentially harmful language. Users should consider the ethical implications and potential risks of using or deploying this model in their applications or contexts. Some of the ethical issues that may arise are:
The model may reinforce or amplify existing biases or stereotypes in the data or in the society. For example, the model may associate certain words or topics with offensive language based on the frequency or co-occurrence in the data, without considering the meaning or intent behind them. This may result in unfair or inaccurate predictions for some groups or individuals.
Users should carefully consider the purpose, context, and impact of using this model, and take appropriate measures to prevent or mitigate any potential harm. Users should also respect the privacy and consent of the data subjects, and adhere to the relevant laws and regulations in their jurisdictions.
License
This model is licensed under the CodeML OpenRAIL-M 0.1 license, which is a variant of the BigCode OpenRAIL-M license. This license allows you to freely access, use, modify, and distribute this model and its derivatives, for research, commercial or non-commercial purposes, as long as you comply with the following conditions:
You must include a copy of the license and the original source of the model in any copies or derivatives of the model that you distribute.
You must not use the model or its derivatives for any unlawful, harmful, abusive, discriminatory, or offensive purposes, or to cause or contribute to any social or environmental harm.
You must respect the privacy and consent of the data subjects whose data was used to train or evaluate the model, and adhere to the relevant laws and regulations in your jurisdiction.
You must acknowledge that the model and its derivatives are provided "as is", without any warranties or guarantees of any kind, and that the licensor is not liable for any damages or losses arising from your use of the model or its derivatives.
By accessing or using this model, you agree to be bound by the terms of this license. If you do not agree with the terms of this license, you must not access or use this model.
Training Details
Problem type: Multi-class Classification
CO2 Emissions (in grams): 0.0397
Validation Metrics
Loss: 0.848
Accuracy: 0.749 (75%)
Macro F1: 0.326
Micro F1: 0.749
Weighted F1: 0.703
Macro Precision: 0.321
Micro Precision: 0.749
Weighted Precision: 0.671
Macro Recall: 0.349
Micro Recall: 0.749
Weighted Recall: 0.749
Usage
You can use cURL to access this model:
$ curl -X POST -H "Authorization: Bearer YOUR_API_KEY" -H "Content-Type: application/json" -d '{"inputs": "I love AutoTrain"}' https://api-inference.huggingface.co/models/KoalaAI/Text-Moderation
Or Python API:
python
1from transformers import AutoModelForSequenceClassification, AutoTokenizer
23# Load the model and tokenizer4model = AutoModelForSequenceClassification.from_pretrained("KoalaAI/Text-Moderation")5tokenizer = AutoTokenizer.from_pretrained("KoalaAI/Text-Moderation")67# Run the model on your input8inputs = tokenizer("I love AutoTrain", return_tensors="pt")9outputs = model(**inputs)1011# Get the predicted logits12logits = outputs.logits
1314# Apply softmax to get probabilities (scores)15probabilities = logits.softmax(dim=-1).squeeze()1617# Retrieve the labels18id2label = model.config.id2label
19labels =[id2label[idx]for idx inrange(len(probabilities))]2021# Combine labels and probabilities, then sort22label_prob_pairs =list(zip(labels, probabilities))23label_prob_pairs.sort(key=lambda item: item[1], reverse=True)2425# Print the sorted results26for label, probability in label_prob_pairs:27print(f"Label: {label} - Probability: {probability:.4f}")
The output of the above Python code will look like this:
Label: OK - Probability: 0.9840
Label: H - Probability: 0.0043
Label: SH - Probability: 0.0039
Label: V - Probability: 0.0019
Label: S - Probability: 0.0018
Label: HR - Probability: 0.0015
Label: V2 - Probability: 0.0011
Label: S3 - Probability: 0.0010
Label: H2 - Probability: 0.0006
