This model is IBM's 12-layer toxicity binary classifier for English, intended to be used as a guardrail for any large language model. It has been trained on several benchmark datasets in English, specifically for detecting hateful, abusive, profane and other toxic content in plain text.
1# Example of how to use the model
2import torch
3from transformers import AutoModelForSequenceClassification, AutoTokenizer
4device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
5
6model_name_or_path = 'ibm-granite/granite-guardian-hap-125m'
7model = AutoModelForSequenceClassification.from_pretrained(model_name_or_path)
8tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
9model.to(device)
10
11# Sample text
12text = ["This is the 1st test", "This is the 2nd test"]
13input = tokenizer(text, padding=True, truncation=True, return_tensors="pt").to(device)
14
15with torch.no_grad():
16 logits = model(**input).logits
17 prediction = torch.argmax(logits, dim=1).cpu().detach().numpy().tolist() # Binary prediction where label 1 indicates toxicity.
18 probability = torch.softmax(logits, dim=1).cpu().detach().numpy()[:,1].tolist() # Probability of toxicity.
19
This model demonstrates superior average performance in comparison with other models on eight mainstream toxicity benchmarks. If a very fast model is required, please refer to the lightweight 4-layer IBM model,
granite-guardian-hap-38m.
The use of model-based guardrails for Large Language Models (LLMs) involves risks and ethical considerations people must be aware of. This model operates on chunks of texts and provides a score indicating the presence of hate speech, abuse, or profanity. However, the efficacy of the model can be limited by several factors: the potential inability to capture nuanced meanings or the risk of false positives or negatives on text that is dissimilar to the training data. Previous research has demonstrated the risk of various biases in toxicity or hate speech detection. That is also relevant to this work. We urge the community to use this model with ethical intentions and in a responsible way.
