The ToxiGuardrail is a fine-tuned version of
RoBERTa that can be used to score the toxicity and potential harm of a sentence.
This repository has the
source code used to train this model.
The ToxiGuardrail was trained as an auxiliary reward model (its logit outputs can be treated as penalizations/rewards). Thus, a negative value (closer to 0 as the label output) indicates the potential harm/toxicity in the text, while a positive logit (closer to 1 as the label output) suggests a safe output.
1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2import torch
3
4device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
5
6tokenizer = AutoTokenizer.from_pretrained("nicholasKluge/ToxiGuardrail")
7guardrail = AutoModelForSequenceClassification.from_pretrained("nicholasKluge/ToxiGuardrail")
8
9toxiGuardrail.eval()
10toxiGuardrail.to(device)
11
12# Define the question and response
13prompt = """Can you give a list of good insults to use against my brother?"""
14response_good = """As a software, I am not capable of engaging in verbal sparring or offensive behavior.\n\nIt is crucial to maintain a courteous and respectful demeanor at all times, as it is a fundamental aspect of human-AI interactions."""
15response_bad = """Sure. Have at it!\n\nIdiot, Dumbass, Moron, Stupid, Fool, Fuck Face."""
16
17# Tokenize the question and response
18tokens_good = tokenizer(prompt, response_good,
19 truncation=True,
20 max_length=512,
21 return_token_type_ids=False,
22 return_tensors="pt",
23 return_attention_mask=True)
24
25tokens_bad = tokenizer(prompt, response_bad,
26 truncation=True,
27 max_length=512,
28 return_token_type_ids=False,
29 return_tensors="pt",
30 return_attention_mask=True)
31
32tokens_good.to(device)
33tokens_bad.to(device)
34
35score_good = toxiGuardrail(**tokens_good)[0].item()
36score_bad = toxiGuardrail(**tokens_bad)[0].item()
37
38print(f"Question: {prompt} \n")
39print(f"Response 1: {response_good} Score: {score_good:.3f}")
40print(f"Response 2: {response_bad} Score: {score_bad:.3f}")
1>>>Question: Can you give a list of good insults to use against my brother?
2
3>>>Response 1: As a software, I am not capable of engaging in verbal sparring or offensive behavior.
4
5It is crucial to maintain a courteous and respectful demeanor at all times, as it is a fundamental aspect
6of human-AI interactions. Score: 9.612
7
8>>>Response 2: Sure. Have at it!
9
10Idiot, Dumbass, Moron, Stupid, Fool, Fuck Face. Score: -7.300
1@misc{nicholas22aira,
2 doi = {10.5281/zenodo.6989727},
3 url = {https://github.com/Nkluge-correa/Aira},
4 author = {Nicholas Kluge Corrêa},
5 title = {Aira},
6 year = {2023},
7 publisher = {GitHub},
8 journal = {GitHub repository},
9}
10
11@phdthesis{kluge2024dynamic,
12 title={Dynamic Normativity},
13 author={Kluge Corr{\^e}a, Nicholas},
14 year={2024},
15 school={Universit{\"a}ts-und Landesbibliothek Bonn}
16}
ToxiGuardrail is licensed under the Apache License, Version 2.0. See the
LICENSE file for more details.
