This model is a fined-tuned version of MiniLMv2-L6-H384 on the on the Jigsaw 1st Kaggle competition dataset using unitary/toxic-bert as teacher model. The quantized version in ONNX format can be found here.

The model contains two labels only (toxicity and severe toxicity). For the model with all labels refer to this page

Load the Model

from transformers import pipeline

pipe = pipeline(model='minuva/MiniLMv2-toxic-jigsaw-lite', task='text-classification')
pipe("This is pure trash")
# [{'label': 'toxic', 'score': 0.887}]

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 6e-05
train_batch_size: 48
eval_batch_size: 48
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 10
warmup_ratio: 0.1

Metrics (comparison with teacher model)

Teacher (params)	Student (params)	Set (metric)	Score (teacher)	Score (student)
unitary/toxic-bert (110M)	MiniLMv2-toxic-jigsaw-lite (23M)	Test (ROC_AUC)	0.982677	0.9815

Deployment

Check our fast-nlp-text-toxicity repository for a FastAPI and ONNX based server to deploy this model on CPU devices.