QwenTox is a parameter-efficient multi-label toxic comment classification model built upon Qwen/Qwen3-0.6B-Base.
By integrating LoRA adapters with a lightweight multi-label classification head, the model is specifically designed to address the severe class imbalance problem commonly observed in toxic comment detection tasks.

The model supports six toxicity categories and emphasizes reproducibility, computational efficiency, and multilingual generalization.

Model Details

Task Description

Task type: Multi-label text classification
Domain: Toxic / abusive language detection
Input: User-generated text (comments)
Output: A 6-dimensional binary label vector

Each comment may belong to multiple toxicity categories or none.

Supported Labels

Label	Description
`toxic`	有毒
`severe_toxic`	严重有毒
`obscene`	淫秽
`threat`	威胁
`insult`	侮辱
`identity_hate`	身份仇恨

Model Architecture

Backbone: Qwen/Qwen3-0.6B-Base (Decoder-only Transformer)
Adaptation: LoRA (Low-Rank Adaptation)
Classifier: Lightweight linear multi-label classification head
Activation: Sigmoid (per-label probability)

Only the LoRA adapters and the classification head are trainable; all backbone parameters remain frozen.

Model Sources

Base Model:
https://huggingface.co/Qwen/Qwen3-0.6B-Base
Training Dataset:
Jigsaw Toxic Comment Classification

Getting Started

python

import torch
from transformers import AutoTokenizer, AutoModel
from peft import PeftModel

# Load base model
base_model = AutoModel.from_pretrained(
    "Qwen/Qwen3-0.6B-Base",
    trust_remote_code=True
)

# Load LoRA adapters
model = PeftModel.from_pretrained(
    base_model,
    "yingfeng64/QwenTox"
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    "Qwen/Qwen3-0.6B-Base",
    trust_remote_code=True
)

# Load classification head
state_dict = torch.load("classifier_head.pt", map_location="cpu")
model.classifier.load_state_dict(state_dict)

model.eval()

Training Details

Training Data

Dataset: Jigsaw Toxic Comment Classification
Label setting: Multi-label
Class distribution: Highly imbalanced
- Non-toxic : toxic ≈ 9 : 1
- Rare categories include threat, severe_toxic, and identity_hate

To mitigate data scarcity, translation-based data augmentation was applied to low-frequency categories.

Training Strategy

Fine-tuning scope:
- Trainable: LoRA adapters + classification head
- Frozen: Qwen3 backbone parameters
Loss function: Focal Loss
Training framework: PEFT (Parameter-Efficient Fine-Tuning)

Hyperparameters

Hyperparameter	Value
Precision	FP16 (mixed precision)
Optimizer	AdamW
Learning rate	5e-5
Epochs	3
Max sequence length	384
LoRA rank (r)	128
LoRA alpha	256
LoRA dropout	0.3

Evaluation

Evaluation Datasets

In-domain: Jigsaw Toxic Comment Test Set
Out-of-domain: Jigsaw Multilingual Toxic Comment Test Set (binary classification)

Evaluation Metrics

Subset Accuracy
Hamming Loss
Macro-F1
Macro-AUC

These metrics jointly evaluate both overall prediction consistency and performance on minority toxicity categories.

Intended Use

Academic research on toxic / abusive language detection
Experiments on parameter-efficient fine-tuning (LoRA, PEFT)
Multilingual and cross-domain generalization analysis

Limitations

Trained primarily on English data; multilingual performance depends on semantic transfer from the backbone model.
Rare toxicity categories remain challenging despite data augmentation.
Not designed for real-time moderation without further calibration.

License

This model is released under the GNU General Public License v3.0 (GPL-3.0).

Under this license:

You are free to use, modify, and redistribute this model.
Any derivative work or fine-tuned version based on this model must also be released under GPL-3.0.
Source code and model modifications must be made publicly available.

This ensures that improvements and downstream adaptations of the model remain open and accessible to the research community.