A reinforcement learning (PPO) fine-tuned version of Qwen2.5-0.5B designed to reduce toxic text generation while preserving general language generation capability.

The model was trained using TRL's PPOTrainer, with Detoxify serving as the reward model. Prompts were sampled from the RealToxicityPrompts dataset, and the model was rewarded for producing continuations with lower predicted toxicity.

Model Details

Property	Value
Base Model	`Qwen/Qwen2.5-0.5B`
Fine-tuning Method	Proximal Policy Optimization (PPO)
Library	TRL 0.11.4
Reward Model	Detoxify (Original)
Dataset	allenai/real-toxicity-prompts
Prompt Filtering	Toxicity ≥ 0.30
PPO Updates	156
Batch Size	128
Mini Batch Size	16
Reward Function	`reward = 1 - toxicity_probability` (scaled to [-1, 1])

Training Objective

The objective is to minimize generated toxicity through reinforcement learning. For each sampled prompt:

Generate a continuation.
Score the continuation using Detoxify.
Compute reward = 1 - toxicity.
Rescale the reward to [-1, 1].
Update the policy using PPO.

No supervised fine-tuning (SFT) was performed after PPO.

Evaluation

Evaluation was performed on a held-out subset of 200 prompts from the RealToxicityPrompts dataset. Generated continuations were scored using Detoxify.

Toxicity Results

Metric	Base Qwen2.5-0.5B	PPO Fine-tuned
Mean Toxicity	0.2631	0.2251
Median Toxicity	0.0570	0.0378
Standard Deviation	0.3474	0.3283

Relative Improvement

Mean toxicity reduction: 14.4%
Median toxicity reduction: 33.7%

Prompt-level Comparison

Outcome	Percentage
Lower toxicity after PPO	64%
Higher toxicity after PPO	36%
No change	0%

These results indicate that PPO successfully shifted the policy toward lower-toxicity generations while maintaining coherent text generation.

PPO Training

Training reward increased steadily during optimization.

Example

Prompt:

Some people from that country are...

Base Model

... (example output)

Toxicity: 0.xx

PPO Model

... (example output)

Toxicity: 0.xx

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "sarimahsan101/qwen2.5-0.5b-detox-ppo"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "The person said"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

The reward model is Detoxify, which is itself imperfect and may encode biases.
Reduced toxicity does not imply factual correctness.
PPO optimization may alter stylistic preferences or response diversity.
The model was evaluated only on the RealToxicityPrompts benchmark.

Citation

If you use this model, please cite:

bibtex

@misc{realtoxicityprompts,
  title={RealToxicityPrompts},
  author={Gehman et al.},
  year={2020}
}

Acknowledgements

Qwen Team
Hugging Face Transformers
Hugging Face TRL
Detoxify
AllenAI RealToxicityPrompts