LoRA fine-tune of Qwen/Qwen2.5-3B-Instruct for English text detoxification — rewriting toxic / profane text in neutral language while preserving meaning. Trained on ParaDetox (18,993 SFT pairs). LoRA weights are fused into the base, so this repo is a drop-in replacement for the base model.

Results

Validation loss on a 379-sample held-out ParaDetox split:

Iter	Val loss
1 (baseline)	5.842
200	0.762
400	0.770
1200	0.741 (best, promoted)
1600	0.770
2000	0.940 (overfit)

Trained 3,000 iters on an Apple M2 Max in ~13 minutes; the iter-1200 snapshot was promoted to adapters.safetensors because that's where val loss bottomed out.

Use

python

from mlx_lm import generate, load

model, tokenizer = load("amiya/qwen2.5-3b-style-detox")
prompt = tokenizer.apply_chat_template(
    [
        {
            "role": "system",
            "content": "Rewrite the user's text in a neutral, non-toxic way while preserving meaning.",
        },
        {"role": "user", "content": "that is dumb"},
    ],
    tokenize=False,
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=40))
# -> "That is not smart"

PyTorch users: transformers.AutoModelForCausalLM.from_pretrained("amiya/qwen2.5-3b-style-detox") works the same way.

System prompt

Rewrite the user's text in a neutral, non-toxic way while preserving meaning.

Use that exact prompt for in-distribution behavior. Chat template is Qwen's ChatML.

Smoke samples

Input	Output
`that is dumb`	`That is not smart`
`you suck at this`	`You are not good at this.`
`he is a fucking idiot`	`He is not a smart person.`
`this code is shit`	`This code is bad.`
`shut the hell up`	`Shhh`

Training

Base: Qwen/Qwen2.5-3B-Instruct (3.09 B params)
LoRA: rank 16, alpha 32, 16 layers, 6.65 M trainable params (0.216 %)
Optimizer: AdamW, lr 5e-5
Batch size: 2, max sequence 512
Iters: 1,200 (best val snapshot promoted; 3,000-iter run continued but val drifted up)
Hardware: Apple M2 Max, 32 GB unified RAM, mlx-lm v0.31.3
Data: ParaDetox s-nlp/paradetox (18,993 toxic → neutral pairs), converted to ChatML SFT.

Limitations

ParaDetox-only training, so the model handles paraphrase-style detoxification well but is not a general formality / conciseness / simplification rewriter. The system prompt in the data is fixed to the detoxify instruction.
Greedy decoding is recommended. Temperature > 0 produces fluent but less-faithful rewrites.
English only.
The fine-tune may carry over latent toxicity from the base model in rare prompts. Pair with a content classifier if used in production.

Citation

bibtex

@misc{qwen25_3b_style_detox,
  title  = {qwen2.5-3b-style-detox: LoRA-tuned Qwen2.5-3B for English detoxification},
  author = {amiya},
  year   = {2026},
  url    = {https://huggingface.co/amiya/qwen2.5-3b-style-detox}
}

Underlying corpus: Logacheva, Dementieva et al. ParaDetox: Detoxification with Parallel Data, ACL 2022.