Model Card for toxishield
A bert-base-uncased model fine-tuned for binary toxicity classification (TOXIC / NON-TOXIC) of GitHub pull request review comments.
Model Details
Model Description
This model classifies a single GitHub pull request comment as toxic or non-toxic. It was fine-tuned from google-bert/bert-base-uncased on 38,761 labelled PR comments (the "38k detection dataset") as part of the ToxiShield project, which studies and filters toxicity in software engineering communication. The model is evaluated with stratified 10-fold cross-validation and also exported to ONNX (INT8-quantized) for lightweight/in-browser inference.
- Developed by: [More Information Needed]
- Funded by [optional]: [More Information Needed]
- Shared by [optional]: [More Information Needed]
- Model type: Transformer encoder (BERT), fine-tuned for binary sequence classification
- Language(s) (NLP): English (software-engineering / code-review domain text)
- License: [More Information Needed] (base model
google-bert/bert-base-uncased is released under Apache-2.0; license for the fine-tuned weights has not been set)
- Finetuned from model [optional]:
google-bert/bert-base-uncased
Model Sources [optional]
- Repository:
huggingface.co/imranraad/toxishield
- Paper [optional]: https://arxiv.org/abs/2604.14408 (FSE 2026)
Uses
Direct Use
Classifying individual GitHub PR/code-review comments as TOXIC or NON-TOXIC, e.g. via transformers.pipeline("text-classification", ...), to flag potentially toxic comments for human review.
Downstream Use [optional]
Integration into CI bots, code-review dashboards, or moderation tooling that triages or surfaces potentially toxic PR comments before/alongside human moderators. The ONNX INT8 export is intended for low-latency or in-browser inference in such tooling.
Out-of-Scope Use
- Not intended for languages other than English.
- Not validated on text outside the GitHub PR/code-review domain (e.g. social media, forums, chat, general web text) — the training distribution is short technical comments (median ~80 characters).
- Not intended as the sole basis for moderation, disciplinary, employment, or legal decisions — outputs should be reviewed by a human.
- Not a general-purpose toxicity/hate-speech classifier.
Bias, Risks, and Limitations
- Class imbalance: the training data is 73.9% non-toxic / 26.1% toxic (28,641 vs 10,120 of 38,761 comments), which can bias the model toward the majority (non-toxic) class.
- Recall on the toxic class is the weaker metric: across 10-fold CV, mean recall on toxic comments is 0.954 (± 0.010) vs. mean precision of 0.975 (± 0.003) — i.e. the model is somewhat more likely to miss a toxic comment than to wrongly flag a non-toxic one.
- Manual review of false positives (n=23 sampled) attributes most errors to:
- Nuance/context misreads, e.g. sarcasm, dry humor, mockery (12/23, ~52%)
- Technical jargon or inline code snippets read as hostile (6/23, ~26%)
- Self-deprecating humor (2/23, ~9%)
- General negative sentiment without toxicity (2/23, ~9%)
- The model was fine-tuned and evaluated on data from GitHub PR comments only, so its calibration will not necessarily transfer to other codebase-hosting platforms or review tools.
Recommendations
Use as a triage/assistive signal rather than an automated blocking mechanism, particularly given the lower recall on the toxic class. Expect false positives on sarcastic, self-deprecating, or jargon-heavy comments, and route model decisions through human review before any moderation action.
How to Get Started with the Model
Use the code below to get started with the model.
1from transformers import pipeline
2
3classifier = pipeline("text-classification", model="<path-or-hub-id-of-saved-model>")
4classifier("this the stupidest code ever")[0]["label"]
Training Details
Training Data
38,761 labelled GitHub PR review comments (38k-detection-dataset, dataset handle [ANONYMIZED-ORG]/38k-dataset-labelled, not yet published): 28,641 NON-TOXIC (label 0) and 10,120 TOXIC (label 1). Comment length ranges from 5 to 998 characters (median ~80). Split into an 80% train / 10% test CSV for the single-run fine-tune; the full dataset is additionally used for stratified 10-fold cross-validation.
Training Procedure
Preprocessing [optional]
Comments are tokenized with the bert-base-uncased WordPiece tokenizer with truncation. The single-run fine-tune uses dynamic padding (DataCollatorWithPadding); the 10-fold cross-validation run uses fixed max_length=128 padding.
Training Hyperparameters
- Single-run fine-tune: learning rate 2e-5, per-device batch size 16, weight decay 0.01, 1 epoch, evaluate/save every epoch, best checkpoint restored at the end.
- 10-fold cross-validation run: learning rate 2e-5, per-device batch size 16, gradient accumulation 8 (effective batch size 128), weight decay 0.01, up to 20 epochs with early stopping (patience 3), stratified 10-fold split (each fold further split ~89/11 into train/validation).
- Training regime: fp16 mixed precision on GPU when available, fp32 on CPU.
Speeds, Sizes, Times [optional]
bert-base-uncased has ~110M parameters. The best checkpoint is additionally exported to ONNX and INT8-quantized for lighter-weight/in-browser inference. [More Information Needed] on wall-clock training time.
Evaluation
Testing Data, Factors & Metrics
Testing Data
Stratified 10-fold cross-validation over the full 38,761-sample dataset (per-fold results in results/kfold-metrics/cross_validation_results.csv); the single-run fine-tune is additionally evaluated on the held-out 10% test split.
Factors
No subpopulation disaggregation performed; results are reported per cross-validation fold and averaged.
Metrics
Accuracy, precision, recall, and F1 (binary, positive class = TOXIC), chosen to capture both overall correctness and toxic-class-specific performance given the class imbalance.
Results
10-fold cross-validation (mean ± std over 10 folds):
| Metric | Mean | Std |
|---|
| Accuracy | 0.9818 | 0.0023 |
| Precision (toxic) | 0.9753 | 0.0033 |
| Recall (toxic) | 0.9543 | 0.0096 |
| F1 (toxic) | 0.9647 | 0.0047 |
Baseline comparison — GPT-4o, zero-shot prompted, on the held-out test split (comparison/openai-detection-inference/):
| Class | Precision | Recall | F1 |
|---|
| Non-toxic | 0.84 | 0.99 | 0.91 |
| Toxic | 0.96 | 0.49 | 0.65 |
| Accuracy | | | 0.86 |
Summary
The fine-tuned BERT model substantially outperforms zero-shot GPT-4o prompting on this task, most notably on toxic-class recall (0.95 vs. 0.49) — GPT-4o zero-shot misses roughly half of toxic comments, while the fine-tuned model catches the large majority at comparable or better precision.
Model Examination [optional]
[More Information Needed]
Environmental Impact
Carbon emissions can be estimated using the
Machine Learning Impact calculator presented in
Lacoste et al. (2019).
- Hardware Type: 2× NVIDIA RTX 6000 Ada Generation (49 GB), local workstation
- Hours used: [More Information Needed]
- Cloud Provider: N/A (local/on-prem)
- Compute Region: [More Information Needed]
- Carbon Emitted: [More Information Needed]
Technical Specifications [optional]
Model Architecture and Objective
bert-base-uncased (12-layer transformer encoder, 110M parameters) with a linear classification head over 2 labels (NON-TOXIC = 0, TOXIC = 1), fine-tuned with cross-entropy loss for binary sequence classification.
Compute Infrastructure
Hardware
2× NVIDIA RTX 6000 Ada Generation (49 GB each).
Software
Python 3.11, PyTorch 2.5.1 (CUDA 12.1 build), Transformers 4.57.6, Datasets 5.0.0, 🤗 Evaluate, scikit-learn, Optimum/ONNX Runtime (for INT8 export).
Citation [optional]
BibTeX:
[More Information Needed]
APA:
[More Information Needed]
Glossary [optional]
- PR: Pull request (GitHub code-review unit).
- TOXIC / NON-TOXIC: The two output labels (id 1 / id 0 respectively).
More Information [optional]
[More Information Needed]
Model Card Authors [optional]
[More Information Needed]
Model Card Contact
[More Information Needed]
