BERT Fine-Tuned for Phishing vs Legitimate URLs
This model is a fine-tuned version of
ealvaradob/bert-finetuned-phishing on the
Gamortsey/url_defanged dataset.
It classifies
URLs (in defanged format) as either
phishing-like or
legitimate.
Dataset
The dataset contains phishing-like and legitimate URLs.
All phishing samples are defanged for safety (e.g., hxxps://example[.]com instead of https://example.com).
Columns
url_defanged (string) → Safe representation of a URL.label (int) →
1 → Phishing-like URL0 → Legitimate URL
Train / Validation Split
After preprocessing and splitting, the dataset was divided into:
1DatasetDict({
2 train: Dataset({
3 features: ['label', 'input_ids', 'attention_mask'],
4 num_rows: 1377
5 })
6 validation: Dataset({
7 features: ['label', 'input_ids', 'attention_mask'],
8 num_rows: 154
9 })
10})
🛠️ Training Details
- Base model:
ealvaradob/bert-finetuned-phishing
- Tokenizer: BERT tokenizer (WordPiece)
- Framework: Hugging Face Transformers + PyTorch
- Hardware: Google Colab (GPU)
- Training epochs: 3
- Batch size: (default from Trainer with padding)
- Optimizer: AdamW
Training Logs
[261/261 09:56, Epoch 3/3]
Step Training Loss
50 0.000000
100 0.000000
150 0.000000
200 0.000000
250 0.000000
- Final Training Loss: ~2.4e-07
- Final Validation Loss: ~1.2e-07
- Validation Accuracy: 100% (on held-out set)
📊 Results
| Metric | Value |
|---|
| Training Loss | 2.4e-07 |
| Validation Loss | 1.2e-07 |
| Validation Accuracy | 1.0 |
⚠️ Note: The dataset is relatively small (≈1.6k URLs). While accuracy reached 100% on validation, this may indicate overfitting. It is recommended to evaluate on a larger, more diverse test set for production use.
Second BERT Fine‑Tuned for Phishing URL Detection
Model Name: Gamortsey/bert-finetuned-phishing
Task: Binary classification (Phishing vs. Legitimate URLs)
Framework: Hugging Face Transformers
Base Model: bert-base-uncased (pretrained), further fine-tuned
Dataset: Gamortsey/Phishing_Urls
⭐ Overview
This model is a fine-tuned version of BERT designed to classify whether a given URL, SMS, or email contains a phishing URL or a legitimate (safe) URL. It is optimized for real-world situations such as bank scams, MoMo scams, delivery fraud, and impersonation attacks.
The goal of the fine-tuning was to:
- Improve the accuracy of detecting phishing URLs
- Handle noisy, real-world URL patterns
- Work effectively within SMS, email, and general text messages
- Support both short and long URLs
📊 Training Summary
| Metric | Value |
|---|
| Epochs | 3 |
| Training Loss | 0.0665 |
| Validation Loss | 0.000002 – 0.000004 |
| Accuracy | 100% |
| F1 Score | 1.00 |
The training metrics show extremely high performance (100% accuracy and F1). This is likely due to dataset repetition, meaning some samples appear in both training and validation sets.
🔎 Notes on Dataset Quality
The dataset used (Gamortsey/Phishing_Urls) contains:
- URL samples labeled as is_legitimate ("true" or "false")
- Some data duplication or repetition
- Some similar URL structures repeated in both splits
Becauseo f this, the evaluation metrics may appear inflated.
Future improvements include:
- Deduplication of repeated URLs
- Hard negative examples
- Mixed SMS + email + URL input
- More domain diversity
🛠 Training Details
- Optimizer: AdamW
- Learning Rate: 5e-5
- Batch Size: 16
- Max Sequence Length: 256
- Hardware: GPU (Google Colab)
- Training Library: Hugging Face Transformers Trainer API
Boolean Conversion: "true" → 1 (legitimate), "false" → 0 (phishing)
🧪 Evaluation Behavior
The fine-tuned model has been tested on synthetic phishing and legitimate URLs, including:
Correctly Identified as PHISHING
Correctly Identified as LEGITIMATE
The model generalizes well to real-world scam patterns such as:
- Suspicious TLDs (.cn, .tk, .xyz)
- Imitation of major brands
- URL structures containing "verify", "update", "security", etc.
🚀 Usage
1from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline
2
3model_name = "Gamortsey/bert-finetuned-defanged-url"
4tokenizer = AutoTokenizer.from_pretrained(model_name)
5model = AutoModelForSequenceClassification.from_pretrained(model_name)
6
7phishing_detector = pipeline("text-classification", model=model, tokenizer=tokenizer)
8
9example = "hxxps://secure-login.bankco[.]com/reset"
10
11
12print(phishing_detector(example))
13[{'label': 'LABEL_1', 'score': 0.9998}] # LABEL_1 → Phishing
Framework versions
- Transformers 4.56.1
- Pytorch 2.8.0+cu126
- Datasets 4.0.0
- Tokenizers 0.22.0
