Model Card for PhishingDistilBERT
Model Summary
PhishingDistilBERT is a DistilBERT-based NLP model fine-tuned specifically for email understanding tasks, particularly phishing and suspicious email detection.custom special tokens to explicitly encode email structure such as subject, body, links, and phone numbers, making it more robust for email-based security applications.
It can be used both as:
a sequence classification model for email safety detection, and
an embedding generator for downstream ML pipelines (e.g., XGBoost).
Model Details
Model Description
This model is fine-tuned from distilbert-base-uncased on curated email datasets. During preprocessing, email-specific entities such as URLs and phone numbers are replaced with dedicated tokens, and the subject and body are explicitly separated using structural markers.
Special Tokens Used
[SSUB], [ESUB] – Start/End of Subject[SBODY], [EBODY] – Start/End of Body[LINK] – URLs[PHONE] – Phone numbers
These design choices help the model better learn semantic and structural patterns commonly found in phishing emails.
Developed by: Atharva GaykarModel type: Transformer-based text classification & embedding modelLanguage: EnglishLicense: Artistic-2.0Finetuned from: distilbert/distilbert-base-uncased
Intended Uses
Primary Use Cases
Phishing email classification
Suspicious vs safe email detection
Feature extraction for traditional ML models
Email embedding generation for downstream classifiers
Out-of-Scope Uses
Non-text email analysis (images, attachments)
Commercial deployment without proper evaluation and compliance
Tasks unrelated to email or message-level text analysis
Bias, Risks, and Limitations
The model is trained on public phishing datasets and may reflect biases present in those sources.
Performance may degrade on highly obfuscated or novel phishing techniques.
Not recommended for direct commercial use without extensive validation.
Users should carefully evaluate the model in their target environment before deployment.
How to Get Started
1 from transformers import DistilBertTokenizerFast , DistilBertForSequenceClassification
2 import torch
3 import numpy as np
4
5 bert_path = "Gaykar/PhishingDistilBERT"
6
7 tokenizer = DistilBertTokenizerFast . from_pretrained ( bert_path )
8 model = DistilBertForSequenceClassification . from_pretrained ( bert_path )
9
10 device = torch . device ( "cuda" if torch . cuda . is_available ( ) else "cpu" )
11 model . to ( device )
12 model . eval ( )
13
14 def get_cls_embedding ( text , model , tokenizer , device ) :
15 with torch . no_grad ( ) :
16 inputs = tokenizer (
17 text ,
18 return_tensors = "pt" ,
19 truncation = True ,
20 padding = True ,
21 max_length = 256
22 )
23 inputs = { k : v . to ( device ) for k , v in inputs . items ( ) }
24 outputs = model . distilbert ( ** inputs )
25 cls_embedding = outputs . last_hidden_state [ : , 0 , : ] . squeeze ( ) . cpu ( ) . numpy ( )
26 return cls_embedding
27
28 text = "[SSUB] Urgent Account Alert [ESUB] [SBODY] Click [LINK] to verify your account. [EBODY]"
29 embedding = get_cls_embedding ( text , model , tokenizer , device )
30
31 print ( "Embedding shape:" , embedding . shape )
32 print ( "First 10 dimensions:" , embedding [ : 10 ] )
Training Details
Training Data
The model was trained using well-known phishing and email security datasets, including CEAS , combined with additional curated CSV sources.
Data Preprocessing
Cleaned and merged multiple CSV datasets
Replaced:
URLs → [LINK]
Phone numbers → [PHONE]
Combined subject and body using structural tokens:
[SSUB], [ESUB], [SBODY], [EBODY]
Training Hyperparameters
1 training_args = TrainingArguments (
2 output_dir = "./distilbert_safe_suspicious" ,
3 eval_strategy = "steps" ,
4 eval_steps = 50 ,
5 save_strategy = "steps" ,
6 save_steps = 50 ,
7 save_total_limit = 3 ,
8 load_best_model_at_end = True ,
9 metric_for_best_model = "eval_loss" ,
10 greater_is_better = False ,
11 learning_rate = 4e-5 ,
12 per_device_train_batch_size = 16 ,
13 per_device_eval_batch_size = 8 ,
14 num_train_epochs = 4 ,
15 weight_decay = 0.01 ,
16 logging_strategy = "steps" ,
17 logging_steps = 50 ,
18 seed = 42 ,
19 )
Evaluation
Evaluation Metrics
Testing Setup
10% held-out test split from the full dataset
Results
DistilBERT (standalone): Strong classification performanceDistilBERT embeddings + XGBoost + URL features:
99.4% accuracy
Evaluation Result Technical Specifications
Model Architecture
DistilBERT encoder
Sequence classification head
CLS-token embedding extraction supported
Compute Infrastructure
Hardware: NVIDIA T4 GPUFrameworks: PyTorch, Hugging Face Transformers
Environmental Impact
Carbon emissions were not explicitly measured.
Users may estimate emissions using the Machine Learning Impact Calculator if needed.
Model Card Authors
Contact
For questions, feedback, or research collaboration, please reach out via the Hugging Face model repository.
