Views
No views yet
xlm-roberta-base specifically designed for detecting passwords and secrets in documents through token classification. Unlike traditional regex-based approaches, this model understands context to identify both structured tokens (API keys, JWTs) and free-form passwords.1LoraConfig(
2 task_type=TaskType.TOKEN_CLS,
3 r=64, # Rank
4 lora_alpha=128, # Scaling parameter
5 lora_dropout=0.05, # Dropout probability
6 bias="none",
7 target_modules=["query", "key", "value", "dense"]
8)0: Non-credential token1: Credential/password token"Your account has been created with username: {user} and password: {pass}"1# Preprocessing
2- Tokenization with offset mapping
3- Label generation based on credential spans
4- Padding to max_length with truncation
5
6# Fine-tuning
7- LoRA adapters applied to attention layers
8- Binary cross-entropy loss
9- Token-level classification head| Metric | Score |
|---|---|
| Strict Accuracy | 86.67% |
| Overlap Accuracy | 97.72% |
| Metric | Count/Rate |
|---|---|
| True Positives | 1,201 |
| True Negatives | 1,112 |
| False Positives | 49 (3.9%) |
| False Negatives | 138 |
| Overlap True Positives | 456 |
| Recall | 89.7% |
pip install transformers torch1from transformers import AutoModelForTokenClassification, AutoTokenizer
2import torch
3
4# Load model and tokenizer
5model_name = "path/to/deeppass2-xlm-roberta"
6tokenizer = AutoTokenizer.from_pretrained(model_name)
7model = AutoModelForTokenClassification.from_pretrained(model_name)
8
9# Classify tokens
10def detect_passwords(text):
11 inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)
12
13 with torch.no_grad():
14 outputs = model(**inputs)
15
16 predictions = torch.argmax(outputs.logits, dim=-1)
17 tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
18
19 # Extract password tokens
20 password_tokens = [
21 token for token, label in zip(tokens, predictions[0])
22 if label == 1
23 ]
24
25 return password_tokens1@software{gupta2025deeppass2,
2 author = {Gupta, Neeraj},
3 title = {DeepPass2: Fine-tuned XLM-RoBERTa for Secret Detection},
4 year = {2025},
5 organization = {SpecterOps},
6 url = {https://huggingface.co/deeppass2-bert},
7 note = {Blog: \url{https://specterops.io/blog/2025/07/31/whats-your-secret-secret-scanning-by-deeppass2/}}
8}