Views
No views yet
MutazYoune/ARAB_BERT | Task: Token Classification | Language: Arabicpip install transformers torch1from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline
2
3# Load model
4tokenizer = AutoTokenizer.from_pretrained("MutazYoune/Arabic-NER-PII")
5model = AutoModelForTokenClassification.from_pretrained("MutazYoune/Arabic-NER-PII")
6
7# Create pipeline
8ner_pipeline = pipeline("ner", model=model, tokenizer=tokenizer, aggregation_strategy="simple")
9
10# Detect PII
11text = "يعمل أحمد محمد في شركة جوجل في الرياض ورقم هاتفه 0501234567"
12entities = ner_pipeline(text)
13print(entities)| Entity | Description | Examples |
|---|---|---|
CONTACT | Email addresses, phone numbers | ahmed@email.com, 0501234567 |
NETWORK | IP addresses, network identifiers | 192.168.1.1, 10-20-30-40 |
IDENTIFIER | National IDs, structured identifiers | ID_123456, user.name |
NUMERIC_ID | Numeric identifiers | 123456789, 12-34-56 |
PII | Generic personal information | Names, personal details |
Maqsam Arabic PII Redaction Challenge - Rank #16
| Metric | Exact | Partial | IoU50 |
|---|---|---|---|
| Precision | 0.029 | 0.647 | 0.295 |
| Recall | 0.020 | 0.455 | 0.208 |
| F1 | 0.024 | 0.534 | 0.244 |
1base_model: MutazYoune/ARAB_BERT
2epochs: 12
3batch_size: 16
4learning_rate: 3e-5
5max_length: 512
6optimization: AdamW1PATTERNS = {
2 "CONTACT": r'[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}|(?:https?|ftp)://[^\s/$.?#].[^\s]*',
3 "NETWORK": r'\d+\.\d+\.\d+\.\d+|\d+\-\d+\-\d+\-\d+',
4 "IDENTIFIER": r'[a-zA-Z]+_[a-zA-Z]+\d*|[a-zA-Z]+\.[a-zA-Z]+',
5 "NUMERIC_ID": r'\d+\-\d+|\d{6,}'
6}1import torch
2from transformers import AutoTokenizer, AutoModelForTokenClassification
3
4def process_arabic_text(text, model, tokenizer):
5 inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
6
7 with torch.no_grad():
8 outputs = model(**inputs)
9 predictions = torch.argmax(outputs.logits, dim=-1)
10
11 tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
12 labels = [model.config.id2label[pred.item()] for pred in predictions[0]]
13
14 # Filter out special tokens
15 results = []
16 for token, label in zip(tokens, labels):
17 if token not in ['[CLS]', '[SEP]', '[PAD]']:
18 results.append((token, label))
19
20 return results1def batch_process_texts(texts, model, tokenizer, batch_size=8):
2 results = []
3 for i in range(0, len(texts), batch_size):
4 batch = texts[i:i+batch_size]
5 batch_results = []
6
7 for text in batch:
8 entities = ner_pipeline(text)
9 batch_results.append(entities)
10
11 results.extend(batch_results)
12
13 return resultsInput: Arabic Text
↓
Tokenization (Arabic BERT Tokenizer)
↓
ARAB_BERT Encoder (12 layers)
↓
Classification Head (11 classes)
↓
BIO Tag PredictionsFinal Score = 0.45 × Precision + 0.45 × Recall + 0.1 × (1/avg_time)1@misc{arabic-ner-pii-2024,
2 author = {MutazYoune},
3 title = {Arabic NER PII: Personally Identifiable Information Detection for Arabic Text},
4 year = {2024},
5 publisher = {Hugging Face},
6 journal = {Hugging Face Model Hub},
7 howpublished = {\url{https://huggingface.co/MutazYoune/Arabic-NER-PII}}
8}