BERT-based Organization Detection Model for Turkish Texts

Model Description

Model Architecture

• Base Model: BERT (dbmdz/bert-base-turkish-uncased)
• Training Data: Twitter data from 3,922 accounts with high organization-related activity as determined by m3inference scores above 0.7. The data was annotated based on user names, screen names, and descriptions by a human annotator.

Training Setup

• Tokenization: Used Hugging Face's AutoTokenizer, padding sequences to a maximum length of 128 tokens.
• Dataset Split: 80% training, 20% validation.
• Training Parameters:
  • Epochs: 3
  • Training batch size: 8
  • Evaluation batch size: 16
  • Warmup steps: 500
  • Weight decay: 0.01

Hyperparameter Tuning

• Learning rate: 1.2323083424093641e-05
• Batch size: 32
• Epochs: 2

Evaluation Metrics

• Precision on Validation Set: 0.94 (organization class)
• Recall on Validation Set: 0.95 (organization class)
• F1-Score (Macro Average): 0.95
• Accuracy: 0.95
• Confusion Matrix on Validation Set:

[[369, 22],
[19, 375]]

• Hand-coded Sample of 1000 Accounts:
• Precision: 0.91
• F1-Score (Macro Average): 0.947
• Confusion Matrix:

  [[936, 3],
   [ 4, 31]]

How to Use