Roman Urdu-English Language Identification Model
Project Information
Project: Code Saviours SI-26 — Week 7
Intern: Muhammad Hassaan
Student ID: SI26-ML-MH-004
Department: AI/ML
Model Description
This model is a fine-tuned XLM-RoBERTa model for word-level language identification in Roman Urdu-English code-switched text.
The model predicts one of three labels for each word:
- URD — Roman Urdu
- ENG — English
- MIX — Mixed Roman Urdu-English token
Base Model
The model was fine-tuned from:
xlm-roberta-base
Dataset
The original dataset contained 150 Roman Urdu-English code-switched sentences.
During Week 7, additional MIX examples were added because the original dataset contained very few MIX-labelled tokens.
The augmented dataset contained 180 sentences.
The dataset was divided into:
- 144 training sentences
- 36 testing sentences
Training
XLM-RoBERTa was fine-tuned for token classification using GPU training.
Training configuration included:
- Epochs: 5
- Learning rate: 2e-5
- Training batch size: 16
- Evaluation strategy: Every epoch
- Labels: URD, ENG, MIX
Because the MIX class was underrepresented, sentences containing MIX-labelled tokens were oversampled during the final training experiment.
Evaluation
The model was evaluated using per-class F1 scores for:
The final balanced model performed better on the MIX class than the original training experiment.
Important: The MIX test set contained only a small number of MIX-labelled examples, so the MIX F1 score should be interpreted carefully.
Final F1 Scores
URD F1 : 0.9865
ENG F1 : 0.9853
MIX F1 : 1.0000
Macro F1 : 0.9906
Accuracy : 0.9863
Example
Input:
Aaj mera meeting schedule bohot busy hai
Expected language identification:
- Aaj → URD
- mera → URD
- meeting → ENG
- schedule → ENG
- bohot → URD
- busy → ENG
- hai → URD
Mixed-token example:
Mujhe fileupload karni hai
- Mujhe → URD
- fileupload → MIX
- karni → URD
- hai → URD
Limitations
- The dataset is relatively small.
- The MIX class has fewer examples than URD and ENG.
- Roman Urdu spelling is highly variable.
- Performance may decrease on unseen spellings or sentence styles.
- The model should be considered a research and internship project rather than a production-ready language identification system.
Future Improvements
Future work could include:
- Collecting a larger Roman Urdu-English dataset.
- Adding more naturally occurring MIX examples.
- Improving class balance.
- Testing on a larger unseen dataset.
- Performing additional hyperparameter tuning.
Author
Muhammad Hassaan
Student ID: SI26-ML-MH-004
Code Saviours SI-26
