Roman Urdu Sentiment Analysis Model (Khubaib01)

Try the Model

Test the model live on Hugging Face Spaces:

Benchmark Highlights

Benchmark notebook: Kaggle

Model Details

Model Description

• Fine-tuned XLM-RoBERTa Base for Roman Urdu sentiment classification
• Trained on a balanced, multi-domain dataset (~99k sentences)
• Labels: Positive, Negative, Neutral
• Sources: WhatsApp chats, YouTube comments, Twitter, and other social media
• Preprocessing: Removed media-only, deleted, very short, and emoji-only messages
• Developed by: Muhammad Khubaib Ahmad
• License: MIT

Key Advantages

• Top-ranking model across slang-heavy and formal datasets
• Detects offensive/toxic content naturally due to slang-heavy negative training data
• Balanced performance across all three sentiment classes
• Robust to multi-domain inputs, including social media and messaging platforms

Uses

Direct Use

• Classifying sentiment in Roman Urdu text
• Automated content moderation
• Identifying offensive, abusive, or highly negative content
• Analytics on Roman Urdu datasets, e.g., public sentiment tracking

Downstream Use

• Feature extraction in multi-task NLP pipelines
• Adaptable for other Roman Urdu classification tasks with additional fine-tuning

Out-of-Scope

• Native Urdu script without transliteration
• Subtle sarcasm, irony, or context-specific toxicity
• Newly coined or highly domain-specific slang not seen in training

Bias, Risks, and Limitations

• May inherit biases from the training data (slang, offensive terms, informal language)
• Accuracy may drop on rare, new, or unseen Roman Urdu slang
• Heavy code-mixing or highly non-standard text may be misclassified
• Should not be relied on as a dedicated toxicity detection tool; flags negative/offensive content as part of sentiment

Recommendations

• Validate predictions for critical applications
• Consider additional fine-tuning for domain-specific scenarios

How to Get Started

Quick Start

Advanced Usage

Training Details

• Base Model: xlm-roberta-base
• Training regime: fp16 mixed precision, batch size 16, gradient accumulation 4
• Epochs: 2
• Learning Rate: 1e-5
• Optimizer: AdamW
• Evaluation: Epoch-level evaluation using stratified train/validation split

Training Data

• Sources: WhatsApp chat exports, YouTube comments, Twitter, and other social media.
• Dataset size: ~99k sentences (balanced across Positive, Negative, Neutral).
• Preprocessing: Removed media-only, deleted, emoji-only, and very short messages.

Evaluation

• Testing: 10% held-out dataset
• Metrics: Accuracy, Macro F1, Confusion Matrix
• Results: Overall Accuracy ~84–85%, Macro F1 ~83–84%
• Benchmarking confirms Khubaib01 is the top-performing model, especially on slang-heavy text

Metrics

• Accuracy and Macro F1 per class
• Confusion matrix for analyzing misclassifications

Results

• Overall Accuracy: ~84–85%
• Macro F1: ~83–84%
• Balanced performance across Positive, Negative, and Neutral classes
• Benchmarking against other Roman Urdu sentiment models shows superior performance, especially on slang-heavy texts

Environmental Impact

• Hardware: NVIDIA T4 GPU (Colab)
• Training Time: ~4–5 hours
• Precision: fp16
• Estimated CO₂ Emissions: Low (approx. 1–2 kg CO₂eq, Colab)

Technical Specifications

• Model Architecture: XLM-RoBERTa Base (Transformer encoder, 12 layers, 768 hidden size)
• Objective: Sentiment classification (3-class softmax output)
• Software: Python, PyTorch, HuggingFace Transformers
• Tokenizer: SentencePiece-based multilingual tokenizer

Citation

• BibTeX:

• APA:

Glossary

• Roman Urdu: Urdu language written using the Latin alphabet.
• Code-mixed text: Sentences containing a mix of Roman Urdu and English words.
• fp16: Half-precision floating-point training for faster GPU usage.

Model Card Author