You can deploy this model and start earning money today!

Umair1710/
xlmt-roberta-twitter-finetuned-sentimentalAnalysis

Name: xlmt-roberta-twitter-finetuned-sentimentalAnalysis
Availability: InStock

safetensors

xlm-roberta

Views

No views yet

XLM-T RoBERTa Fine-tuned for Roman Urdu Sentiment Analysis

Model Overview

This model is a fine-tuned version of cardiffnlp/twitter-xlm-roberta-base-sentiment specifically optimized for Roman Urdu sentiment analysis. Roman Urdu refers to Urdu language written using the Latin/Roman script, commonly used in social media, messaging, and online forums.

Fine-tuned by: Umair
Base Model: Twitter-XLM-RoBERTa-base (cardiffnlp)
Language: Roman Urdu, Urdu, English (mixed)
Task: 3-class Sentiment Classification (Negative, Neutral, Positive)
Training Date: June 2026

Performance Metrics

Metric	Base Model	Fine-tuned Model	Improvement
Accuracy	61.51%	86.89%	+25.38%
F1-Score	63.98%	86.82%	+22.84%
Precision	65.54%	86.80%	+21.26%
Recall	72.57%	86.89%	+14.32%

Per-Class Performance (Fine-tuned Model)

Class	Precision	Recall	F1-Score	Support
Negative	0.85	0.86	0.85	208
Neutral	0.83	0.80	0.81	161
Positive	0.90	0.91	0.90	478

Dataset

Total Samples: 4,233 annotated Roman Urdu texts
Class Distribution:
- Positive: 2,388 (56.4%)
- Negative: 1,038 (24.5%)
- Neutral: 806 (19.1%)
Split: 80% Training, 20% Test (stratified)
Training Samples: 3,046
Test Samples: 847

Training Details

Preprocessing Pipeline

The model uses a comprehensive Roman Urdu normalization pipeline with dictionary mapping for common variations.

Roman Urdu Normalization Examples:

'aj' → 'aaj' (today)
'acha' → 'achha' (good)
'bohat' → 'bahut' (very)
'nahi' → 'nahin' (no)
'thora' → 'thoda' (little)

Fine-tuning Configuration

Method: LoRA (Low-Rank Adaptation) - Parameter Efficient Fine-Tuning
LoRA Rank (r): 12
LoRA Alpha: 24
LoRA Dropout: 0.1
Target Modules: query, key, value, dense
Trainable Parameters: ~0.5M (only 0.2% of total)
Total Model Parameters: ~280M

Hyperparameters

Parameter	Value
Learning Rate	2e-5
Batch Size	16
Epochs	8 (early stopping at 3)
Optimizer	AdamW
Weight Decay	0.01
Warmup Steps	50
Max Sequence Length	128 tokens
Mixed Precision	FP16 (GPU)

Class Imbalance Handling

To address class imbalance (positive samples being 56% of data), the model used:

Balanced Class Weights: [1.33, 1.97, 0.59] for [Negative, Neutral, Positive]
Weighted Loss Function: Custom trainer with per-sample weights
Stratified Splitting: Maintained class distribution across train/val/test

How to Use

Installation

pip install transformers torch

Inference Code

python

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

# Load model and tokenizer
model = AutoModelForSequenceClassification.from_pretrained("Umair1710/xlmt-roberta-twitter-finetuned-sentimentalAnalysis")
tokenizer = AutoTokenizer.from_pretrained("Umair1710/xlmt-roberta-twitter-finetuned-sentimentalAnalysis")

# Preprocessing function
def preprocess_text(text):
    import re
    roman_urdu_dict = {
        'aj': 'aaj', 'acha': 'achha', 'bohat': 'bahut',
        'nahi': 'nahin', 'kya': 'kya', 'yaar': 'yar',
    }
    text = text.lower()
    words = text.split()
    normalized = [roman_urdu_dict.get(word, word) for word in words]
    return ' '.join(normalized)

# Prediction function
def predict_sentiment(text, model, tokenizer):
    text = preprocess_text(text)
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128, padding=True)
    
    with torch.no_grad():
        outputs = model(**inputs)
        probs = torch.nn.functional.softmax(outputs.logits, dim=-1)
        pred_class = torch.argmax(probs, dim=-1).item()
    
    labels = {0: "negative", 1: "neutral", 2: "positive"}
    confidence = probs[0][pred_class].item()
    
    return labels[pred_class], confidence

# Example usage
text = "aj bohat achha din tha jeet gaye"
sentiment, confidence = predict_sentiment(text, model, tokenizer)
print(f"Text: {text}")
print(f"Sentiment: {sentiment} (confidence: {confidence:.3f})")

Example Predictions

Text	Predicted	Confidence
jeet gaye badminton singles	Positive	0.962
My alarm didn't go off	Negative	0.953
aj thora relax feel ho raha hai	Neutral	0.978

Technical Approach

Why LoRA Fine-tuning?

LoRA (Low-Rank Adaptation) was chosen to:

Prevent overfitting on limited data (only 3k training samples)
Reduce memory usage (only 0.2% parameters trainable)
Faster training on Kaggle GPU
Better generalization compared to full fine-tuning

Why XLM-T RoBERTa?

Pre-trained on multilingual Twitter data (100+ languages)
Understands code-mixed text (Roman Urdu + English + Urdu script)
768-dimensional embeddings capture semantic nuances
State-of-the-art on sentiment analysis tasks

Citation

bibtex

@misc{umair2024xlmt,
  author = {Umair},
  title = {XLM-T RoBERTa Fine-tuned for Roman Urdu Sentiment Analysis},
  year = {2024},
  publisher = {Hugging Face},
  url = {https://huggingface.co/Umair1710/xlmt-roberta-twitter-finetuned-sentimentalAnalysis}
}

Limitations and Future Work

Current Limitations

Best suited for social media length text (under 128 tokens)
May struggle with very sarcastic or nuanced expressions
Performance may vary on pure Urdu script

Future Improvements

Expand training data to 10k+ samples
Add sarcasm detection capabilities
Support for longer sequences (>128 tokens)

License

This model is released under the Apache 2.0 license.

Contact

Hugging Face: Umair1710
Model Card: View on HF Hub

⭐ If you find this model useful, please give it a star on Hugging Face!