Fine-tuned XLM-RoBERTa Base for 3-class sentiment analysis on Nepali text across all three scripts: Devanagari (देवनागरी), Romanized Nepali, and code-mixed Nepali-English.
This is currently one of the largest open fine-tunes for Nepali sentiment analysis, trained on a merged corpus of ~125k samples from 7 public datasets. The model handles real-world Nepali social media text including emoji, slang, and script-switching within a single sentence.
Labels
ID
Label
Description
0
negative
Negative sentiment
1
neutral
Neutral / factual
2
positive
Positive sentiment
Quick Start
python
1from transformers import pipeline
23pipe = pipeline(4"text-classification",5 model="YOUR_HF_USERNAME/xlm-roberta-base-nepali-sentiment-v1",6 tokenizer="YOUR_HF_USERNAME/xlm-roberta-base-nepali-sentiment-v1",7)89# Devanagari10pipe("यो चलचित्र साँच्चै राम्रो छ!")11# [{'label': 'positive', 'score': 0.92}]1213# Romanized Nepali14pipe("yo movie ekdam ramro thiyo bro")15# [{'label': 'positive', 'score': 0.88}]1617# Code-mixed18pipe("Movie chai ramro thiyo but ending weak lagyo 😕")19# [{'label': 'negative', 'score': 0.71}]2021# Devanagari negative22pipe("सेवा एकदमै खराब थियो, फेरि कहिल्यै आउँदिन।")23# [{'label': 'negative', 'score': 0.89}]
With explicit probabilities
python
1import torch
2from transformers import AutoTokenizer, AutoModelForSequenceClassification
3import torch.nn.functional as F
45model_id ="YOUR_HF_USERNAME/xlm-roberta-base-nepali-sentiment-v1"6tokenizer = AutoTokenizer.from_pretrained(model_id)7model = AutoModelForSequenceClassification.from_pretrained(model_id)8model.eval()910id2label ={0:"negative",1:"neutral",2:"positive"}1112defpredict(text:str):13 inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)14with torch.no_grad():15 logits = model(**inputs).logits
16 probs = F.softmax(logits, dim=-1).squeeze()17for i, p inenumerate(probs):18print(f" {id2label[i]}: {p:.3f}")1920predict("सरकारले राम्रो काम गरेको छ।")
Training Data
The model was trained on a merged dataset built from 7 publicly available Nepali sentiment corpora. All datasets were fetched programmatically without manual downloads.
Total after merge and deduplication: ~125,624 samples
Label Unification
All datasets were mapped to a single 3-class schema: {0: negative, 1: neutral, 2: positive}.
Binary datasets (sources 3 and 7) contributed only to the negative (0) and positive (2) classes — no synthetic neutral labels were generated.
Star-rating datasets (sources 5 and 6) were mapped as:
Stars
→ Label
1–2
negative (0)
3
neutral (1)
4–5
positive (2)
Script Type Tagging
Each sample was automatically tagged with a script type using Unicode range detection:
devanagari — >30% of characters in U+0900–U+097F
romanized — ASCII-dominant with Nepali lexical markers (e.g., cha, thyo, haru, ramro)
english — ASCII-dominant, no Nepali markers
mixed — neither purely Devanagari nor ASCII-dominant
Final Split
Split
Samples
Train
100,499
Validation
12,562
Test
12,563
— test_devanagari
9,294
— test_romanized
1,251
— test_mixed
2,018
Splits were stratified by label class. All per-script test sets are held-out subsets of the same test split.
Class Distribution (Training Set)
Label
Count
%
negative (0)
37,762
37.6%
neutral (1)
9,878
9.8%
positive (2)
52,859
52.6%
Neutral is underrepresented (~5.3× fewer samples than positive). Inverse-frequency class weights were applied during training to partially compensate.
Preprocessing
Only minimal cleaning was applied:
Unicode NFC normalization (critical for Devanagari combining characters)
Lowercase
URL removal
Duplicate whitespace collapse
Emojis retained — they carry strong sentiment signal in Nepali social media text
Aggressive cleaning (punctuation stripping, stopword removal) was intentionally avoided. Romanized normalization (xa→cha, xaina→chaina) was not applied to preserve the original signal distribution.
Training Procedure
Base Model
xlm-roberta-base — chosen for native Nepali, English, and code-mixed support with a strong multilingual pretraining foundation.
Full fine-tune. No LoRA or adapters — dataset size (~100k) and GPU budget (2× NVIDIA T4) made full fine-tuning appropriate.
Hyperparameters
Parameter
Value
max_length
128
per_device_train_batch_size
32
gradient_accumulation_steps
2
Effective batch size
128 (32 × 2 devices × 2 accum)
learning_rate
2e-5
weight_decay
0.01
warmup_ratio
0.1
num_train_epochs
4
fp16
True
lr_scheduler
linear with warmup
Class weights
Inverse-frequency, normalized
Early stopping patience
2 epochs
Best model selection
macro F1 on validation set
Hardware
2× NVIDIA Tesla T4 (16 GB VRAM each), Kaggle notebook environment. Total training time: ~1 hr 40 min.
Training Curve
Epoch
Train Loss
Val Loss
Accuracy
Macro F1
1
1.2324
0.6287
0.7441
0.6823
2
1.0643
0.5346
0.7785
0.7183
3
0.9381
0.5406
0.8027
0.7406
4
0.8494
0.5503
0.8149
0.7525 ← saved
Validation loss bottomed at epoch 2; macro F1 continued improving through epoch 4. The model was still converging at epoch 4 (train loss descending), indicating further gains are achievable with continued training at a reduced learning rate.
Evaluation Results
All results are on the held-out test set (12,563 samples, never seen during training).
Overall Test Set
Metric
Score
Accuracy
0.8159
Macro F1
0.7533
Weighted F1
0.8251
Per-Class (Full Test Set)
Class
Precision
Recall
F1
Support
negative
0.85
0.81
0.83
4,721
neutral
0.45
0.73
0.56
1,234
positive
0.91
0.84
0.87
6,608
Per-Script Breakdown
Script
Samples
Accuracy
Macro F1
F1 neg
F1 neu
F1 pos
Devanagari
9,294
0.8072
0.7369
0.8448
0.5296
0.8363
Romanized
1,251
0.7346
0.7019
0.6241
0.6097
0.8719
Mixed/English
2,018
0.9063
0.7777
0.7684
0.5995
0.9651
Known Limitations
Neutral class precision is low (0.45). The model over-predicts neutral — more than half of samples it labels neutral are actually negative or positive. Root causes: neutral is the rarest class (~10% of training data), and neutral annotation criteria differ significantly across the 7 source datasets. For pipelines where neutral precision matters, apply a confidence threshold (≥0.55–0.60 on the neutral logit) before accepting a neutral label.
Romanized negative detection is weak (F1=0.624). The romanized training set is small (~3k samples) and likely skewed positive. Romanized negative expressions are underrepresented. Use with caution for negative sentiment triage in romanized text.
Binary use case is strongest. For positive/negative classification (ignoring neutral), the model performs well across all scripts. Treating low-confidence neutral predictions as "uncertain" rather than a hard label is recommended for downstream applications.
Dialect and register gaps. Training data skews toward social media, news, and review text. Performance on formal Nepali (legal, administrative, academic) or strongly dialectal text (Terai, Hill dialects) is untested.
Dataset label inconsistency. Labels were merged from 7 independent annotation efforts with no inter-annotator agreement study across sources. Noise from inconsistent labeling is present, especially at the neutral/negative boundary.
Intended Use
Suitable for:
Bulk positive/negative triage of Nepali social media, reviews, and news comments
Script-agnostic Nepali sentiment scoring (Devanagari, Romanized, code-mixed)
Downstream tasks requiring a multilingual Nepali sentiment feature
Not suitable for:
High-stakes per-sample decisions without human review
Neutral class as a reliable hard label without threshold tuning
Languages other than Nepali and English
Citation
If you use this model, please cite the base model and the primary training dataset:
bibtex
1@article{conneau2019unsupervised,
2 title = {Unsupervised Cross-lingual Representation Learning at Scale},
3 author = {Conneau, Alexis and others},
4 journal = {arXiv preprint arXiv:1911.02116},
5 year = {2019}
6}
Acknowledgements
Training data sourced from the open Nepali NLP community. Primary dataset backbone: Sagar32/NepaliDevanagariSentimentAnalysis. Full dataset list and dataset-building pipeline available in the linked training repository.
