Benchmarking BERT-based Models for Sentence-level Topic Classification in Nepali Language

Overview

Objectives

• Benchmark multiple BERT-based models on Nepali text classification
• Analyze performance differences across multilingual, Indic, and monolingual models
• Establish a strong baseline for future Nepali NLP tasks
• Provide insights into low-resource language modeling

Dataset

• 🌾 Agriculture
• 🏥 Health
• 🎓 Education & Technology
• 🏔️ Culture & Tourism
• 💬 General Communication

Models Evaluated

Multilingual Models

• mBERT
• XLM-RoBERTa
• mDeBERTa

Indic Models

• MuRIL (base & large)
• IndicBERT
• DevBERT

Language-Specific Models

• HindiBERT
• NepBERTa

English Model

• RoBERTa

Experimental Setup

• Training samples: 20,005
• Validation samples: 2,500
• Test samples: 2,501
• Epochs: 10
• Learning rate: 2e-5
• Batch size: 8
• Gradient accumulation: 2
• Max sequence length: 256

Evaluation Metrics

• Accuracy
• Precision (Weighted)
• Recall (Weighted)
• F1-score (Weighted)
• AUROC

Key Results

• MuRIL-large achieved the best performance:
  • F1-score: 90.60%
• NepBERTa showed strong competitive performance:
  • F1-score: 88.26%
• Indic models outperformed multilingual and general models overall

Key Insights

• Region-specific (Indic) models perform better for Nepali
• Monolingual pretraining (NepBERTa) is highly effective
• General multilingual models are slightly less optimized for Nepali

Limitations

• Limited to sentence-level classification
• Dataset covers only five domains
• No extensive error analysis performed
• Results may not generalize to other NLP tasks

Future Work

• Extend to document-level classification
• Expand dataset with more domains
• Perform detailed error analysis
• Explore ensemble methods
• Improve Nepali-specific pretraining

Citation