IRIISNEPAL/RoBERTa_Nepali_110M is a RoBERTa-based transformer model developed specifically for the Nepali language. This 110-million-parameter model is intended for tasks in natural language understanding (NLU), such as sentiment analysis, text classification, and named entity recognition in Nepali.
Model Details
Model Description
Developed by: Institute of Research and Innovation in Intelligent Systems (IRIIS)
Model type: RoBERTa-based transformer model specifically trained on Nepali language data
Model Size: 110 million parameters
Language (NLP): Nepali
Training Objective: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP)
The IRIISNEPAL/RoBERTa_Nepali_110M model aims to provide a robust tool for NLP tasks specific to the Nepali language, supporting NLP research and applications within low-resource languages.
Uses
Direct Use
The model provides contextual embeddings for each token in an input sequence (last_hidden_state) and a pooled representation of the entire input (pooler_output). These outputs can be used for:
Text Classification: Using pooler_output to classify the overall sentiment, intent, or category of a sentence.
Token-Level Tasks: Leveraging last_hidden_state to perform tasks like named entity recognition (NER) or part-of-speech tagging by predicting labels for individual tokens.
Sentence Embeddings: Using pooler_output as an embedding for the entire input text for similarity search or clustering tasks.
These results indicate the model’s effectiveness in capturing language nuances for multiple NLU tasks in Nepali.
Bias, Risks, and Limitations
The model may exhibit biases present in its training data, especially regarding social, cultural, and regional aspects of the Nepali language. Users should exercise caution when deploying it in applications that might perpetuate stereotypes or cultural biases.
Recommendations
It’s advisable for users to monitor model outputs for fairness and avoid high-stakes applications without thorough testing. Fine-tuning or retraining may be necessary for sensitive applications.
How to Get Started with the Model
Use the code below to get started with the model.
python
1# Load model directly2from transformers import AutoTokenizer, AutoModel
34tokenizer = AutoTokenizer.from_pretrained("IRIISNEPAL/RoBERTa_Nepali_110M")5model = AutoModel.from_pretrained("IRIISNEPAL/RoBERTa_Nepali_110M")67text ="नेपालमा पर्यटनको विकास गर्नुपर्ने आवश्यकता छ।"8inputs = tokenizer(text, return_tensors="pt")9outputs = model(**inputs)
Training Details
Training Data
The model was trained on a 27.5 GB Nepali language corpus compiled from 99 Nepali news websites. This dataset represents the largest Nepali language corpus to date, providing a significant expansion in training resources for the language. The preprocessing involved deduplication, translation/removal of non-Nepali content, and noise reduction.
Training Regime: Mixed precision (fp16) on TPU v4-8 hardware
Batch Size: 256
Learning Rate: 1e-4 with a warmup over the first 10,000 steps followed by linear decay
Preprocessing [optional]
[More Information Needed]
Training Hyperparameters
Max Sequence Length: 512 tokens
Learning Rate Scheduler: Linear with warmup
Optimizer: AdamW with β1 = 0.9, β2 = 0.999, and L2 weight decay of 0.01
Dropout Probability: 0.1 across all layers
Activation Function: GELU
Speeds, Sizes, Times [optional]
[More Information Needed]
Evaluation
Testing Data, Factors & Metrics
Testing Data
The model was evaluated on the Nepali Language Evaluation Benchmark (Nep-gLUE), which includes tasks like Named Entity Recognition (NER), Part-of-Speech (POS) Tagging, text classification, and categorical pair similarity.
On Nep-gLUE, the model outperformed existing state-of-the-art models with an overall score of 95.60, reflecting its strong language understanding capabilities.
Model Examination
Performance analysis indicates robustness in capturing grammatical and syntactical features of Nepali. However, the model may have limited effectiveness in handling dialect-specific content or informal language.
RoBERTa architecture with 12 transformer layers, hidden size of 768, 12 attention heads, and 110 million parameters. This architecture facilitates strong bidirectional attention for accurate language understanding.
Compute Infrastructure
Hardware: TPU v4-8 and Nvidia GeForce RTX 3090 GPUs
Software: Python, PyTorch, Hugging Face Transformers
Citation
@misc{thapa2024developmentpretrainedtransformerbasedmodels,
title={Development of Pre-Trained Transformer-based Models for the Nepali Language},
author={Prajwal Thapa and Jinu Nyachhyon and Mridul Sharma and Bal Krishna Bal},
year={2024},
eprint={2411.15734},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2411.15734},
}
