This model is pre-trained on
nepalitext dataset consisting of over 13 million Nepali text sequences using a masked language modeling (MLM) objective. Our approach trains a Sentence Piece Model (SPM) for text tokenization similar to
XLM-ROBERTa and trains
distilbert model for language modeling. Find more details in
this paper.
This backbone model intends to be fine-tuned on Nepali language focused downstream task such as sequence classification, token classification or question answering.
The language model being trained on a data with texts grouped to a block size of 512, it handles text sequence up to 512 tokens and may not perform satisfactorily on shorter sequences.
1>>> from transformers import pipeline
2>>> unmasker = pipeline('fill-mask', model='Sakonii/distilbert-base-nepali')
3>>> unmasker("मानविय गतिविधिले प्रातृतिक पर्यावरन प्रनालीलाई अपरिमेय क्षति पु्र्याएको छ। परिवर्तनशिल जलवायुले खाध, सुरक्षा, <mask>, जमिन, मौसमलगायतलाई असंख्य तरिकाले प्रभावित छ।")
4
5[{'score': 0.04128897562623024,
6 'sequence': 'मानविय गतिविधिले प्रातृतिक पर्यावरन प्रनालीलाई अपरिमेय क्षति पु्र्याएको छ। परिवर्तनशिल जलवायुले खाध, सुरक्षा, मौसम, जमिन, मौसमलगायतलाई असंख्य तरिकाले प्रभावित छ।',
7 'token': 2605,
8 'token_str': 'मौसम'},
9 {'score': 0.04100276157259941,
10 'sequence': 'मानविय गतिविधिले प्रातृतिक पर्यावरन प्रनालीलाई अपरिमेय क्षति पु्र्याएको छ। परिवर्तनशिल जलवायुले खाध, सुरक्षा, प्रकृति, जमिन, मौसमलगायतलाई असंख्य तरिकाले प्रभावित छ।',
11 'token': 2792,
12 'token_str': 'प्रकृति'},
13 {'score': 0.026525357738137245,
14 'sequence': 'मानविय गतिविधिले प्रातृतिक पर्यावरन प्रनालीलाई अपरिमेय क्षति पु्र्याएको छ। परिवर्तनशिल जलवायुले खाध, सुरक्षा, पानी, जमिन, मौसमलगायतलाई असंख्य तरिकाले प्रभावित छ।',
15 'token': 387,
16 'token_str': 'पानी'},
17 {'score': 0.02340106852352619,
18 'sequence': 'मानविय गतिविधिले प्रातृतिक पर्यावरन प्रनालीलाई अपरिमेय क्षति पु्र्याएको छ। परिवर्तनशिल जलवायुले खाध, सुरक्षा, जल, जमिन, मौसमलगायतलाई असंख्य तरिकाले प्रभावित छ।',
19 'token': 1313,
20 'token_str': 'जल'},
21 {'score': 0.02055591531097889,
22 'sequence': 'मानविय गतिविधिले प्रातृतिक पर्यावरन प्रनालीलाई अपरिमेय क्षति पु्र्याएको छ। परिवर्तनशिल जलवायुले खाध, सुरक्षा, वातावरण, जमिन, मौसमलगायतलाई असंख्य तरिकाले प्रभावित छ।',
23 'token': 790,
24 'token_str': 'वातावरण'}]
1from transformers import AutoTokenizer, AutoModelForMaskedLM
2
3tokenizer = AutoTokenizer.from_pretrained('Sakonii/distilbert-base-nepali')
4model = AutoModelForMaskedLM.from_pretrained('Sakonii/distilbert-base-nepali')
5
6# prepare input
7text = "चाहिएको text यता राख्नु होला।"
8encoded_input = tokenizer(text, return_tensors='pt')
9
10# forward pass
11output = model(**encoded_input)
This model is trained on
nepalitext language modeling dataset which combines the datasets:
OSCAR ,
cc100 and a set of scraped Nepali articles on Wikipedia.
As for training the language model, the texts in the training set are grouped to a block of 512 tokens.
A Sentence Piece Model (SPM) is trained on a subset of
nepalitext dataset for text tokenization. The tokenizer trained with vocab-size=24576, min-frequency=4, limit-alphabet=1000 and model-max-length=512.
The model is trained with the same configuration as the original
distilbert-base-uncased; 512 tokens per instance, 28 instances per batch, and around 35.7K training steps.
The following hyperparameters were used for training of the final epoch: [ Refer to the Training results table below for varying hyperparameters every epoch ]
