RoBERTa base model for Marathi language (मराठी भाषा)
Pretrained model on Marathi language using a masked language modeling (MLM) objective. RoBERTa was introduced in
this paper and first released in
this repository. We trained RoBERTa model for Marathi Language during community week hosted by Huggingface 🤗 using JAX/Flax for NLP & CV jax.
huggingface-marathi-roberta
Model description
Marathi RoBERTa is a transformers model pretrained on a large corpus of Marathi data in a self-supervised fashion.
Intended uses & limitations❗️
You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task.
Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
to make decisions, such as sequence classification, token classification or question answering. We used this model to fine tune on text classification task for iNLTK and indicNLP news text classification problem statement. Since marathi mc4 dataset is made by scraping marathi newspapers text, it will involve some biases which will also affect all fine-tuned versions of this model.
How to use❓
You can use this model directly with a pipeline for masked language modeling:
python
1>>>from transformers import pipeline
2>>> unmasker = pipeline('fill-mask', model='flax-community/roberta-base-mr')3>>> unmasker("मोठी बातमी! उद्या दुपारी <mask> वाजता जाहीर होणार दहावीचा निकाल")4[{'score':0.057209037244319916,'sequence':'मोठी बातमी! उद्या दुपारी आठ वाजता जाहीर होणार दहावीचा निकाल',5'token':2226,6'token_str':'आठ'},7{'score':0.02796074189245701,8'sequence':'मोठी बातमी! उद्या दुपारी २० वाजता जाहीर होणार दहावीचा निकाल',9'token':987,10'token_str':'२०'},11{'score':0.017235398292541504,12'sequence':'मोठी बातमी! उद्या दुपारी नऊ वाजता जाहीर होणार दहावीचा निकाल',13'token':4080,14'token_str':'नऊ'},15{'score':0.01691395975649357,16'sequence':'मोठी बातमी! उद्या दुपारी २१ वाजता जाहीर होणार दहावीचा निकाल',17'token':1944,18'token_str':'२१'},19{'score':0.016252165660262108,20'sequence':'मोठी बातमी! उद्या दुपारी ३ वाजता जाहीर होणार दहावीचा निकाल',21'token':549,22'token_str':' ३'}]
Training data 🏋🏻♂️
The RoBERTa Marathi model was pretrained on mr dataset of C4 multilingual dataset:
The dataset can be downloaded in a pre-processed form from allennlp or huggingface's datsets - mc4 dataset.
Marathi (mr) dataset consists of 14 billion tokens, 7.8 million docs and with weight ~70 GB of text.
Data Cleaning 🧹
Though initial mc4 marathi corpus size ~70 GB, Through data exploration, it was observed it contains docs from different languages especially thai, chinese etc. So we had to clean the dataset before traning tokenizer and model. Surprisingly, results after cleaning Marathi mc4 corpus data:
Train set:
Clean docs count 1581396 out of 7774331. ~20.34% of whole marathi train split is actually Marathi.
Validation set
Clean docs count 1700 out of 7928. ~19.90% of whole marathi validation split is actually Marathi.
Training procedure 👨🏻💻
Preprocessing
The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 50265. The inputs of
the model take pieces of 512 contiguous token that may span over documents. The beginning of a new document is marked
with <s> and the end of one by </s>
The details of the masking procedure for each sentence are the following:
15% of the tokens are masked.
In 80% of the cases, the masked tokens are replaced by <mask>.
In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
In the 10% remaining cases, the masked tokens are left as is.
Contrary to BERT, the masking is done dynamically during pretraining (e.g., it changes at each epoch and is not fixed).
Pretraining
The model was trained on Google Cloud Engine TPUv3-8 machine (with 335 GB of RAM, 1000 GB of hard drive, 96 CPU cores) 8 v3 TPU cores for 42K steps with a batch size of 128 and a sequence length of 128. The
optimizer used is Adam with a learning rate of 3e-4, β1 = 0.9, β2 = 0.98 and
ε = 1e-8, a weight decay of 0.01, learning rate warmup for 1,000 steps and linear decay of the learning
rate after.
Huge thanks to Huggingface 🤗 & Google Jax/Flax team for such a wonderful community week. Especially for providing such massive computing resource. Big thanks to @patil-suraj & @patrickvonplaten for mentoring during whole week.
