MaithiliBERT is a RoBERTa-base language model (124M parameters) pretrained from scratch on MaithiliCorpus — the largest open Maithili text corpus comprising 39.8M clean, LID-filtered, and deduplicated words. This is the first RoBERTa ever trained specifically for Maithili, a low-resource Indo-Aryan language spoken by approximately 50 million people primarily in Bihar, India and Nepal.

Model Architecture

Property	Value
Architecture	RoBERTa-base
Parameters	124,493,648
Vocabulary	Byte-level BPE, 50,000 tokens
Context Window	512 tokens
Hidden Size	768
Intermediate Size	3072
Attention Heads	12
Hidden Layers	12
Training Data	MaithiliCorpus gold_deduped (47,882 documents, 39.8M words)
Training Steps	14,210 (10 epochs)
Effective Batch Size	32
Optimizer	AdamW (lr=5e-4)
Precision	FP16
License	CC-BY-4.0

Tokenizer

The tokenizer is a byte-level BPE trained from scratch on MaithiliCorpus with a vocabulary of 50,000 tokens — 60% smaller than mBERT (119K) and 80% smaller than XLM-R/IndicBERTv2 (250K). This reduces embedding parameters from 192M to 38.4M, shrinking the embedding layer from 61% to 31% of total model parameters.

Property	Value
Algorithm	Byte-level BPE
Vocabulary Size	50,000
Training Data	MaithiliCorpus (39.8M words)
Training	SentencePiece, `--model_type=bpe`
Special Tokens	`<s>`, `</s>`, `<unk>`, `<pad>`, `<mask>`
UNK Rate (on Maithili)	0.00%

Acknowledgments

The authors gratefully acknowledge AnK Komputing for providing the computational resources and infrastructure used to conduct the simulations/analyses presented in this work. We thank the team at AnK Komputing for their technical support and access to their high-performance computing services.

Intended Use

Masked language modeling (fill-mask)
Fine-tuning for text classification, NER, and other NLU tasks
Feature extraction for downstream Maithili NLP

Usage

python

from transformers import pipeline
fill = pipeline("fill-mask", model="AiventraLab/maithili-roberta")
results = fill("भगवान् <mask>क ध्यान")