Mari mBERT: Fine-tuning Multilingual BERT for the Mari Language
This repository contains the code and methodology for fine-tuning bert-base-multilingual-cased (mBERT) on a monolingual Mari language corpus using Masked Language Modeling (MLM). The project was developed as part of a Master's thesis in Linguistics at Volga State University of Technology (VSTU), within the research area of AI-assisted language revitalization.
Background
The Mari language is a Finno-Ugric language of the Volga region classified as low-resource: despite an active speaker community, it remains underrepresented in digital NLP infrastructure. This project addresses that gap by adapting a multilingual transformer model to Mari through targeted fine-tuning, following a methodology comparable to the LakotaBERT project.
The central technical contribution is the extension of the mBERT tokenizer vocabulary with Mari-specific graphemes absent from the standard token set, combined with MLM fine-tuning on a large monolingual corpus. The result is a reproducible, open-source baseline model for Mari NLP tasks.
Evaluation was performed on 2,000 sampled sequences (13,292 masked positions), following the LakotaBERT Table 4 protocol.
Metric
Mari mBERT (this work)
LakotaBERT
mBERT baseline
Accuracy
53.57%
51.48%
54.42%
Precision
0.54
0.56
0.50
F1-Score
0.54
0.49
0.54
MRR
0.63
0.51
0.51
CER
0.32
0.43
0.38
Hit@10
0.81
0.31
0.24
BLEU
0.41
0.09
0.09
The model shows strong ranking performance (MRR, Hit@10) relative to comparable low-resource projects, attributable in part to the typological proximity of Mari to other Finno-Ugric languages already represented in the mBERT pretraining corpus (Finnish, Estonian, Hungarian).
Output Artifacts
After training, the following files are saved to the output directory:
Prepare the corpus as a JSONL file with one field text per line, then run:
python mari_mbert_a100.py
Key environment variables for configuration:
Variable
Default
Description
SEQ
200
Sequence length
BS
128
Batch size per device
EP
5
Number of training epochs
LR
3e-5
Learning rate
WORKERS
16
DataLoader worker count
METRICS_SAMPLE
2000
Sequences sampled for evaluation
USE_COMPILE
0
Enable torch.compile (experimental)
Intended Applications
The fine-tuned model is intended as a foundation for downstream Mari NLP tools, including spell-checking and autocorrection systems, predictive text input for mobile devices and messengers, fill-in-the-blank and grammar exercises for language learners, and further fine-tuning for POS tagging and syntactic parsing.
Limitations
The training corpus is monolingual; parallel Mari-Russian data was not used
Dialectal variation between Hill Mari and Meadow Mari is not separated in the corpus
Automated metrics do not substitute for evaluation by native speakers
Reproducibility
All code is published under an open license. The training corpus is publicly available on Hugging Face. Any researcher can reproduce the full experiment using the provided script and corpus.
Citation
If you use this work, please cite:
Zadvornykh, A. S. Use of AI systems to preserve and revitalize languages. Yoshkar-Ola, 24 June 2026. Yoshkar-Ola: VSTU, 2026.
Lexei1297/mari_mbert
Acknowledgments
Mari Laboratory for the open monolingual Mari corpus
Google Research for the bert-base-multilingual-cased model
The LakotaBERT project for the evaluation methodology
