A masked language model trained from scratch on Old East Slavic and Old Church Slavonic texts, using a RoFormer architecture with BPE tokenisation. Based on mini-roformer-ancient-rus-v2.

Note: BPE token boundaries do not always align with lacuna boundaries in editorial markup, which inflates span-level CER. For character-level restoration tasks consider using DualEmbLM instead.

Architecture

Tokenisation: BPE (Byte Pair Encoding), vocabulary size 50k
Architecture: RoFormer encoder with Rotary Position Embeddings (RoPE)
Size: 6 layers, hidden size 512, 8 attention heads

Training

The model was trained on a corpus of Old Russian and Church Slavonic texts assembled from the following sources:

Source	Language	Word Tokens	Link
Birchbark manuscripts	Old Novgorodian (mostly)	19,045	gramoty.ru
Epigraphy	Old Church Slavonic (mostly)	7,095	epigraphica.ru
DIACU	Old Church Slavonic; Church Slavonic (Old Russian, Middle Bulgarian, Serbian, Resava recensions); Middle Russian	1,588,323	ACL Anthology
TOROT	Old Russian; Church Slavonic	603,047	torottreebank.github.io
Bible (Ponomar)	Church Slavonic	682,430	GitHub
Byliny	Old Russian (11th–17th c.)	42,412	rusneb.ru
Pushkin House	Old Russian	430,103	lib2.pushkinskijdom.ru
Military Statute (Part 2)	Old Russian	49,787	rusneb.ru
NKRYA (historical)	Old Russian (11th–18th c.), Old Novgorodian	327,315	ruscorpora.ru

Masking details: MLM probability 8%, span masking, edge masking, random gap augmentation.

Usage

python

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained(
    "MaximEremeev/RoFormer-slav",
    trust_remote_code=True,
)
model = AutoModelForMaskedLM.from_pretrained(
    "MaximEremeev/RoFormer-slav",
    trust_remote_code=True,
)

Tasks

Generated lacunae restoration (Test A Hit@1: 0.281, CER: 0.839)
Real lacunae restoration (Test B char Hit@1: 0.145, span Hit@1: 0.021)

Contact

Maxim Eremeev, maeremeev@edu.hse.ru