You can deploy this model and start earning money today!

NIRVLab/
xnli-morphel-ru-32k

Name: xnli-morphel-ru-32k
Availability: InStock

Views

No views yet

NIRVLab — MorpheL Tokenizer for Russian XNLI

MorpheL: MI-Guided Stochastic Segmentation tokenizer for morphologically rich low-resource languages (Proposal: MorpheL: Morphology-Aware Tokenizer Adaptation for Pretrained Models in Low-Resource Languages).

Trained on the Russian (ru) subset of facebook/xnli — all splits.

Algorithm

MorpheL scores candidate intra-word boundaries via pointwise mutual information (MI) between prefix and suffix substrings, then stochastically selects the number of cuts via Gumbel perturbation (Eq. 6–8). Key distinctions from BPE-Dropout:

Randomness is MI-informed — only high-MI boundaries enter the candidate pool
Vowel-consonant transition heuristic pre-filters linguistically implausible positions
Global MI table aggregated over full corpus (not per-sentence) for stability

Training Config

Parameter	Value
Algorithm	MorpheL (MI + Gumbel)
Vocabulary size	32,083
top_k	4
temperature (T)	1.0 (vocab induction: T=0)
mi_threshold	0.0 (keep MI > 0)
min_frequency	2
Special tokens	`<s>, <pad>, </s>, <unk>, <mask>`
Corpus	`facebook/xnli/ru` — all splits (800,404 sentences)
Vowel set	Russian Cyrillic (а е ё и о у ы э ю я — 10 vowel letters; й/ь/ъ excluded)

Evaluation Metrics (vs Baselines, vocab_size=32000, same corpus)

Metric	BPE	WordPiece	Unigram	MorpheL
Fertility ↓	—	—	—	1.4282
Tokens/char ↓	—	—	—	0.2076
Avg seq len ↓	—	—	—	19.27
Vocab coverage ↑	—	—	—	1.0000
OOV rate ↓	—	—	—	0.0000

Fill baseline columns after running baseline notebooks.

Usage

python

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("NIRVLab/xnli-morphel-ru-32k")

Note: MorpheL segments words before passing to the tokenizer. At downstream training time, use temperature=1.0 for stochastic segmentation. For inference, use temperature=0 (deterministic).