🤖 Fully AI-generated. Design, code (benchmark.py, benchmark_unt.py), plots and this card were produced end-to-end by an AI agent (Claude, Hermes ML research loop). Compute: a single Kaggle T4, inference-only, no training. No credentials used or embedded — all models/datasets are public and the code is included for transparency.

ModernBERT-style comparison of KazBERT vs four Kazakh-capable encoders, zero-shot, across tokenizer efficiency, exam-QA accuracy and embedding quality.

TL;DR

⚡ Smallest & fastest: KazBERT hits top-tier quality with a 32k vocab (vs 120k–250k) and the fastest inference.
🎯 Beats multilingual on Kazakh: on tokenizer efficiency and MLM answer-scoring (PLL), KazBERT and the Kazakh-specific models clearly outperform mBERT and XLM-R.
📊 Two evaluations: (1) dastur-mc reading MC, (2) the full Kazakh ЕНТ / UNT — 14,850 real exam questions across 7 subjects.
🥈 Honest: KazBERT isn't #1 on every metric, but it's consistently in the leading group at the smallest footprint.

Part 1 — dastur-mc (reading comprehension MC) + tokenizer

model	vocab	fertility ↓	MC-PLL ↑	MC-embed ↑	sec
KazBERT	`32k`	1.636	42.4%	36.8%	80.5
mBERT	`119k`	2.759	27.5%	45.8%	271.6
XLM-R base	`250k`	2.150	34.8%	33.8%	262.8
kaz-roberta	`52k`	1.598	44.5%	38.8%	60.4
KazakhBERTmulti	`100k`	1.457	36.6%	40.5%	95.6

(bold = best in column; random MC = 25%)


fertility	mc-pll
mc-embed	embed-sep

Fertility — subword tokens per Kazakh word (lower = better tokenizer). Fair across models.
MC-PLL — zero-shot MC where each option is scored by length-normalised pseudo-log-likelihood under the MLM head. Main task signal.
MC-embed — same task via cosine of mean-pooled embeddings (diagnostic; all these encoders are MLM-only, so raw embeddings are weak for everyone).

Qualitative fill-mask (illustrative, not scored)

Астана — Қазақстанның [MASK] қаласы.

model	top-3
KazBERT	`астана`, `ірі`, `алматы`
mBERT	`Астана`, `бар`, `1`
XLM-R base	`бас`, `1`, `19`

Мен қазақ [MASK] сөйлеймін.

model	top-3
KazBERT	`тілінде`, `тіліне`, `тілі`
mBERT	`##қа`, `##стан`, `##та`
XLM-R base	`тілінде`, `ша`, `тілін`

Абай Құнанбаев — ұлы қазақ [MASK].

model	top-3
KazBERT	`ақыны`, `энциклопедиясы`, `сср`
mBERT	`[UNK]`, `##ты`, `##тар`
XLM-R base	`ақын`, `жазушы`, `ғалым`

KazBERT and the Kazakh-trained models return fluent Kazakh; mBERT often falls back to sub-word fragments or [UNK].

Part 2 — Full Kazakh ЕНТ / UNT (14,850 questions, 7 subjects)

Zero-shot on every question of kazakh-unified-national-testing-mc (4–8 options each). Random baseline ≈ 17.9%.

model	ЕНТ acc — PLL ↑	ЕНТ acc — embed ↑	sec
KazBERT	22.3%	21.3%	801.8
mBERT	22.1%	23.3%	2631.3
XLM-R base	21.9%	20.9%	2867.7
kaz-roberta	22.7%	24.2%	680.2
KazakhBERTmulti	20.7%	22.5%	1173.4

Per-subject (PLL accuracy)

⚠️ Caveats

Zero-shot, no fine-tuning — intrinsic probes, not fine-tuned scores; fine-tuning could reorder models.
Cross-tokenizer comparisons are imperfect — PLL/fill-mask depend on how each vocab segments text; numbers are indicative.
MLM answer-scoring ≠ reasoning — exams need world knowledge these small encoders lack, so absolute accuracies are modest; the point is the relative Kazakh-language signal.

Reproduce

Runs on a free Kaggle T4 with pip install transformers datasets — public models/datasets only:

bash

python benchmark.py       # Part 1: dastur-mc + tokenizer
python benchmark_unt.py   # Part 2: full ЕНТ

Provenance

Models: KazBERT, mBERT, XLM-R base, kaz-roberta-conversational, KazakhBERTmulti
Data: kazakh-unified-national-testing-mc, kazakh-dastur-mc, kazakh_wiki_articles
Compute: Kaggle Notebook, 1× NVIDIA Tesla T4 · Pipeline: Hermes ml-research-loop
Author: AI agent (Claude). Human: chose the models and pressed go.

Eraly-ml/KazBERT-benchmark

KazBERT Benchmark 🇰🇿 — Kazakh Encoders Compared