
Steps	10,000
Batch size	32
Sequence length	256
Train blocks	29,859 (≈7.6M tokens)
Optimizer	AdamW, lr 1e-4, warmup 10%, weight decay 0.01
Precision	fp16
Hardware	RTX A4000 (16 GB)
Wallclock	17.2 min

Evaluation

metric	value
eval_loss	7.6138
eval_perplexity	2025.952

Usage

python

from transformers import AutoTokenizer, AutoModelForMaskedLM
tok = AutoTokenizer.from_pretrained("Abzalbek89/kk-bert-small-bpe")
model = AutoModelForMaskedLM.from_pretrained("Abzalbek89/kk-bert-small-bpe")

Citation

If you use this model, please cite the upcoming paper:

Tokenizer Optimization for Kazakh Small Language Models: A Comparative Study of BPE, Unigram, and Morphological Segmentation. (in prep.)

Companion artifacts

Aggregate metrics: Abzalbek89/kk-tokenizer-fertility-baseline
Sister models: kk-bert-small-bpe, kk-bert-small-unigram, kk-bert-small-morph-bpe