uzbek-gpt-103m
A 103M-parameter, decoder-only transformer for Uzbek, pre-trained from scratch — not fine-tuned from a multilingual base. Trained on a single consumer GPU in a few hours for a few dollars, it reaches a lower bits-per-byte than a 1.3B multilingual model, while being roughly one-thirteenth the size.
The project's thesis: for a low-resource language, a purpose-built tokenizer plus a small model trained from scratch can beat a much larger multilingual model — and the advantage comes from the tokenizer, not from training from scratch (shown by the experiments below). The paired tokenizer is
IslombekT/uzbek-bpe-16k.
- Author: Islombek Turdiyev (GitHub · website)
- Language: Uzbek (Latin script,
uzn_Latn)
- Architecture: decoder-only transformer with RoPE, RMSNorm, and SwiGLU
- Parameters: ~103M
- Tokenizer: uzbek-bpe-16k (16,384 vocab)
Results
Models are compared by bits-per-byte on held-out Uzbek text — how well a model predicts raw text, independent of how it tokenizes (the fair metric across different tokenizers). Lower is better.
| Model | Params | Approach | Bits/byte ↓ |
|---|
| uzbek-gpt-103m (this model) | ~103M | trained from scratch | 1.105 |
| mGPT-1.3B | 1.3B | base, zero-shot | 1.163 |
| mGPT-1.3B | 1.3B | QLoRA fine-tune | 1.147 |
A ~13× smaller model, trained from nothing, comes out ahead. Two controlled experiments show why:
- The advantage is fair. Re-training the mGPT+QLoRA baseline on 10× more Uzbek data (1M → 10M tokens) improves it by only 0.002 bits/byte; it plateaus at ~1.120, still above this model's 1.105. A bootstrap gives the advantage over the zero-shot baseline as 0.058 bits/byte (95% CI [0.037, 0.078]) — a resolvable gap, not noise. More data does not close it.
- The advantage is the tokenizer. The same architecture trained from scratch on the same text but with mGPT's tokenizer — a larger, 232M-parameter model — scores 1.158 bits/byte, worse than this 103M model. With architecture, data, and training held constant, the tokenizer is the deciding factor.
So the advantage comes from a tokenizer that fits the language, not from model size or from training from scratch. Full experiments and scripts are in the
repository.
Training
| |
|---|
| Data | ~1.06B Uzbek tokens (FineWeb-2, uzn_Latn) |
| Hardware | 1× NVIDIA RTX 4090 |
| Wall-clock | ~3.4 hours |
| Cost | ~$3.60 |
| Best validation loss | 3.059 |
| Tokenizer | uzbek-bpe-16k · fertility 1.839 tokens/word (lowest of 10 benchmarked) |
Intended use
Research and experimentation with Uzbek language modeling, tokenizer studies, and low-resource-NLP baselines. This is a base model — it predicts the next token and has not been instruction-tuned or aligned, so it is not a chat assistant and should not be deployed in user-facing products without further training and evaluation.
Limitations and bias
Trained on a web corpus, the model reflects the coverage, style, and biases of that data, and can produce inaccurate or inappropriate text. It targets Uzbek in Latin script; Cyrillic-script input is out of scope. As a small base model, generation quality is limited, and it has no factual grounding or knowledge of events beyond its training data.
How to use
The model uses a custom from-scratch architecture. See the training and inference code in the repository:
github.com/TurdiyevIslombek/uzbek-gpt-from-scratch — it includes the model definition, the tokenizer, a generation script, and the evaluation experiments.
Citation
1@misc{turdiyev2026uzbekgpt103m,
2 title = {uzbek-gpt-103m: a from-scratch language model for Uzbek},
3 author = {Turdiyev, Islombek},
4 year = {2026},
5 howpublished = {\url{https://huggingface.co/IslombekT/uzbek-gpt-103m}}
6}
Related
