Emhotob is a family of small Arabic language models pretrained from scratch on Arabic web text. This is the 5M rung of the ladder (~5.08M parameters), part of a scaling series ranging from 500K to 25M parameters that all share the same tokenizer, context length, and training recipe.

⚠️ These are tiny, research-scale models trained on a limited token budget. They are intended for scaling-law experiments, education, and Arabic NLP research — not for production use.

Model details

Property	Value
Architecture	Llama (decoder-only, RoPE, GQA)
Parameters	5,080,448 (~5.08M)
Hidden size	128
Layers	5
Attention heads	4 (KV heads: 2)
Intermediate size	384
Context length	2048
Vocabulary	32,000 (custom Byte-Level BPE)
Tied embeddings	Yes
RoPE theta	10,000
Precision	bf16

Training

Property	Value
Data	`kaust-generative-ai/fineweb-edu-ar` (Arabic)
Tokens seen	~500M (1 epoch)
Optimizer	AdamW (fused), β=(0.9, 0.95), wd=0.1
LR schedule	6e-4, cosine, 2% warmup
Effective batch	128 sequences × 2048 tokens
Grad clipping	1.0

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "oddadmix/Emhotob-5M"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)

prompt = "الذكاء الاصطناعي هو"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50, do_sample=True, top_p=0.9, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))

Limitations

Given its size and limited pretraining budget, Emhotob-5M has a narrow capability range and will produce factually unreliable and sometimes incoherent text. It has not been instruction-tuned or aligned, and no safety filtering has been applied. Use accordingly.