Saudi Trained AI Model — 8.3M Parameters
A small decoder-only language model trained entirely from scratch on a colloquial Saudi/Gulf Arabic corpus, then instruction-fine-tuned to converse in Saudi dialect. Built end-to-end: custom tokenizer, architecture, pretraining, evaluation, and instruction fine-tuning.
Scope, honestly: at ~8.3M parameters this model captures dialect, register, and conversational format well. It is knowledge-thin and is NOT a factual assistant. It will often produce a fluent, plausible, but factually wrong answer.
Model details
- Architecture: decoder-only transformer (mini-LLaMA style)
- Parameters: 8,279,712 (~8.3M)
- Components: RMSNorm (pre-norm), RoPE, SwiGLU, weight-tied embeddings, no biases
- Tokenizer: byte-level BPE, vocab 8,000
- Context length: 256 tokens; trained in bf16
Usage
Install: pip install torch tokenizers numpy
Run: python3 sft_chat.py --ckpt sft_ckpt.pt
The SFT model expects this prompt template (handled automatically by sft_chat.py):
السؤال:
{question}
الجواب:
Example — السؤال: كيف حالك؟ -> الجواب: الله يحييك، تمام الحمد لله. تفضّل وش تبي؟
Results
- Base pretraining: test perplexity ~118 (valid ~109).
- Instruction tuning: reliably answers in dialect (greetings convincing); factual accuracy limited by answer-side data diversity and small model capacity.
Files
- sft_ckpt.pt — recommended instruction-tuned model (v1)
- sft_ckpt_v2.pt — variant (1 epoch, higher dropout)
- ckpt.pt — base pretrained model
- tokenizer.json — tokenizer; model.py — definition (required to load)
- sft_chat.py / sample.py — inference; train.py / finetune.py / prepare_data.py / train_tokenizer.py / eval.py — pipeline
Limitations & intended use
Educational / research / dialect-style generation. NOT for factual answers or financial/medical/legal advice. May generate confident but incorrect content.
