🇵🇰 ZabaanAI - Pakistan Multilingual Language Model
An open-source, CPU-friendly AI model for Pakistan's major languages
Python
License
Model Size
Languages🌐 Supported Languages
| Language | Script | Coverage |
|---|
| Urdu | Arabic (Nastaliq) | ✅ News, poetry, social media |
| Punjabi (Shahmukhi) | Arabic | ✅ News, literature, transliteration |
| Sindhi | Arabic | ✅ Large corpus (505M tokens) |
| Saraiki | Arabic | ⚠️ Augmented from related languages |
| Balochi | Arabic | ✅ Small curated datasets |
| Pashto | Arabic | ✅ ZamAI corpus, textbooks, news |
| Hindko | Arabic | ⚠️ Augmented from related languages |
| Brahui | Arabic | ⚠️ Bridged from Balochi |
| Kashmiri | Arabic | ✅ Dedicated corpora + news |
| English | Latin | ✅ Wikipedia + parallel corpora |
| Roman Urdu | Latin | ✅ Social media, code-mixed |
🏗️ Base Model: google/mt5-small
We chose mT5-small over XLM-R and mBERT because it's the only architecture that natively supports all required tasks:
| Requirement | mT5-small ✅ | XLM-R ❌ | mBERT ❌ |
|---|
| Text Generation | ✅ Native seq2seq | Encoder-only | Encoder-only |
| Translation | ✅ Native | Can't generate | Can't generate |
| Summarization | ✅ Native | Can't generate | Can't generate |
| Chatbot/QA | ✅ Native | Can't generate | Can't generate |
| Sentiment | ✅ Text-to-text | Native | Native |
| CPU Efficiency | ✅ 47% lower latency* | Good | Good |
*Encoder-decoder achieves 47% lower first-token latency and 4.7x higher throughput on CPU vs decoder-only (Microsoft 2025).
🎯 Supported Tasks
All tasks use mT5's unified text-to-text format:
| Task | Input Format | Example Output |
|---|
| Text Generation | generate in Urdu: {prompt} | "پاکستان ایک خوبصورت ملک ہے..." |
| Translation | translate English to Urdu: {text} | "پاکستان ایک خوبصورت ملک ہے" |
| Summarization | summarize: {long_text} | "Short summary..." |
| Sentiment | sentiment: {text} | "positive" / "negative" / "neutral" |
| QA | question: {Q} context: {C} | "Answer text..." |
| Chatbot | conversation: user: {msg} assistant: | "Response text..." |
📂 Repository Structure
├── scripts/
│ ├── 01_collect_datasets.py # Download 35+ HF datasets
│ ├── 02_clean_text.py # Clean, deduplicate, normalize
│ ├── 03_train_tokenizer.py # Extend mT5 tokenizer
│ ├── 04_prepare_training_data.py # T5 span corruption prep
│ ├── 05_train_cpu.py # CPU-optimized LoRA pretraining
│ ├── 06_finetune_lora.py # LoRA fine-tuning (5 tasks)
│ ├── 07_evaluate.py # BLEU, ROUGE, perplexity
│ ├── 08_quantize.py # ONNX INT8 / 8-bit quantization
│ └── 09_deploy_space.py # HF Space deployment
├── docs/
│ ├── 01_DATASET_COLLECTION.md # Full dataset sourcing guide
│ ├── 02_TOKENIZER_GUIDE.md # Tokenizer improvement strategy
│ ├── 03_TRAINING_GUIDE.md # CPU training configuration
│ ├── 04_FINETUNING_GUIDE.md # LoRA fine-tuning recipes
│ └── 05_DEPLOYMENT_GUIDE.md # 5 deployment options
├── deployment/
│ ├── app.py # Gradio web app (6 tasks)
│ ├── api_server.py # FastAPI REST API
│ ├── Dockerfile # Docker container
│ └── requirements.txt # All dependencies
├── notebooks/
│ └── 01_explore_datasets.ipynb # Dataset visualization
├── tokenizer/ # Extended tokenizer (250,141 vocab)
├── config.yaml # Central configuration
├── setup.py # Environment setup
├── QUICKSTART.md # 10-step quick start
└── README.md # This file
⚡ Quick Start
1. Install Dependencies
1pip install transformers datasets tokenizers accelerate peft sentencepiece
2pip install torch --index-url https://download.pytorch.org/whl/cpu
3pip install protobuf tiktoken gradio optimum[onnxruntime]
2. Collect Datasets
python scripts/01_collect_datasets.py3. Clean & Merge
python scripts/02_clean_text.py4. Extend Tokenizer
python scripts/03_train_tokenizer.py5. Prepare Training Data
python scripts/04_prepare_training_data.py6. Start CPU Training
python scripts/05_train_cpu.py --max_steps 50000 --batch_size 2 --lora_r 647. Fine-Tune for Tasks (GPU recommended)
1python scripts/06_finetune_lora.py --task translation --batch_size 8
2python scripts/06_finetune_lora.py --task sentiment --batch_size 8
8. Deploy
python scripts/09_deploy_space.py --username your-username --space_name zabaanai-demo🔧 CPU Training Configuration
| Setting | Value | Why |
|---|
| Base Model | google/mt5-small | 300M params, 250k vocab, seq2seq |
| LoRA Rank | 64 | ~15% of params trainable (~45M) |
| Batch Size | 2 | Low for CPU RAM |
| Gradient Accumulation | 32 | Effective batch = 64 |
| Learning Rate | 2e-4 | Higher LR works with LoRA |
| Max Steps | 50,000 | Minimum for meaningful adaptation |
| Warmup | 500 steps | 1% of total |
| Scheduler | Cosine | Smooth decay |
| Checkpointing | Every 2,000 steps | Auto-resume supported |
| Precision | FP32 | CPU-safe (no fp16/bf16) |
| Gradient Checkpointing | Enabled | 40-50% memory savings |
Training time: ~70 hours for 50K steps on 8-core CPU, or ~3 hours on T4 GPU.
🎯 Task Fine-Tuning
Translation
1python scripts/06_finetune_lora.py --task translation
2# Datasets: Humair332/Vast-Urdu (10M pairs), Kashmiri-English, Pashto-English
Sentiment Analysis
1python scripts/06_finetune_lora.py --task sentiment
2# Datasets: community-datasets/roman_urdu, urdu_sentiment_corpus
Summarization
1python scripts/06_finetune_lora.py --task summarization
2# Datasets: News articles with headlines as pseudo-summaries
Question Answering
1python scripts/06_finetune_lora.py --task qa
2# Datasets: Sindhi encyclopedia, generated Q&A pairs
Chatbot / Conversational AI
1python scripts/06_finetune_lora.py --task chatbot
2# Datasets: Urdu-Instruct, Sindhi SFT, Pashto Alpaca
🚀 Deployment Options
| Option | Best For | Setup | Cost |
|---|
| Hugging Face Spaces | Public demo | 2 minutes | Free / Paid GPU |
| Local CPU Server | Privacy, on-premise | 10 minutes | Free |
| Docker Container | Scalable cloud | 15 minutes | Varies |
| FastAPI API | Production microservices | 10 minutes | Varies |
| ONNX Quantized | Edge/embedded | 5 minutes | Free |
📊 Data Mixture for Pretraining
Urdu: 35% (news, poetry, social media, Wikipedia)
Sindhi: 20% (large corpus available)
Pashto: 15% (ZamAI corpora)
Punjabi: 10% (news, transliteration)
Roman Urdu: 8% (social media, tweets)
English: 7% (prevent catastrophic forgetting)
Kashmiri: 3% (smaller corpus)
Balochi: 1.5% (very limited data)
Saraiki/Hindko/Brahui: 0.5% (augmented from related)
🔤 Tokenizer
mT5's native tokenizer already covers all Pakistan scripts with 250,000 tokens + byte fallback. We extended it with 52 Pakistan-specific tokens:
- Cities:
لاہور, کراچی, اسلام آباد, پشاور, کوئٹہ
- Names:
محمد, علی, فاطمہ
- Institutions:
سپریم کورٹ, نیشنل اسمبلی
- Roman Urdu:
bhai, yaar, theek
Vocabulary size: 250,141 (was 250,100)
📈 Evaluation
1python scripts/07_evaluate.py --model_path training/checkpoints/final --task translation
2# Metrics: BLEU (translation), ROUGE (summarization), Perplexity (generation)
💾 Quantization for Production
1# ONNX INT8 (recommended for CPU)
2python scripts/08_quantize.py --model_path training/checkpoints/final --method onnx
3
4# 8-bit (for GPU)
5python scripts/08_quantize.py --model_path training/checkpoints/final --method 8bit
Benefits: 60-70% smaller model, 2-3x faster inference on CPU.
🤝 Contributing
We welcome contributions! Priority areas:
- More datasets for Saraiki, Hindko, Brahui
- Better evaluation benchmarks
- Fine-tuned task-specific adapters
- Documentation translations
📜 License
Apache 2.0 (same as mT5 base model)
🙏 Acknowledgments
- Google Research for mT5
- ZamAI for Pashto datasets
- aakashMeghwar01 for Sindhi corpus
- Humair332 for Vast-Urdu parallel corpus
- Omarrran for Kashmiri datasets
- community-datasets for Roman Urdu sentiment data
📚 Citation
1@software{zabaanai2025,
2 title = {ZabaanAI: Pakistan Multilingual Language Model},
3 author = {Open Source Contributors},
4 year = {2025},
5 url = {https://huggingface.co/shaikhsalman/zabaanai-pakistan-multilingual}
6}
Built with ❤️ for Pakistan's languages 🇵🇰
