Property	Value
Parameters	2,088,768,048 (2.09B)
Architecture	NanoChatGPT (Transformer decoder)
Layers	24
Hidden size	1536
Attention heads	12
Vocabulary	65,536 (Arabic BPE)
Context length	2,048 tokens
Precision	bfloat16
Language	Arabic

Training

Pretraining

Dataset: AraMix (8.19B tokens, Chinchilla-optimal)
Steps: 7,812 steps
Hardware: 8x NVIDIA H20 GPUs
Batch size: 1,048,576 tokens

SFT (Supervised Fine-Tuning)

Datasets: alpaca-gpt4-arabic (50K) + CIDAR (10K) + ArabicMMLU (x3) + ArabicGSM8K (x4) + Fasih identity conversations
Total mixture: 90,621 rows
Steps: 16 (1 epoch)

Evaluation Results

Benchmark	Accuracy	Random Baseline
ArabicMMLU	29.91%	25%
ACVA	49.29%	50%
ArabicGSM8K	4.47%	0%
ChatCORE	0.0175	0.0

Usage

This model uses the nanochat framework. To use it:

bash

git clone https://github.com/karpathy/nanochat
cd nanochat
# Copy model files to $NANOCHAT_BASE_DIR/chatsft_checkpoints/d24/
python -m scripts.chat_web -i sft

Chat Format

The model uses nanochat's special token format:

<|bos|><|user_start|>مرحبا، من أنت؟<|user_end|><|assistant_start|>

Limitations

Small model (2B params) — limited knowledge and reasoning compared to larger models
Trained primarily on Arabic text — limited multilingual capability
Short context (2048 tokens)
This is a research/educational model trained from scratch in ~12 hours

License

Apache 2.0

HeshamHaroon/Fasih-2B

Fasih-2B (فصيح)

Model Details

Training

Pretraining

SFT (Supervised Fine-Tuning)

Evaluation Results

Usage

Chat Format

Limitations

License

HeshamHaroon/
Fasih-2B