You can deploy this model and start earning money today!

Babajaan/
nanochat-ar

Name: nanochat-ar
Availability: InStock

Views

No views yet

nanochat-ar: Arabic LLM Proof Run on Aziz HPC

This repository contains a small Arabic nanochat proof-of-competence checkpoint trained from scratch on King Abdulaziz University Aziz HPC.

This is not a production Arabic LLM and does not claim state-of-the-art performance. The purpose is to document and release an end-to-end HPC training artifact: Arabic corpus processing, tokenizer training, base pretraining, evaluation, supervised fine-tuning, and checkpoint generation.

What is included

Arabic tokenizer: tokenizer/tokenizer.pkl and tokenizer/token_bytes.pt
Base checkpoint: base_checkpoints/ar-d12-eager/model_000050.pt
SFT checkpoint: chatsft_checkpoints/ar-d12-eager/model_000002.pt
Metadata JSON files for both checkpoints
Training/evaluation reports in reports/
Final HPC scripts in scripts/
200-example SFT sample in data_samples/

Optimizer states and the full raw corpus are not uploaded.

Training summary

Stage	Result
Arabic Wikipedia dump	1,219,201 articles
Clean train split	1,190,864 lines
Clean validation split	12,028 lines
Tokenizer vocabulary	32,768
Base model	12 layers, 286,261,730 parameters
Base checkpoint	ar-d12-eager, step 50
Base validation BPB	1.214387
Base eval train BPB	1.118719
Base eval validation BPB	1.214387
SFT dataset	2,000 generated Arabic instruction examples
SFT checkpoint	chatsft_checkpoints/ar-d12-eager, step 2
SFT validation BPB	1.0228

How to load with nanochat

Clone karpathy/nanochat, install its environment, then use this repository as the NANOCHAT_BASE_DIR layout.

Commands:

export NANOCHAT_BASE_DIR=/path/to/this/repo
export NANOCHAT_DTYPE=float32
cd /path/to/nanochat
source .venv/bin/activate
python -m scripts.chat_cli --device-type cpu --source sft --model-tag ar-d12-eager --step 2 --prompt 'مرحبا، من أنت؟'

For GPU inference, use an A100 or similar GPU and the standard nanochat chat CLI/web server.

Limitations

Very short proof training run: base model trained for only 50 steps.
SFT is a compact generated instruction dataset, not a curated production instruction set.
Chat quality is expected to be weak and repetitive.
This release is best used as a reproducibility artifact and HPC workflow demonstration.

Citation / Acknowledgement

This project used King Abdulaziz University Aziz HPC resources. Built on karpathy/nanochat under the MIT license.

Babajaan/nanochat-ar