LoRA adapter for ALLaM-7B-Instruct-preview, fine-tuned on the synthetically generated portion of InstAr-500k under a fixed budget of 1M training tokens. One of six adapters from a controlled study comparing human-curated versus synthetic Arabic instruction data under matched token budgets.

Model Details

Base model: humain-ai/ALLaM-7B-Instruct-preview
Adapter type: LoRA (QLoRA, 4-bit NF4)
Training data: InstAr-500k (synthetically generated portion; human-crafted sources, including CIDAR, excluded)
Token budget: 1M tokens
Language: Arabic

Training Configuration

Setting	Value
Quantization	4-bit NF4 (QLoRA)
LoRA rank / alpha	16 / 32
LoRA dropout	0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Optimizer	Paged AdamW (8-bit)
Learning rate	2e-4
LR scheduler	cosine, 100 warmup steps
Epochs	3
Effective batch size	16 (2 × 8 grad. accum.)
Max sequence length	512
Precision	fp16
Seed	42
Hardware	NVIDIA A100 (40GB)

Evaluation

Evaluated with lm-evaluation-harness on seven Arabic benchmarks (ACVA 5-shot; others zero-shot). Accuracy:

Benchmark	Score
Arab Culture	0.355
AlGhafa	0.538
AraDiCE	0.584
ACVA	0.753
Arabic Exams	0.497
ArabicMMLU	0.640
OpenAI MMLU (Ar)	0.378

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "humain-ai/ALLaM-7B-Instruct-preview"
tok = AutoTokenizer.from_pretrained(base, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(base, trust_remote_code=True, device_map="auto")
model = PeftModel.from_pretrained(model, "ManarAlrabie/arabic-llm-synthetic-1m")

Intended Use & Limitations

Research artifact for studying instruction-data quality vs. quantity in Arabic LLM fine-tuning. Not intended for production. As a single-seed fine-tune of a 7B model, outputs may contain inaccuracies.

Citation

Associated paper is under review; citation will be added upon publication. Until then, please link to this repository.