This is a v1 trained on a single-domain corpus. Dialect transfer is partial: it shifts most on finance/market topics and still tends to fall back to MSA on general prompts. Not production-ready. Shared for experimentation and as a training-pipeline reference.

Training data

~19k short text segments scraped from a Saudi stock-market forum (hawamer.com), cleaned and filtered for dialect density (dedup, boilerplate removal, dialect-marker scoring, thread/author capping). Register skews adult, male, finance-oriented, Najdi-leaning.

Training setup

Method: QLoRA (4-bit NF4 base + LoRA), bf16 compute
LoRA: r=8, alpha=16, dropout=0.05, all attention + MLP projections
1 epoch, lr 2e-4 cosine, effective batch 16, max_len 512
Held-out perplexity (300 samples): ~21.5

Limitations

Single-domain (finance) data caps general-topic dialect quality
Occasional refusals / MSA fallback inherited from the instruct base
Rare language instability on longer generations
Not evaluated by native-speaker dialect rating yet

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct", dtype=torch.bfloat16, device_map="cuda")
model = PeftModel.from_pretrained(base, "AbdulrahmanFA/saudi-dialect-qwen2.5-3b-lora-v1")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")