Fine-tuned Qwen2.5-1.5B-Instruct model for Palestinian Arabic Dialect generation and translation, prepared for the AMIYA (Arabic Modeling In Your Accent) Shared Task at VarDial 2026.

Model Details

Base Model: Qwen/Qwen2.5-1.5B-Instruct
Fine-tuning Method: LoRA (Low-Rank Adaptation)
Target Dialect: Palestinian Arabic (Palestinian DA)
Task: Generation and Translation (MSA↔Palestinian, English↔Palestinian)
Competition: AMIYA Shared Task @ VarDial 2026

Model Architecture

Base Model: Qwen2.5-1.5B-Instruct (1.5B parameters)
Adapter: LoRA with rank=16, alpha=32
Target Modules: q_proj, k_proj, v_proj, o_proj
Trainable Parameters: ~4.2M (0.28% of base model)

Training Details

Training Data

This model was fine-tuned on Palestinian Arabic dialect data prepared from multiple publicly available sources:

AMIYA Training Dataset (amiya_data/train.jsonl)
- Source: AMIYA Shared Task training data
- Examples: 15,000 training examples (sampled from 38,610 available)
- Task Types: Generation (100%)
- Format: Qwen2.5 chat template formatted instruction-output pairs
Additional Data Sources Used for Data Preparation:
- Combined Dialect Dataset: Aggregated Palestinian dialect text examples
- Maknuune Corpus (v1.0.1): Palestinian Arabic dialect lexicon with translation pairs
- Shami Dataset: Palestinian dialect corpus
- Casablanca Dataset: Palestinian dialect speech transcriptions

Training Configuration

Epochs: 1
Batch Size: 16 (per device) × 2 (gradient accumulation) = 32 effective
Learning Rate: 2e-4
Max Sequence Length: 256 tokens
Warmup Steps: 100
Optimization: bfloat16 mixed precision, gradient checkpointing
Training Time: <8 hours on single GPU

AMIYA Track Classification

Track: Open

This submission uses:

✅ Publicly available base model (Qwen/Qwen2.5-1.5B-Instruct)
✅ Publicly available training data sources (Maknuune, Shami, Casablanca, Combined Dataset)
✅ AMIYA provided training data

The model was trained on data prepared from publicly available Palestinian dialect corpora, in addition to the AMIYA provided training set.

Usage

Installation

pip install transformers peft torch

Loading the Model

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct",
    torch_dtype="auto",
    device_map="auto"
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "Khamad/qwen2.5-1.5b-amiya-palestinian")

# Set padding token
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

Generation Example

python

# Format prompt using Qwen2.5 chat template
prompt = """<|im_start|>system
You are a helpful assistant that generates text in Palestinian Arabic dialect. Write naturally in Palestinian dialect.<|im_end|>
<|im_start|>user
Write a greeting in Palestinian dialect.<|im_end|>
<|im_start|>assistant
"""

# Tokenize
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate
outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    temperature=0.7,
    do_sample=True,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id
)

# Decode
response = tokenizer.decode(outputs[0], skip_special_tokens=False)
print(response)

Translation Example

python

# MSA to Palestinian translation
prompt = """<|im_start|>system
Translate the following Modern Standard Arabic text to Palestinian Arabic dialect.<|im_end|>
<|im_start|>user
السلام عليكم ورحمة الله وبركاته<|im_end|>
<|im_start|>assistant
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50, temperature=0.7)
translation = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(translation)

Evaluation

This model is evaluated on the AMIYA Shared Task benchmark (AL-QASIDA) with the following metrics:

ADI2 Dialect Fidelity Score: Measures dialect authenticity
chrF++ Translation Score: Evaluates translation quality (DA↔English, DA↔MSA)
Human Evaluation: Fluency and dialect adherence

Limitations

Model is fine-tuned primarily on generation tasks (translation examples were limited during training)
Small model size (1.5B parameters) may limit performance on complex translation tasks
Training data focused on Palestinian dialect, performance on other Arabic dialects may vary

Citation

If you use this model, please cite:

bibtex

@misc{qwen2.5-1.5b-amiya-palestinian,
  title={Qwen2.5-1.5B-AMIYA-Palestinian: Fine-tuned Model for Palestinian Arabic Dialect},
  author={Khamad},
  year={2025},
  howpublished={\url{https://huggingface.co/Khamad/qwen2.5-1.5b-amiya-palestinian}},
  note={Submission to AMIYA Shared Task @ VarDial 2026}
}

References

AMIYA Shared Task: VarDial 2026
Base Model: Qwen2.5-1.5B-Instruct
LoRA Method: LoRA: Low-Rank Adaptation of Large Language Models
VarDial 2026: EACL 2026 Workshop in Rabat, Morocco

License

This model is released under the Apache 2.0 license, consistent with the base Qwen2.5-1.5B-Instruct model.

Contact

For questions about this model or the AMIYA submission:

HuggingFace Model: Khamad/qwen2.5-1.5b-amiya-palestinian
Competition: AMIYA Shared Task @ VarDial 2026

Note: This model was developed for the AMIYA Shared Task evaluation. Results may vary depending on the specific evaluation setup and prompts used.