This is a 1.5B parameter small language model (SLM) fine-tuned on a synthetically generated dataset for Arabic summarization. High-quality summaries were generated using a larger model ("Qwen/Qwen2.5-14B-Instruct-AWQ") on Arabic documents derived from the GEM/xlsum dataset.

The model was trained using supervised fine-tuning (SFT) with LoRA adapters, enabling training on consumer GPUs with limited memory (e.g., 16GB).

Intended Use

This model is intended for generating concise, accurate Arabic summaries from input texts. It performs best when used with the specific prompt format seen during training.

Training Data

Training used a synthetic summarization dataset created as follows:

Source: Arabic subset of the GEM/xlsum dataset
Steps:
- Noise cleaning and Arabic text normalization
- Filtering for text length (300–2500 characters)
- Duplicate removal (SHA1 hashing)
- Language filtering (Arabic-dominant only)
- Topic stratification using TF-IDF + NMF (~5000 samples)
Synthetic summaries generated by "Qwen/Qwen2.5-14B-Instruct-AWQ"

Training Procedure

Key details:

Base model: unsloth/Qwen2.5-1.5B-Instruct
LoRA (PEFT) settings:
- r: 16, alpha: 16, dropout: 0.1
- Target modules: q_proj, v_proj, up_proj, down_proj
Quantization: 4-bit NF4 (bnb_4bit_compute_dtype=torch.bfloat16)
Optimizer: paged_adamw_32bit
Learning rate: 2e-4, cosine schedule, warmup 3%
Epochs: 2
Batch: 2 per device, 4 accumulation steps
Eval: Based on validation loss
Gradient clipping: 0.3
Checkpointing: Best eval loss model saved

Framework Versions

TRL: 0.18.0
Transformers: 4.52.3
PyTorch: 2.7.0
Datasets: 3.6.0
Tokenizers: 0.21.1

How to Use

python

from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
from peft import PeftModel
import torch

base_model = AutoModelForCausalLM.from_pretrained(
    "unsloth/Qwen2.5-1.5B-Instruct",
    torch_dtype=torch.float16,
    device_map="auto",
)

tokenizer = AutoTokenizer.from_pretrained("unsloth/Qwen2.5-1.5B-Instruct")
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"

model = PeftModel.from_pretrained(base_model, "ml-maverick/Qwen2.5-1.5B-Instruct-ArabicSum")
model = model.merge_and_unload()
model.eval()

instruction = (
    "أنت كاتب عربي محترف ذو خبرة واسعة في تلخيص النصوص بدقة وإيجاز."
    " عند استلام نص، اتبع الخطوات التالية لضمان تقديم ملخص فعّال:\n"
    "1. قم بتحليل المحتوى بعناية لتحديد الفكرة الرئيسية.\n"
    "2. استخرج المعلومات الجوهرية.\n"
    "3. صغ ملخصًا واضحًا وموجزًا لا يتجاوز ثلاث جمل.\n"
    "4. تجنب التفاصيل غير الموجودة، والتزم بالدقة.\n\n"
)
text = "أظهرت دراسة حديثة أن..."
input_prompt = f"{instruction}{text}\n\nالملخص:"
input_ids = tokenizer(input_prompt, return_tensors="pt").input_ids.to(model.device)

generation_config = GenerationConfig(
    max_new_tokens=200,
    num_beams=1,
    early_stopping=True,
    repetition_penalty=1.1,
    temperature=0.4,
    top_p=0.9,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
)

with torch.no_grad():
    output_ids = model.generate(
        input_ids=input_ids,
        generation_config=generation_config,
        attention_mask=input_ids.ne(tokenizer.pad_token_id),
    )

output = tokenizer.decode(output_ids[0], skip_special_tokens=True)
summary = output.split("الملخص:")[-1].strip()
print("Generated Summary:", summary)

Limitations and Bias

Synthetic bias from source LLM
Requires exact prompt format
Arabic only
Not guaranteed factual accuracy

Citation

bibtex

@misc{vonwerra2022trl,
  title = {{TRL: Transformer Reinforcement Learning}},
  author = {Leandro von Werra et al.},
  year = 2022,
  howpublished = {\url{https://github.com/huggingface/trl}}
}
@article{qwen2024qwen2,
  title={{Qwen2}: A Strong Large Language Model Family},
  author={Qwen Team},
  journal={arXiv preprint arXiv:2406.01175},
  year={2024}
}
@article{wolf2020transformers,
  title={Transformers: State-of-the-Art NLP},
  author={Wolf, Thomas et al.},
  journal={arXiv:1910.03771},
  year={2020}
}
@article{lhoest2021datasets,
  title={Datasets: A Community Library},
  author={Lhoest, Quentin et al.},
  journal={arXiv:2109.02844},
  year={2021}
}
@software{peft,
  title={{PEFT}: Parameter-Efficient Fine-Tuning},
  author={Hugging Face},
  year={2023},
  url={https://github.com/huggingface/peft}
}