language:
license: apache-2.0
library_name: peft
pipeline_tag: text-generation
base_model: ALLaM-AI/ALLaM-7B-Instruct-preview
tags:
- allam
- arabic
- qlora
- lora
- peft
- unsloth
- transformers
- trl
- education
- question-answering
- deepbac
DeepBac ALLaM QLoRA Adapter
This repository contains a QLoRA adapter obtained by fine-tuning ALLaM-7B-Instruct-preview for the DeepBac project.
The adapter is designed to improve Arabic educational question answering and student support for the Algerian Baccalaureate.
Note: This repository contains only the QLoRA adapter weights. The original ALLaM model is not included.
Model Information
| Item | Value |
|---|
| Base Model | ALLaM-AI/ALLaM-7B-Instruct-preview |
| Fine-tuning Method | QLoRA |
| Framework | Unsloth |
| PEFT Method | LoRA |
| Task | Instruction Following / Question Answering |
| Language | Arabic |
| Domain | Education |
Training Configuration
The adapter was trained using the following configuration:
| Parameter | Value |
|---|
| Quantization | 4-bit |
| LoRA Rank (r) | 16 |
| LoRA Alpha | 16 |
| LoRA Dropout | 0 |
| Optimizer | AdamW 8-bit |
| Learning Rate | 2e-4 |
| Epochs | 5 |
| Batch Size | 2 |
| Gradient Accumulation | 4 |
| Maximum Sequence Length | 2048 |
Target modules:
- q_proj
- k_proj
- v_proj
- o_proj
- gate_proj
- up_proj
- down_proj
Intended Use
This adapter is intended for:
- Arabic educational assistants
- Educational question answering
- Student tutoring
- Personalized learning
- AI-assisted education
Loading the Adapter
1from transformers import AutoModelForCausalLM, AutoTokenizer
2from peft import PeftModel
3
4base_model = AutoModelForCausalLM.from_pretrained(
5 "ALLaM-AI/ALLaM-7B-Instruct-preview"
6)
7
8model = PeftModel.from_pretrained(
9 base_model,
10 "Belkacemdz/allam-deepbac-adapter"
11)
12
13tokenizer = AutoTokenizer.from_pretrained(
14 "ALLaM-AI/ALLaM-7B-Instruct-preview"
15)
Limitations
This adapter was fine-tuned specifically for educational applications.
Performance outside educational and Arabic question-answering tasks has not been evaluated.
License
This repository distributes only the QLoRA adapter.
The original ALLaM model remains subject to its own license and terms of use.
Author
Developed by Belkacem Achraf ZAGHEZ
Master's Student in Artificial Intelligence
University of Biskra, Algeria
