🦙 ALLaM-7B-Instruct-GGUF
This repository provides quantized GGUF versions of ALLaM-7B-Instruct , optimized for efficient inference using llama.cpp.
⚠️ Acknowledgment
The
original model was developed by
ALLaM-AI and is available here:
🔗
ALLaM-7B-Instruct-Preview This repository only provides quantized versions for improved performance on different hardware.
✨ Overview
ALLaM-7B-Instruct is an Arabic-centric instruction-tuned model based on Meta’s LLaMA architecture , designed for natural language understanding and generation in Arabic.
🚀 What’s New? ✅ GGUF Format – Optimized for llama.cppMultiple Quantization Levels – Balance between precision and efficiencyRun on CPUs & Low-Resource Devices – No need for high-end GPUs!
📂 Available Model Quantizations
Model Variant Precision Size Best For ALLaM-7B-Instruct-f16.ggufFP16 Large High-precision tasks ALLaM-7B-Instruct-Q8_0.gguf8-bit Medium Balanced quality & speed ALLaM-7B-Instruct-Q6_K.gguf6-bit Small Good trade-off ALLaM-7B-Instruct-Q5_0.gguf5-bit Small Alternative quantization ALLaM-7B-Instruct-Q5_K_M.gguf5-bit Smaller Fast inference ALLaM-7B-Instruct-Q4_0.gguf4-bit Very Small Legacy format ALLaM-7B-Instruct-Q4_K_M.gguf4-bit Very Small Low-memory devices ALLaM-7B-Instruct-Q2_K.gguf2-bit Smallest Extreme efficiency
📖 Installation & Setup
1️⃣ Install llama.cpp Clone and build llama.cpp:
1 git clone https://github.com/ggerganov/llama.cpp.git
2 cd llama.cpp
3 make
2️⃣ Download the Model Choose and download a .gguf file from this repository.
3️⃣ Run Inference Use llama.cpp to generate responses:
./main -m ALLaM-7B-Instruct-Q4_0.gguf -p "كيف أجهز كوب شاهي؟"Expected Output:
لتحضير كوب شاي، اغلي الماء، ضع الشاي في الكوب، واسكب الماء الساخن فوقه. اتركه لدقائق ثم استمتع بمذاقه!📊 Benchmarks & Performance
Quantization Format Model Size CPU (Tokens/sec) GPU (Tokens/sec) FP16 Large ~2 ~15 Q8_0 Medium ~4 ~30 Q6_K Smaller ~6 ~40 Q5_0 Small ~7 ~42 Q5_K_M Smaller ~8 ~45 Q4_0 Very Small ~9 ~48 Q4_K_M Very Small ~10 ~50 Q2_K Smallest ~12 ~55
Performance may vary based on hardware and configuration.
📜 License
This model follows the
ALLaM-AI license. Refer to their
Hugging Face repository for details.
❤️ Acknowledgments
ALLaM-AI for developing the original ALLaM-7B-Instruct model.llama.cpp by ggerganov for optimized inference.
⭐ Contributions & Feedback
If you find this quantized model useful, feel free to contribute, provide feedback, or share your results!
