This repository provides a high-efficiency GGUF quantization of Alibaba's Qwen2.5-0.5B-Instruct. This 0.5B model punches significantly above its weight in coding and mathematics.

📊 Quantization Benchmarks

Our validation process for this Q4_K_M build yielded the following metrics:

Perplexity (PPL): 9.249 🙂
KL Divergence: 0.025 🤩
FLIP Score: 7.8% 🙂

⚙️ Implementation Details

Format: GGUF (Quantized to Q4_K_M)
Architecture: Transformer-based decoder with RoPE, SwiGLU, RMSNorm, Attention QKV bias, and Tied Word Embeddings.
Parameters: 494 Million (0.49B)
Context Window: Native 32,768 tokens (Supports generation up to 8,192 tokens).
Attention: Grouped Query Attention (GQA) with 14 Query heads and 2 KV heads.
Multilingual: Supports over 29+ languages (English, Chinese, French, Spanish, etc.).
Strengths: Logic, reasoning, and instruction following in a compact size.
Ideal Use Case: Edge devices, mobile applications, and high-speed basic automation.

📊 Hardware Requirements & Performance

Quantization	File Size	Recommended VRAM	Recommended Device
Q8_0	~531 MB	1.2 GB	Desktop / Server
Q5_K_M	~420 MB	0.9 GB	Standard Smartphones
Q4_K_M	~398 MB	0.8 GB	Low-end Mobile / IoT
IQ4_XS	~349 MB	0.7 GB	Ultra-constrained Edge

💬 Prompt Template

Qwen2.5 uses the standard ChatML format:

text

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

⚖️ Legal Notice

This model is a Derivative Work quantized from the original BF16 weights to GGUF format by Modelfix.com. It is released under the Apache 2.0 License, matching the original release by the Qwen Team at Alibaba Cloud.

modelfix/Qwen2.5-0.5B-Instruct

Qwen2.5-0.5B-Instruct-GGUF (Q4_K_M)

📊 Quantization Benchmarks

⚙️ Implementation Details

📊 Hardware Requirements & Performance

💬 Prompt Template

⚖️ Legal Notice

modelfix/
Qwen2.5-0.5B-Instruct