Qwen2.5-1.5B-Instruct-Extreme-Compression (789 MB GGUF)
This is an extremely optimized, lightweight version of Qwen2.5-1.5B-Instruct, compressed down to just 789 MB in GGUF format (INT4 quantization).
To achieve this ultra-compact size, advanced optimization techniques were applied, including heavy pruning of non-essential language components and embedding reduction. The model's primary focus has been narrowed down to English language proficiency and Python programming/scripting tasks, while completely preserving its conversational, reasoning, and chain-of-thought ("thinking") capabilities.
🚀 Key Features
- Ultra-Lightweight: Compressed to only 789 MB—ideal for mobile devices, low-spec hardware, and edge computing.
- Specialized Focus: Retains high performance in English text generation and Python coding/scripting tasks.
- Intact Reasoning: Conversational logic, structural chat formats, and reasoning capabilities were left untouched during the pruning process.
- Format: GGUF (4-bit quantization), fully ready for local deployment.
🛠️ How to Use
1. LM Studio / AnythingLLM / KoboldCPP
- Download the
.gguf file from this repository.
- Load the file directly into LM Studio or your preferred GGUF-compatible local GUI.
- Set the system prompt and use the standard Qwen template.
2. Python via llama-cpp-python
from llama_cpp import Llama
1. Initialize the model (Make sure 'model_q4_k_m.gguf' is in the same directory)
llm = Llama(
model_path="./model_q4_k_m.gguf",
n_ctx=2048, # Context window size
n_threads=4, # Number of CPU threads (adjust based on your CPU)
n_gpu_layers=0 # Increase if you want to offload layers to GPU
)
2. Prepare the prompt using the standard ChatML / Qwen template
system_prompt = "You are a helpful AI assistant specialized in English and Python coding."
user_prompt = "Write a clean Python script to scrape data from a website using requests and BeautifulSoup."
full_prompt = (
f"<|im_start|>system\n{system_prompt}<|im_end|>\n"
f"<|im_start|>user\n{user_prompt}<|im_end|>\n"
f"<|im_start|>assistant\n"
)
3. Run generation
print("\n--- Generating Response ---")
output = llm(
prompt=full_prompt,
max_tokens=1024,
temperature=0.7,
stop=["<|im_end|>", "<|im_start|>"], # Crucial stop tokens for GGUF format
echo=False
)
4. Print the clean output
print(output["choices"][0]["text"].strip())
