# Download the GGUF file
# Then create a Modelfile:
cat > Modelfile << EOF
FROM ./qwen2.5-abliterated-v2-q4_K_M.gguf
TEMPLATE "{{{{ if .System }}}}<|im_start|>system
{{{{ .System }}}}<|im_end|>
{{{{ end }}}}{{{{ if .Prompt }}}}<|im_start|>user
{{{{ .Prompt }}}}<|im_end|>
{{{{ end }}}}<|im_start|>assistant
"
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|endoftext|>"
EOF

# Import into Ollama
ollama create qwen2.5-abliterated-v2-q4 -f Modelfile

With llama.cpp

bash

# Download the GGUF file and use with llama.cpp
./llama-cli -m qwen2.5-abliterated-v2-q4_K_M.gguf -p "Your prompt here"

Quantization Details

Original Size: 14.19 GB (FP16)
Quantized Size: 4.36 GB (Q4_K_M)
Reduction: ~69.3%
Quantization Method: llama.cpp Q4_K_M

Performance

This quantized model maintains high quality while using significantly less VRAM:

Full precision: ~8 GB VRAM
Q4_K_M: ~4.5 GB VRAM

Credits

Base model: huihui-ai/Qwen2.5-7B-Instruct-abliterated-v2
Quantization tool: llama.cpp

koorbmeh/qwen2.5-7b-instruct-abliterated-v2-q4_K_M

Qwen2.5-7B-Instruct-Abliterated-v2 Q4_K_M

Model Details

Usage

With Ollama

With llama.cpp

Quantization Details

Performance

Credits

koorbmeh/
qwen2.5-7b-instruct-abliterated-v2-q4_K_M