GGUF format models for Qwen 2.5 7B with safety abliterations. Ready for local deployment with llama-cpp-python or compatible runtimes.
Models
qwen2.5-7b-abliterated.gguf - Full precision model
qwen2.5-7b-Q4_K_M-abliterated.gguf - Q4_K_M quantized model (4-bit quantization, recommended for most use cases)
Quick Start
python
1from llama_cpp import Llama
23# Load the quantized model (recommended)4llm = Llama(5 model_path="qwen2.5-7b-Q4_K_M-abliterated.gguf",6 n_ctx=2048,7 n_threads=48)910# Generate text11output = llm(12"Q: What is 2+2? A:",13 max_tokens=50,14 temperature=0.5,15 top_p=0.9516)17print(output['choices'][0]['text'])
Model Details
Base Model: Qwen 2.5 7B
Format: GGUF (llama-cpp-python compatible)
Modifications: Safety abliterations removed
Vocabulary Size: 152,064
Context Window: 32,768 tokens
Recommended Quantization: Q4_K_M (best balance of quality and performance)
Test Suite
Automated test suite for testing Qwen 2.5 7B GGUF models with llama-cpp-python.
Test Coverage
Each model is tested for:
Model Loading
File existence and size validation
Successful loading into memory
Model attributes and metadata
Model Metadata
Context size validation
Vocabulary size verification
Model type detection
Text Generation
Basic text completion
Token limit enforcement
Stop sequence handling
Mathematical reasoning
Multiple completions generation
Code generation (Q4 model only)
Performance
Generation speed measurement
Memory usage validation
Quantization-specific performance metrics
Robustness
Empty prompt handling
Long prompt processing
Special characters support
Unicode character support
Repeated call stability
Setup
The test suite uses the .venv virtual environment with the following packages:
llama-cpp-python - Python bindings for llama.cpp
pytest - Testing framework
Running Tests
Option 1: Using the test script (Recommended)
bash
1# Run all tests2./scripts/run_tests.sh
34# Run tests for full model only5./scripts/run_tests.sh full
67# Run tests for Q4 model only8./scripts/run_tests.sh q4
Option 2: Using pytest directly
bash
1# Activate virtual environment2source .venv/bin/activate
34# Run all tests5pytest tests/ -v
67# Run specific test file8pytest tests/test_full_model.py -v
9pytest tests/test_q4_model.py -v
1011# Run specific test class12pytest tests/test_full_model.py::TestModelLoading -v
1314# Run with output visible15pytest tests/ -v -s
1617# Run with coverage (if pytest-cov is installed)18pytest --cov=. tests/
tests/test_full_model.py - Tests for qwen2.5-7b-abliterated.gguf
tests/test_q4_model.py - Tests for qwen2.5-7b-Q4_K_M-abliterated.gguf
tests/conftest.py - Shared fixtures and utilities
tests/pytest.ini - Pytest configuration
scripts/run_tests.sh - Test runner script
Test Structure
Each test file contains:
TestModelLoading - Validates model file and loading
TestModelMetadata - Checks model configuration and metadata
TestTextGeneration - Tests various generation scenarios
TestModelPerformance - Measures and validates performance
TestModelRobustness - Tests edge cases and error handling
TestQuantizationComparison - Q4-specific tests (Q4 model only)
Directory Structure
├── README.md # This file (model card)
├── qwen2.5-7b-abliterated.gguf # Full precision model
├── qwen2.5-7b-Q4_K_M-abliterated.gguf # Q4_K_M quantized model
├── config.json # Model configuration
├── generation_config.json # Generation parameters
├── tests/ # Test suite
│ ├── __init__.py
│ ├── conftest.py
│ ├── pytest.ini
│ ├── test_full_model.py
│ └── test_q4_model.py
├── scripts/ # Utility scripts
│ └── run_tests.sh
└── .gitignore # Git ignore rules
Configuration
Test parameters can be adjusted in tests/conftest.py:
python
1"max_tokens":50,# Maximum tokens to generate2"temperature":0.5,# Sampling temperature3"top_p":0.95,# Top-p sampling4"n_threads":4,# CPU threads to use5"n_ctx":512,# Context window size
Expected Results
All tests should pass if:
Model files exist and are valid GGUF format
Sufficient system memory is available
llama-cpp-python is properly installed
Performance Metrics
Tests will output performance metrics:
Generation time (seconds)
Tokens per second
This allows comparison between full precision and quantized models.
Troubleshooting
Model not found
Ensure GGUF files are in the root directory of the repository.
Out of memory
Reduce n_ctx in tests/conftest.py or use the Q4_K_M quantized model.
Slow performance
Increase n_threads in tests/conftest.py to use more CPU cores.
Notes
Tests use module-scoped fixtures to load models only once
First test run will be slower due to model loading
Q4_K_M model should be faster and use less memory than full precision
Temperature is set low (0.1-0.5) for more deterministic test results
