Views
No views yet
imatrix Calibration Q4_K_M Quantization.llama.cpp CUDA backend.Temperature=0.6, TopP=0.95):| Evaluation Metric | Base Model (Qwen3-0.6B) | Abliterated Safetensors (FP16) | This Model (Q4_K_M GGUF) |
|---|---|---|---|
| SorryBench Refusal Rate | 20.86% (97/465) | 0.00% (0/465) | 0.00% (0/465) |
| GSM8K Accuracy (CoT Reasoning) | 66.00% (66/100) | 62.00% (62/100) | 57.00% (57/100) |
| GSM8K Accuracy (Direct Answer) | - | - | 43.00% (43/100) |
| KL Divergence ($D_{\text{KL}}$) | 0.00 | 0.0999 (< 0.2 threshold) | 0.0999 (< 0.2 threshold) |
| Model Disk Size | 1.20 GB | 1.20 GB | 378.33 MB (-70%) |
| C++ Inference Speed | ~80 t/s | ~80 t/s | 490+ t/s (6x boost) |
mlabonne/harmless_alpaca) to prevent vector norm cancellation in topic-matched scenarios (Petrov, 2026).o_proj and down_proj weight matrices.imatrix Protection Quantization: Generated importance calibration matrix (imatrix.dat) over 200 diverse samples prior to 4-bit Q4_K_M quantization to protect sensitive orthogonal cut channels.llama.cpp1llama-cli \
2 -m Qwen3-0.6B-Uncensored-Q4_K_M.gguf \
3 -cnv \
4 -c 2048 \
5 --temp 0.6 \
6 --top-p 0.95 \
7 -ngl 991llama-server \
2 -m Qwen3-0.6B-Uncensored-Q4_K_M.gguf \
3 --port 8089 \
4 -c 2048 \
5 -ngl 99