Views
No views yet
| File | Quant | Size | BPW | Description |
|---|---|---|---|---|
| Qwen2.5-7B-Instruct-heretic-BF16.gguf | BF16 | 15.2 GB | 16.00 | Full precision, no quantization loss |
| Qwen2.5-7B-Instruct-heretic-Q8_0.gguf | Q8_0 | 8.1 GB | 8.50 | Near-lossless quantization |
| Qwen2.5-7B-Instruct-heretic-Q6_K.gguf | Q6_K | 6.3 GB | 6.56 | High quality, good balance |
| Qwen2.5-7B-Instruct-heretic-Q5_K_M.gguf | Q5_K_M | 5.4 GB | 5.71 | Recommended for most users |
| Qwen2.5-7B-Instruct-heretic-Q4_K_M.gguf | Q4_K_M | 4.7 GB | 4.91 | Good quality at small size |
attn.o_proj (1 per layer), mlp.down_proj (1 per layer)heretic/model.py to produce correct results:pad_token = eos_token fallback causes batched inference to produce garbage output on ROCm. Replace with a dedicated <|pad|> token and resize embeddings.attn_implementation="eager" in from_pretrained() to avoid SDPA backend issues on RDNA3.nan KL divergence and meaningless refusal counts on AMD GPUs.llama-cli -m Qwen2.5-7B-Instruct-heretic-Q5_K_M.gguf -p "You are a helpful assistant." --chat-template chatmlllama-server -m Qwen2.5-7B-Instruct-heretic-Q5_K_M.gguf -ngl 99 --chat-template chatml<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant