This GGUF is the quantized version of the
heretic-abliterated model. Refusal mechanisms were reduced to
~37% using
heretic while preserving coding and reasoning quality.
1sudo docker run --rm -p 8080:8080 \
2 -v /root/models/:/models \
3 --gpus all \
4 --ulimit memlock=-1:-1 \
5 --env CUDA_VISIBLE_DEVICES=0 \
6 ghcr.io/ggml-org/llama.cpp:full-cuda --server \
7 -m /models/Agents-A1-4B-kimi-Preview-heretic-IQ4_NL.gguf \
8 --host 0.0.0.0 --port 8080 \
9 --n-gpu-layers 999 \
10 --ctx-size 131072 \
11 --batch-size 4096 \
12 --ubatch-size 256 \
13 --cache-type-k f16 \
14 --cache-type-v f16 \
15 --flash-attn on \
16 --cont-batching \
17 --mlock \
18 --temp 0.95 \
19 --top-k 40 \
20 --top-p 0.9 \
21 --min-p 0.0 \
22 -n -1 \
23 --no-mmap \
24 --parallel 1 --tools all \
25 --dry-multiplier 0.05 \
26 --jinja --dry-sequence-breaker none \
27 --repeat-penalty 1.1
1# Quick test
2./llama-cli -m Agents-A1-4B-kimi-Preview-heretic-IQ4_NL.gguf \
3 -p "Write a Python function to sort a list" -n 256 --temp 0.6 -ngl 999
This is a GGUF quantized version of
hotdogs/Agents-A1-4B-kimi-Preview-heretic, which is an abliterated fine-tune of
InternScience/Agents-A1-4B on coding agent traces.
