The other quantized 4B GGUFs available lost their chat template somewhere in conversion. Without it llama.cpp falls back to ChatML, whose <|im_start|> and <|im_end|> are not tokens in this vocabulary, so nothing ever ends a turn - the model writes both sides of the conversation until it hits your token limit.

Files

	Size
`apertus-v1.1-4b-instruct-q8_0.gguf`	4.1 GB	basically lossless, start here
`apertus-v1.1-4b-instruct-q4_k_m.gguf`	2.4 GB	smaller and quicker, a bit worse

Running it

llama-server -m apertus-v1.1-4b-instruct-q8_0.gguf --ctx-size 4096

4096 is what it was trained on.

Made with

docker run --rm -v ./source:/src:ro -v ./out:/models \
    ghcr.io/ggml-org/llama.cpp:full \
    --convert /src --outfile /models/f16.gguf --outtype f16

docker run --rm -v ./out:/models ghcr.io/ggml-org/llama.cpp:full \
    --quantize /models/f16.gguf /models/apertus-v1.1-4b-instruct-q8_0.gguf Q8_0

llama.cpp build FILL IN.

jondale/Apertus-v1.1-4B-Instruct-GGUF

Apertus v1.1 4B Instruct - GGUF

Why another one

Files

Running it

Made with

jondale/
Apertus-v1.1-4B-Instruct-GGUF