GGUF conversions of swiss-ai/Apertus-v1.5-70B,
made because there were no GGUF quants of v1.5 anywhere at the time of upload — the only
Apertus GGUFs on the Hub are for the older 2509 (v1.0) release, and people who just want to
try the model were stuck setting up the SwissAI vLLM fork.
⚠️ Read this before downloading
These are TEST quants. They were made to check whether the model answers at all, not
as a polished release. No imatrix, no perplexity measurements, no benchmark runs. See
What was actually tested.
This is text-only. The vision and audio towers of v1.5 were stripped, not converted.
You cannot feed images or audio to these files. See How this was made.
Two launch flags are mandatory — --override-kv tokenizer.ggml.eos_token_id=int:68
(or the model never stops generating) and -sp (or reasoning appears to be missing).
Neither is a quirk of these quants; both follow from the model itself. See
Required flags for copy-paste commands.
If someone produces proper imatrix quants of this model, prefer theirs over mine.
Files
File
Size
Notes
apertus-70b-Q4_K_M.gguf
43.7 GB (40.7 GiB)
The one that was actually run and tested
apertus-70b-q8_0.gguf
75.0 GB (69.9 GiB)
Converted directly from bf16; see testing notes
Both were produced from the original bf16 weights, Q4_K_M by quantizing the q8_0 file.
How this was made
Apertus v1.5 is Apertus1p5ForConditionalGeneration — a multimodal model: a text decoder plus a
VQ-VAE image tokenizer and a WavTokenizer audio tokenizer. llama.cpp does not support that
architecture.
What it does support is LLM_ARCH_APERTUS, added for Apertus 1.0. And the v1.5 decoder is
architecturally the same as 1.0 — non-gated MLP with xIELU, mandatory q_norm/k_norm. So instead
of teaching llama.cpp a new architecture, the model was reshaped into something the existing
converter already understands:
Both towers dropped — every tensor under model.vision_tokenizer. and
model.audio_tokenizer. is excluded from the weight map. This is what makes the result
text-only, and it is irreversible from these files.
Vocabulary truncated from vocab_size 266752 down to output_vocab_size 131072. The tail
of the embedding matrix holds image and audio codebook entries; the text model never emits
them, and lm_head was already 131072 rows in the original. Multimodal entries were also
removed from added_tokens in the tokenizer.
Decoder prefix rewritten from model.language_model. to model., which is where the
Apertus 1.0 converter expects it.
The result is an ordinary apertus-architecture GGUF. llama.cpp was not patched by a single
byte — the conversion class is registered at runtime and then stock convert_hf_to_gguf.main()
does the work.
Both scripts are included in this repo:
prep_apertus_text.py — builds a text-only HF folder. Weights are symlinked, not copied,
so it costs ~20 MB of disk and a few seconds rather than a second copy of a 70B model.
convert_apertus15.py — registers Apertus1p5ForConditionalGeneration and calls the stock
converter.
No special build is needed. These load in stock upstream llama.cpp — the GGUF declares
general.architecture = apertus, which has been supported since
PR #15852, released in b6671
(2 Oct 2025). Anything newer works; these files were run on a b10069-era build.
There is no vLLM path for these files — vLLM does not read GGUF for this architecture. If you
want v1.5 in vLLM, you need the SwissAI fork and the original safetensors.
Required flags — do not skip these
Two flags are mandatory. Launch without them and the model looks broken when it is not:
Flag
Without it
--override-kv tokenizer.ggml.eos_token_id=int:68
The model never stops. It runs past the end of its reply, answers its own question, starts a new turn, and only halts when it hits your token limit.
-sp (--special)
Reasoning appears to be missing. The model still thinks correctly, but the server strips the markers out of content, so you never see the block — and your frontend cannot split thinking from the answer.
Why the EOS one is needed: the original config declares three EOS ids,
eos_token_id: [2, 68, 72]. GGUF only stores one, and the converter writes 2 (</s>) — but the
token that actually ends an assistant turn is 68 = <|assistant_end|>. So the correct stop
token is present in the file, it is simply not the one marked as EOS. (72 is <|tools_suffix|>,
for tool-calling mode.)
-sp is available on both llama-cli and llama-server.
1llama-cli \2 -m apertus-70b-Q4_K_M.gguf \3 -sp \4 --override-kv tokenizer.ggml.eos_token_id=int:68 \5 -ngl 999 -c 4096\6 -p "<|system_start|>You are a helpful assistant.<|system_end|><|user_start|>What is the capital of Switzerland?<|user_end|><|assistant_start|>"
That prompt is the literal format the model expects — <|system_start|> … <|system_end|>,
<|user_start|> … <|user_end|>, then <|assistant_start|> to hand it the turn. It is exactly
the call used to verify these files, and it answers The capital of Switzerland is Bern.
To turn on reasoning, add the developer block after the system part — see
Deliberation mode below.
Deliberation (reasoning) mode
Apertus v1.5 reasons between its own dedicated tokens, not <think>:
id
token
meaning
32
<|inner_prefix|>
start of reasoning
33
<|inner_suffix|>
end of reasoning
68
<|assistant_end|>
end of the whole reply
Reasoning is enabled through a developer-role block, not a flag:
Quantization did not damage this. On a raw-token check of Q4_K_M, the model emitted token
32 at position 0, token 33 at position 131, and token 68 at position 262 — reasoning opened,
closed, and the turn ended, exactly where they belong.
But two things get in the way of seeing it:
llama-server strips control tokens from content. Add -sp to get them through.
llama.cpp's reasoning parser knows <think> and <|channel|>, not the Apertus markers, so
/v1/chat/completions will not populate reasoning_content. Parse <|inner_prefix|> /
<|inner_suffix|> on your side — or let your frontend do it (see below).
SillyTavern
Ready-made master-import presets are in this repo — import via Settings → Master Import:
File
System prompt
sillytavern/Apertus-v1.5-EN.json
English
sillytavern/Apertus-v1.5-RU.json
Russian
The token layout is identical in both — only the system prompt language differs, so pick by the
language you want the model instructed in, not by the language you intend to chat in (it follows
the user either way). The system prompts are deliberately plain and general-purpose: swap in your
own, the point of the presets is the plumbing below. No sampler settings are included, because
none were tuned for this model — use your own.
If you would rather set it up by hand, this is the same working configuration:
Instruct Template
System Sequence <|system_start|>
System Suffix <|system_end|>
Input Sequence <|user_start|>
Input Suffix <|user_end|>
Output Sequence <|assistant_start|>
Output Suffix <|assistant_end|>
Stop Sequence <|assistant_end|>
Story String Prefix <|system_start|>
Story String Suffix <|system_end|><|developer_start|>Deliberation: enabled
Tool Capabilities: disabled<|developer_end|>
Note where the deliberation block goes: on the Story String suffix, not the System suffix.
Put it on System Suffix and it gets repeated after every system message instead of being stated
once. Also enable System same as user, Sequences as stop strings, and set Names behavior
to force; leave wrap off.
Reasoning settings — SillyTavern will then fold the thinking into a collapsible block by
itself, no manual parsing:
Honest scope, on 2×RTX 3090 + Tesla V100 (64 GB VRAM total):
Q4_K_M
Loads and generates coherent multi-turn text (tested in Russian), ~17 tok/s generation,
~81 tok/s prompt at 65536 context with q8_0 KV cache across three GPUs.
Deliberation markers verified at the raw token-id level, as described above.
Stops correctly on <|assistant_end|> once the EOS override is applied.
q8_0
Read end-to-end during quantization to Q4_K_M, so the file is structurally sound and every
tensor is readable.
Load-tested on stock upstream llama.cpp b8861, CPU-only (-ngl 0), with no patches of any
kind — loads and answers correctly ("What is the capital of Switzerland?" → "The capital of
Switzerland is Bern."). This is the run that confirms no custom build is needed.
Not tested by anyone yet: long-context behaviour anywhere near the 262144 the GGUF advertises,
tool calling, and any quality benchmark whatsoever. Quality relative to the original bf16 weights
is unmeasured — no perplexity, no KL divergence. Treat both files as "it answers, and the
answers look sane", nothing stronger.
Credits
Model by SwissAI (ETH Zurich / EPFL / CSCS), Apache 2.0.
All the interesting work is theirs; this repo is just a format conversion.
