FP8 (E4M3) quantisation of
swiss-ai/Apertus-v1.5-70B
with static fp8 KV-cache scales calibrated on German text and written into
the checkpoint.
Read this before choosing. The weights here are quantised data-free:
FP8_DYNAMIC derives per-channel weight scales from weight min/max alone, so
the German calibration reaches only the KV scales, never the weights.
Measured against the BF16 reference, this checkpoint is the stronger of the
pair on English and roughly six times weaker on German than the W4A16
sibling, at 71 GiB against 40 GiB.
For German-language work, take the
W4A16.
Take this one if English matters more, if you want higher weight fidelity, or
as the data-free control that makes the W4A16 result interpretable.
The vision and audio tokenizers are not quantised. Swiss AI note they are
precision-sensitive and stay in float32 on half-precision loads; the
WavTokenizer additionally uses weight-norm parametrised convolutions.
recipe.yaml in this repository records the exact recipe used.
Requirements
This checkpoint does not load with upstream transformers. Apertus 1.5 uses
Apertus1p5ForConditionalGeneration, whose config class ships only in Swiss
AI's transformers fork:
You also need a vLLM with Apertus 1.5 support — Swiss AI's fork, upstream PR
#50496, or the prebuilt
image onpremai/vllm-apertus-1p5. Verify both before downloading 71 GiB:
At 71 GiB this needs roughly 36 GiB per GPU at TP=2.
KV cache precision
--kv-cache-dtype autoresolves to fp8 because this checkpoint carries KV
scales, which roughly doubles the KV budget. Pass
--kv-cache-dtype bfloat16 explicitly for full-precision KV. One checkpoint
covers both cases.
Known serving issue
On some builds vLLM's xIELU activation falls back to forward_native under
torch.compile and then emits a log line dynamo cannot trace:
torch._dynamo.exc.Unsupported: logging.Logger method not supported
Workaround: --enforce-eager. This is a vLLM/xIELU interaction, not a property
of the checkpoint.
Evaluation
Everything below was measured. Perplexity and top-1 agreement on 64 held-out
samples × 2048 tokens per language from wikimedia/wikipedia
(20231101.de / 20231101.en), 131,008 scored positions each. Generations
greedy (temperature=0, top_p=1, seed=0). Measured on one NVIDIA H200 (SM90)
under vLLM with --enforce-eager. Eval samples use a different shuffle seed
from the calibration set (1234 vs 42).
Perplexity
config
German
Δ
English
Δ
BF16 reference
6.1042
—
3.8545
—
FP8, bf16 KV
6.3876
+4.64%
3.9712
+3.03%
FP8, fp8 KV
6.3867
+4.63%
4.0033
+3.86%
Top-1 agreement with BF16
Fraction of positions where the argmax token is unchanged. More sensitive than
perplexity to damage that shows up as malformed morphology, and independent of
the perplexity baseline.
config
German
English
FP8, bf16 KV
0.9168
0.9220
fp8 KV cache is close to free on German
Same weights, KV dtype the only difference:
German
English
Perplexity cost
−0.014%
+0.807%
Zero cost on German — noise-level, slightly negative — and 0.8% on English.
The KV scales are the only thing that differs between those two runs, and they
were computed from German activations. That asymmetry is the cleanest
single piece of evidence in this family that calibration language matters:
there is no other candidate explanation.
This is also the payoff from calibrating them offline. vLLM's runtime
--calculate-kv-scales, which calibrates against dummy tokens during startup
profiling, produced unusable output on the same architecture.
Generation probes
Two five-sentence German prompts, each requiring five specific terms, greedy.
Fluent, grammatical German with well-formed nominal compounds; zero non-Latin
token bleed.
Where this sits among the alternatives
All built from the same base model and the same German calibration corpus, all
measured against the same BF16 reference on the same eval:
The ranking is monotonic in scale granularity, not bit width. Group-128
INT4 beats per-channel FP8, which beats per-tensor FP8, even though the bit
widths run the other way. The last row also quantises activations statically,
which compounds the problem: a fixed per-tensor activation scale must
accommodate the largest outlier seen during calibration, crushing the
resolution available to everything else.
Calibration length matters too. The two GPTQ rows differ only in sequence
length — 4096 against 8192 — and that is worth 0.15 points of German
perplexity. It also accounts for the apparent gap between AutoRound and GPTQ:
at matched calibration depth the two are indistinguishable.
Limitations
Weights are quantised data-free.FP8_DYNAMIC derives weight scales from
min/max alone. The German calibration reaches only the KV scales. That is why
German degrades more than English here (152% of the English damage in absolute
nats) while the W4A16 builds show the reverse (8–11%).
Not benchmarked on a standard suite. No MMLU, HellaSwag or similar.
Perplexity, top-1 agreement and targeted German probes only.
Calibration corpus is general German, not domain text: 512 samples × 4096
tokens from German Wikipedia (wikimedia/wikipedia, 20231101.de), filtered
to samples that fill the full window. Domain-matched calibration would
plausibly do better on that domain and is untested here.
fp8 KV tested to ~2k context in the perplexity runs, and to ~6k on the
sibling checkpoint. Not validated at the model's full 262,144-token window. KV
quantisation error accumulates with depth; test at your own context lengths.
Single-GPU measurement. All numbers from one NVIDIA H200 (SM90). Not tested
under tensor parallelism, and not measured on Ada (SM89) despite that being the
intended serving target.
Reproducing this checkpoint
The quantisation script is in this repository as
quantize_apertus.py. Its APERTUS_* environment
variables have since been renamed to QUANT_* in the generalised
quantize_model.py
published alongside the W4A16 sibling; the old names still work and print a
deprecation note.
bash
1# isolated venv: the swiss-ai transformers fork is OLDER than current upstream2# releases and will replace whatever you have3python3 -m venv venv &&source venv/bin/activate
45pip install"transformers[torch,vision,audio] @ \
6 git+https://github.com/swiss-ai/transformers.git@3797303dda74844e3d1f8977ff5518bb91f818b4"7pip install llmcompressor datasets accelerate
89hf auth login # swiss-ai repos are gated1011# verify the environment first -- full code path, token calibration set, minutes12APERTUS_TESTING=1APERTUS_SIZE=8B APERTUS_SCHEME=fp8dyn APERTUS_KV_FP8=1\13 python quantize_apertus.py
1415# this checkpoint16APERTUS_SIZE=70B APERTUS_SCHEME=fp8dyn APERTUS_KV_FP8=1\17CUDA_VISIBLE_DEVICES=0 python quantize_apertus.py
APERTUS_SCHEME=fp8dyn is FP8_DYNAMIC — the data-free path that produced
this checkpoint. APERTUS_SCHEME=fp8 is a different thing: GPTQ with the FP8
preset, which uses per-tensor weight scales and static activations and measured
substantially worse (see the table above). APERTUS_SCHEME=w4a16 builds the
sibling.
The script writes to Apertus-v1.5-70B-FP8dyn-KV8-de; this repository renames
it, since -de would imply the weights were German-calibrated and they were
not.
Notes for this architecture
Quantised with llm-compressor
0.13.0. These cost real time to find:
Apertus1p5ForConditionalGeneration and Apertus1p5Model both lack their own
forward(), so llm-compressor can trace neither. Pass
model.model.language_model (Apertus1p5TextModel) to oneshot(), then save
the parent so the wrapper config, lm_head and encoders are written too.
Quantising the wrapper matches more Linears than intended: the audio
tokenizer contributes 25 (24 Vocos backbone + 1 ISTFT head). llm-compressor
matches module paths, and the path is audio_tokenizer even though the
class is WavTokenizerModel, so a regex on wavtokenizer silently matches
nothing. Targeting language_model avoids this structurally.
A submodule has no name_or_path, so llm-compressor cannot auto-load a
processor. Pass processor=tok explicitly.
save_pretrained must be patched on the parent via modify_save_pretrained(),
or the weights are written as fake-quantised BF16 — all of the quality loss,
none of the size saving. Check the output size, not just that it saved. The
patch uses functools.wraps, so __qualname__ still reports
PreTrainedModel.save_pretrained even when active; test the _overridden
attribute instead.
_retie_embeddings() compares input and output embeddings with
torch.equal() without aligning devices, and compression onloads modules to
the GPU before it runs. On an untied model it returns early anyway, so the
script patches it to skip.
llmcompressor and compressed-tensors must be a matched pair, and vLLM
pins compressed-tensors exactly. Installing vLLM into the same venv drags
the version back and breaks the import.
Inherits the licence and usage policy of
swiss-ai/Apertus-v1.5-70B.
The base model ships EU AI Act documentation and a public training-data summary;
consult the upstream repository for both.
Quantisation only — no fine-tuning, no additional training data, no
modification of model behaviour beyond numerical precision.
