Full-precision FP8 quantization of swiss-ai/Apertus-v1.5-70B
using the FP8_DYNAMIC scheme on all linear projections (excluding lm_head,
embeddings, and vision/audio tokenizers which remain in BF16).
Checkpoint
Size
Compression
Quality (MMLU)
BF16 (original)
135 GiB
1.0x
baseline
FP8 (this repo)
71 GiB
1.9x
no measurable loss
NVFP4 (onprem-ai/Apertus-v1.5-70B-NVFP4)
48 GiB
2.8x
-0.8 points
Use this checkpoint when maximum quality is critical and GPU memory allows --
71 GiB leaves approximately 14 GiB for KV cache, enough for roughly 60k
context with fp8 KV cache on a 96 GiB card.
Measured on NVIDIA RTX PRO 6000 Blackwell (96 GiB VRAM) using
llmapibenchmark.
Metric
Value
Decode throughput
~20 TPS (single user)
Prefill throughput
~2500 TPS
Note: The NVFP4 checkpoint achieves ~50% higher decode throughput (31 TPS)
due to 33% smaller weights reducing memory bandwidth pressure. For
throughput-constrained deployments, prefer the NVFP4 checkpoint.
Quality
Benchmarked via lm-evaluation-harness (MMLU, 5-shot, 100 samples) on
NVIDIA RTX PRO 6000 Blackwell. FP8_DYNAMIC produces no statistically
significant quality loss -- a well-established result across dozens of
open models.
Metric
BF16 (official)
FP8
Delta
MMLU overall
baseline
0.7205
indistinguishable
humanities
baseline
0.7646
indistinguishable
social sciences
baseline
0.8158
indistinguishable
STEM
baseline
0.6258
indistinguishable
other
baseline
0.7269
indistinguishable
Multimodal Support
All vision and audio capabilities from the original checkpoint are preserved.
The quantization respects the ignore list: vision tokenizer, audio tokenizer,
and embedding layers remain unquantized.
Quantization Process
Single-pass calibration-free quantization using llm-compressor 0.12.0:
scheme: FP8_DYNAMIC
targets: Linear
ignore: lm_head, model.language_model.embed_tokens, model.embed_tokens,
re:.*vision_tokenizer.*, re:.*audio_tokenizer.*
Duration: ~4 minutes on RTX PRO 6000 Blackwell.
Limitations
vLLM only. SGLang does not support compressed-tensors checkpoints.
Requires vLLM with Apertus1p5 arch support. Use the ghcr.io/swiss-ai/vllm_apertus_1.5_release
image or build from upstream PR #50496.
Requires 96 GiB GPU for full multimodal. For text-only inference with
short context, the 71 GiB weights may fit on 80 GiB cards when multimodal
is disabled.
Supported Hardware
Hardware
Compatible
Notes
NVIDIA RTX PRO 6000 Blackwell (GB202, SM100)
Yes
Primary target
B100/B200/B300/GB200 (SM100)
Yes
H100/H200 (SM90)
Yes
DGX Spark (GB10, SM121)
Yes
FP8_DYNAMIC kernels work correctly on SM121. For the NVFP4 checkpoint, see the DGX Spark note in that repo.
