Dynamic FP8 quant of google/gemma-4-31B-it, produced with llm-compressor using the FP8_DYNAMIC scheme (per-tensor FP8 weights + dynamic per-token FP8 activations, no calibration data). Format is compressed-tensors (float-quantized), loadable natively by vLLM.

~33 GB on disk (vs ~62 GB bf16) -> fits a single 48 GB GPU. FP8 compute needs sm_89+ (Ada/Hopper): an L40S works, but an Ampere A6000 cannot run fp8 kernels.
Only the text decoder Linear layers are quantized. The vision tower and lm_head are kept in bf16 (small, precision-sensitive), so multimodal + output quality are preserved.
Self-quant so the provenance is trusted, and future serves skip the 62 GB pull + on-the-fly quant.

Serve with vLLM

bash

vllm serve aaronday3/gemma-4-31B-it-FP8-Dynamic \
  --max-model-len 8192 --gpu-memory-utilization 0.90 --host 0.0.0.0 --port 8000