Weights & activations: FP8 E4M3, per-tensor static scaling (W8A8).
KV cache: not quantized (runs in model precision). FP8 KV-cache scales were calibrated but removed for compatibility with vLLM's Olmo loader, which does not support per-layer KV scales. The scales are preserved under appendix/ (see appendix/RESTORE_NOTE.md). For runtime FP8 KV cache, serve with vLLM's --kv-cache-dtype fp8.
Kept in BF16: lm_head; and, under the windowed attention recipe, the q/k/v/o projections of every full-attention layer and the windowed layer immediately preceding each full layer (32 layers total for Olmo3's 3:1 sliding:full pattern). All MLP layers are quantized to FP8.

This selective recipe protects the long-context global-attention path (the most sensitive component for reasoning) while quantizing the bulk of the parameters.

Calibration

Data: chankhavu/smolmo-olmo3-calib-4k, all ~4000 samples.
Sequence length: 24576 tokens per sequence (truncated to natural length, no padding).
Algorithm: max (per-tensor absmax) for FP8 scales.
Procedure: data-parallel calibration across 4 GPUs (one full replica per GPU, each over a disjoint shard of the data); per-tensor amax is merged across ranks by elementwise max — equivalent to single-process calibration over the full set.

Usage

Loads natively in vLLM and SGLang:

python

from vllm import LLM
llm = LLM(model="chankhavu/smolmo-32b-fp8-step12750")

FP8 compute requires an FP8-capable GPU (Hopper/Ada/Blackwell).

Notes

Exported as a unified Hugging Face checkpoint; quantization details are recorded in hf_quant_config.json. This is PTQ only (no quantization-aware training).