MXFP8 quantization of NousResearch/Hermes-4-70B with an FP8 KV cache. Weights identical to Hermes-4-70B-MXFP8; prompt format, chat template, tool calling, and license are the base model's.

Requires Blackwell (compute capability 12.0). MXFP8 needs Blackwell tensor-core support in vLLM; does not run on Ada/Hopper.

Weights: MXFP8, compressed-tensors format (llm-compressor 0.11.1)
KV cache: FP8, calibrated k_scale/v_scale baked in — vLLM uses it automatically, no --kv-cache-dtype flag
Not quantized: lm_head, embeddings
Size: 74.8 GB, 11 shards (base BF16 ≈ 141 GB)
Context on one 96 GB card: --max-model-len 118288 measured no-OOM (BF16-KV build: ~49K). Arch max 131072
Calibration: 2048 samples, neuralmagic/calibration (LLM), max seq 2048

MMLU

0-shot log-likelihood, lm-evaluation-harness on vLLM, measured on the identical-weights BF16-KV build. Not comparable to chat/thinking-mode numbers.

	acc
Overall	78.91% ± 0.33
Social sciences	88.20%
Other	82.56%
Humanities	75.39%
STEM	71.52%

Serving

bash

vllm serve amanwalksdownthestreet/Hermes-4-70B-MXFP8-KV8 \
    --quantization compressed-tensors \
    --max-model-len 118288 --gpu-memory-utilization 0.95 --max-num-seqs 2

Quantization only. Credit: NousResearch.