Format: NVFP4 (NVIDIA FP4, W4A4, weights and activations both 4-bit)
Source: pyloxsystems/hochelaga-8b (BF16)
Calibration: 512-sample QFrCoLA dev set + 1,024-sample Wikipedia FR-QC sample
Compression: ~4× smaller than BF16 (~6 GB total vs ~16 GB)
Hardware target: NVIDIA Blackwell (sm_120 / sm_121), Hopper (sm_90), or any NVFP4-capable GPU

Measured inference throughput (RTX 5090 sm_120, verified 2026-05-20)

Configuration	Concurrency	Aggregate tok/s	Per-stream tok/s
NVFP4 (CUDA Graphs ON, single-stream)	1	212.3	211.9 ⭐
NVFP4 baseline	100	6,551.9	74.8
NVFP4 baseline	400	8,760.0	25.1
NVFP4 baseline	800	8,907.9	12.3
NVFP4 baseline (peak aggregate)	1,600	8,919.4 ⭐	10.5
NVFP4 baseline (saturated)	2,400	8,880.0	10.5

Software stack: vLLM 0.19.1 + torch 2.7.0+cu128 + VLLM_NVFP4_GEMM_BACKEND=marlin. Raw JSON results bundled in the model card provenance archive.

Backend note: This NVFP4 model currently runs via the Marlin INT4-with-FP4-dequant kernel on sm_120 because vLLM's FlashInfer-CUTLASS and TRT-LLM NVFP4 kernels do not compile for major version 12 in any current stable vLLM release (verified across vLLM 0.10.0 / 0.19.1 / 0.21.0 / nightly). When FP4-native sm_120 kernels ship in stable vLLM, aggregate throughput is expected to improve substantially above the current Marlin peak. (Higher single-run figures measured in May 2026 on vLLM-1.0-rc FP4-native kernels are not reproducible on the current public stack and are not claimed here.)

To use with EAGLE-3 speculative decoding, pair with pyloxsystems/hochelaga-8b-eagle3-drafter. Note: under the current Marlin backend, EAGLE-3 K=1 at single-stream (197 tok/s) does not exceed the NVFP4 baseline (212 tok/s) because the Marlin kernel absorbs the drafter latency win. EAGLE-3 still helps at high concurrency in the saturation analysis, but is not the recommended single-user-latency configuration on this backend.

Usage with vLLM

bash

pip install vllm

vllm serve pyloxsystems/hochelaga-8b-nvfp4 \
  --quantization nvfp4 \
  --max-model-len 8192 \
  --tensor-parallel-size 1

For speculative decoding:

bash

vllm serve pyloxsystems/hochelaga-8b-nvfp4 \
  --quantization nvfp4 \
  --speculative-config '{"model": "pyloxsystems/hochelaga-8b-eagle3-drafter", "num_speculative_tokens": 1, "method": "eagle3"}'

Quality impact

NVFP4 quantization introduces a small accuracy delta vs the BF16 reference. See the base model card for QFrCoLA evaluation methodology.

License

Apache 2.0. Inherits all licensing from the base model.

Built in Montréal · Pylox Systems Inc. · 2026