Model Description
GLM-5-NVFP4 is an NVFP4-quantized version of
zai-org/GLM-5, a 744B-parameter Mixture-of-Experts language model with 40B active parameters, 256 experts per MoE layer (8 activated per token), and DeepSeek Sparse Attention (DSA).
Quantized directly from the full BF16 checkpoint (
zai-org/GLM-5),
not the FP8 release, to NVFP4 (4-bit with blockwise FP8 scales per 16 elements) using
NVIDIA Model Optimizer.
What's quantized
Only the MoE expert MLP layers (gate, up, and down projections) are quantized to NVFP4. Attention layers are left in BF16. Since the expert weights constitute the vast majority of model parameters in an MoE architecture, this still yields significant memory savings.
Calibration uses natural top-k routing rather than forcing all experts to activate, so each expert's quantization scales reflect the token distributions it actually sees during inference. To compensate, calibration was run on a much larger number of samples than typical to ensure broad expert coverage through natural routing alone.
Calibration dataset
Three calibration passes were run:
- Coding pass — Agentic coding samples (tool calling, multi-turn code generation, function calling) with English and Chinese system prompts.
- Broad pass — Large-scale diverse samples drawn from WildChat and LMSYS-Chat covering real user conversations across a wide range of topics and languages.
- Deep pass — Long-context samples (>8K tokens) from coding and diverse sources to exercise deep-sequence expert activation patterns.
Merged via element-wise max across all calibration runs.
How to Run
NVFP4 requires Blackwell GPUs (RTX 5090, RTX Pro 6000, B100, B200, etc.). Even quantized, this is a huge model — tested on 8x RTX Pro 6000 Blackwell (96 GB each, 768 GB total).
If you experience NCCL hangs with P2P, make sure you have iommu=pt (and amd_iommu=pt on AMD platforms) in your kernel command line.
SGLang
1export NCCL_IB_DISABLE=1
2export NCCL_P2P_LEVEL=PHB
3export NCCL_ALLOC_P2P_NET_LL_BUFFERS=1
4export NCCL_MIN_NCHANNELS=8
5export OMP_NUM_THREADS=8
6export SAFETENSORS_FAST_GPU=1
7
8python3 -m sglang.launch_server \
9 --model lukealonso/GLM-5-NVFP4 \
10 --served-model-name glm-5 \
11 --reasoning-parser glm45 \
12 --tool-call-parser glm47 \
13 --trust-remote-code \
14 --tp 8 \
15 --mem-fraction-static 0.95 \
16 --max-running-requests 8 \
17 --kv-cache-dtype fp8_e4m3 \
18 --quantization modelopt_fp4 \
19 --attention-backend flashinfer \
20 --moe-runner-backend flashinfer_cutlass \
21 --disable-custom-all-reduce \
22 --enable-flashinfer-allreduce-fusion \
23 --host 0.0.0.0 \
24 --port 8000
vLLM
Please contribute vLLM instructions if you successfully manage to run this model.
