festr2/glm51-nvfp4-w4a16-fp8pbwo-l45-47-l51-62-20260517
GLM-5.1 mixed precision checkpoint for vLLM experiments.
- Base checkpoint:
lukealonso/GLM-5.1-NVFP4
- Selected MoE expert layers replaced with FP8 blockwise weight-only (
FP8_PB_WO)
tensors from zai-org/GLM-5.1-FP8.
- Remaining MoE expert layers stay NVFP4 and are intended to run with the B12X
NVFP4 backend.
- Selected FP8_PB_WO layers:
45-47,51-62
- Quantization mode in
config.json: modelopt / MIXED_PRECISION
- Expected runtime mode used in validation: W4A16 (
B12X_MOE_FORCE_A16=1)
Validation summary against FP8 reference logits:
| windows | KLD |
|---|
| 8 | 0.044499 |
| 42 | 0.054121 |
Runtime footprint observed in vLLM KLD runner:
| model memory / rank | KV cache tokens |
|---|
| 46.77 GiB | 654080 |
Canonical vLLM support commit:
61eac2779 Support ModelOpt mixed FP8_PB_WO MoE
