festr2/glm51-nvfp4-w4a16-fp8pbwo-l42-62-20260517
GLM-5.1 mixed precision checkpoint for vLLM experiments.
- Base checkpoint:
lukealonso/GLM-5.1-NVFP4
- Selected MoE expert layers replaced with FP8 blockwise weight-only (
FP8_PB_WO)
tensors from zai-org/GLM-5.1-FP8.
- Remaining MoE expert layers stay NVFP4 and are intended to run with the B12X
NVFP4 backend.
- Selected FP8_PB_WO layers:
42-62
- Quantization mode in
config.json: modelopt / MIXED_PRECISION
- Expected runtime mode used in validation: W4A16 (
B12X_MOE_FORCE_A16=1)
Validation summary against FP8 reference logits:
| windows | KLD |
|---|
| 8 | 0.040911 |
| 42 | 0.049504 |
Runtime footprint observed in vLLM KLD runner:
| model memory / rank | KV cache tokens |
|---|
| 49.73 GiB | 602432 |
Canonical vLLM support commit:
61eac2779 Support ModelOpt mixed FP8_PB_WO MoE
