This is a mixed precision NVFP4 quantization of zai-org/GLM-4.7-Flash, a 30B-A3B (30B total, 3B active) Mixture-of-Experts model. This version preserves MTP (Multi-Token Prediction) layers in BF16 for speculative decoding compatibility.

What's Different from GLM-4.7-Flash-NVFP4?

Feature	GLM-4.7-Flash-NVFP4	This Model
MTP Layers	NVFP4	BF16
Calibration Samples	128	512
Calibration Seq Length	2048	4096
MMLU-Pro Accuracy	23.56%	23.91%

Quantization Strategy

This model uses mixed precision to preserve accuracy and MTP functionality:

Component	Precision	Rationale
MLP Experts	FP4 (E2M1)	64 routed experts, 4 active per token
Dense MLP	FP4 (E2M1)	First layer dense MLP
Attention (MLA)	BF16	Low-rank compressed Q/KV projections are sensitive
MTP Layers	BF16	`eh_proj`, `shared_head.head` for speculative decoding
Norms, Gates, Embeddings	BF16	Standard practice

Performance

Metric	BF16	NVFP4	This Model
MMLU-Pro	24.83%	23.56%	23.91%
Size	62.4 GB	20.4 GB	20.9 GB
Compression	1x	3.1x	3.0x
Accuracy Loss	-	-1.27%	-0.92%

MTP Acceptance Rate

Model	Acceptance Rate	Mean Accepted Length
BF16 (baseline)	60%	1.60
This Model	63%	1.63

MTP quality is preserved (actually slightly improved) after quantization.

MTP Performance Note

MTP speculative decoding currently shows overhead rather than speedup due to missing torch.compile support for the MTP drafter model in vLLM. For best throughput, run without MTP enabled until this is resolved upstream.

Configuration	Tokens/sec
Without MTP	78.1 tok/s
With MTP (1 token)	64.7 tok/s
With MTP (2 tokens)	56.8 tok/s
With MTP (4 tokens)	44.5 tok/s

Usage

Requirements

vLLM: 0.8.0+ (for compressed-tensors NVFP4 support)
transformers: 5.0.0+ (for glm4_moe_lite architecture)
GPU: NVIDIA GPU with FP4 tensor core support (Blackwell, Hopper, Ada Lovelace)

Installation

bash

pip install vllm>=0.8.0
pip install git+https://github.com/huggingface/transformers.git

Inference with vLLM (Recommended)

python

from vllm import LLM, SamplingParams

model = LLM(
    "GadflyII/GLM-4.7-Flash-MTP-NVFP4",
    tensor_parallel_size=1,
    max_model_len=4096,
    trust_remote_code=True,
    gpu_memory_utilization=0.90,
)

params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = model.generate(["Explain quantum computing in simple terms."], params)
print(outputs[0].outputs[0].text)

Serving with vLLM

bash

# Standard serving (recommended for performance)
VLLM_ATTENTION_BACKEND=TRITON_MLA vllm serve GadflyII/GLM-4.7-Flash-MTP-NVFP4 \
    --tensor-parallel-size 1 \
    --max-model-len 4096 \
    --trust-remote-code \
    --gpu-memory-utilization 0.90

# With MTP speculative decoding (experimental)
VLLM_ATTENTION_BACKEND=TRITON_MLA vllm serve GadflyII/GLM-4.7-Flash-MTP-NVFP4 \
    --tensor-parallel-size 1 \
    --max-model-len 4096 \
    --trust-remote-code \
    --gpu-memory-utilization 0.90 \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

Model Details

Base Model: zai-org/GLM-4.7-Flash
Architecture: Glm4MoeLiteForCausalLM
Parameters: 30B total, 3B active per token (30B-A3B)
MoE Configuration: 64 routed experts, 4 active, 1 shared expert
Layers: 47 (with 1 MTP layer)
Context Length: 202,752 tokens (max)
Languages: English, Chinese

Quantization Details

Format: compressed-tensors (NVFP4)
Block Size: 16
Scale Format: FP8 (E4M3)
Calibration: 512 samples from wikitext dataset
Calibration Sequence Length: 4096
Full Expert Calibration: All 64 experts calibrated per sample

Tensors by Precision

Precision	Count	Description
NVFP4	9,168	MLP/FFN weights
BF16	240	Attention weights (MLA)
BF16	2	MTP layers (eh_proj, shared_head.head)

Evaluation

MMLU-Pro Overall Results

Model	Accuracy	Correct	Total
BF16 (baseline)	24.83%	2988	12032
NVFP4-v1	23.56%	2835	12032
This Model	23.91%	2877	12032

MMLU-Pro by Category

Category	BF16	This Model	Difference
Social Sciences	32.70%	31.26%	-1.44%
Other	31.57%	29.85%	-1.72%
Humanities	23.78%	22.82%	-0.96%
STEM	19.94%	19.48%	-0.46%

MMLU-Pro by Subject

Subject	BF16	This Model	Difference
Biology	50.35%	48.12%	-2.23%
Psychology	44.99%	41.23%	-3.76%
History	33.60%	34.12%	+0.52%
Health	35.21%	34.11%	-1.10%
Economics	36.37%	33.06%	-3.31%
Philosophy	31.46%	29.26%	-2.20%
Other	28.35%	26.08%	-2.27%
Computer Science	26.10%	21.95%	-4.15%
Business	16.35%	19.26%	+2.91%
Law	16.89%	15.99%	-0.90%
Math	14.06%	14.73%	+0.67%
Physics	15.32%	15.24%	-0.08%
Engineering	16.00%	14.96%	-1.04%
Chemistry	14.13%	14.84%	+0.71%

Citation

If you use this model, please cite the original GLM-4.7-Flash:

bibtex

@misc{glm4flash2025,
  title={GLM-4.7-Flash},
  author={Zhipu AI},
  year={2025},
  howpublished={\url{https://huggingface.co/zai-org/GLM-4.7-Flash}}
}

License

This model inherits the Apache 2.0 license from the base model.