Quantized with AutoRound v0.9.4 (bits=4, group_size=128, sym=True) in auto_gptq format.
Model size: ~124 GB (3.8x compression from the unquantized pruned model).
Compatible with vLLM, Transformers + AutoGPTQ, ExLlamaV2, etc.

Example vLLM serving command

(tested on 2x NVIDIA RTX PRO 6000 Blackwell with the following settings)

bash

vllm serve Jon-Nielsen/GLM-4.7-REAP-30-W4A16 \
  --tensor-parallel-size 2 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.96 \
  --max-model-len 196608 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 16384 \
  --trust-remote-code \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --enable-expert-parallel \
  --tool-call-parser=glm47 \
  --reasoning-parser=glm45 \
  --enable-auto-tool-choice \
  --disable-custom-all-reduce