Note: RTX 3090 lacks native FP8 support, so vLLM uses the Marlin kernel for weight-only FP8 decompression. GPUs with native FP8 (RTX 40xx, Ada Lovelace+) will achieve higher throughput.

Usage with vLLM

Requires vLLM 0.13.0+ and transformers 5.0+ for glm4_moe_lite architecture support.

python

from vllm import LLM, SamplingParams

llm = LLM(
    model="marksverdhei/GLM-4.7-Flash-fp8",
    tensor_parallel_size=2,
    max_model_len=8192,
    enforce_eager=True,  # Optional: disable CUDA graphs to save VRAM
)

outputs = llm.generate(["Hello, world!"], SamplingParams(max_tokens=100))
print(outputs[0].outputs[0].text)

License

MIT (same as base model)

marksverdhei/GLM-4.7-Flash-FP8

GLM-4.7-Flash FP8

Quantization Details

Performance

Usage with vLLM

License

marksverdhei/
GLM-4.7-Flash-FP8