llama-server -m GLM-4.7-Flash-Q8_0.gguf \
    -ngl 99 \
    -c 131072 \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --split-mode layer \
    --tensor-split 32,24 \
    --host 0.0.0.0 \
    --port 8080

OpenAI-compatible API

After starting the server, the API is available at:

http://localhost:8080/v1/chat/completions
http://localhost:8080/v1/completions

Quantization Notes

This model was quantized from ngxson/GLM-4.7-Flash-GGUF F16 version.

The missing deepseek2.rope.scaling.yarn_log_multiplier metadata key was added to enable quantization with llama.cpp.

Quality Comparison

Quantization	Size	Perplexity Impact
F16	56 GB	Baseline
Q8_0	30 GB	~0.1%
Q4_K_M	18 GB	~2-4%

Q8_0 provides near-lossless quality while reducing model size by ~47%.

khasinski/GLM-4.7-Flash-Q8_0-GGUF

GLM-4.7-Flash Q8_0 GGUF

Model Details

Hardware Requirements

Usage

Basic usage with llama.cpp

Full 128K context on dual GPU

OpenAI-compatible API

Quantization Notes

Quality Comparison

khasinski/
GLM-4.7-Flash-Q8_0-GGUF