This model has been superseded by the BaseQuant_XL variant, which keeps routing-critical layers (MoE router gate, shared expert, lm_head) in full bf16 precision for improved quality. Benchmark comparisons are available in the XL model card.
leonsarmiento/GLM-4.7-Flash-6bit-mlx
This model was converted to MLX format from
zai-org/GLM-4.7-Flash using
mixed 8/6-bit quantization optimized for Apple Silicon.
GLM-4.7-Flash is a 31B-parameter text-only MoE (Mixture of Experts) model with 64 routed experts (4 active per token + 1 shared expert), MLA-style attention with LoRA-rank Q/KV compression, and speculative decoding support (MTP). Despite 31B total parameters, only ~3B are activated per token for efficient inference.
Mixed Quantization Strategy
This model uses layer-aware mixed-bit quantization that allocates higher precision to sensitive layers and lower precision to bulk parameters, maximizing quality per gigabyte.
| Bit Depth | Layers | Rationale |
|---|
| 8-bit | embed_tokens, lm_head, router gate + e_score_correction_bias, shared_experts, self_attn (MLA), dense mlp (layer 0), layernorms | All critical layers preserved at full 8-bit — embeddings, routing, shared representation, attention, and the dense MLP |
| 6-bit | switch_mlp (routed experts) | Bulk of parameters, only 4 of 64 experts active per token (6.25%) — natural redundancy tolerates lower precision |
Why this matters for MoE
In Mixture of Experts models, the router gate determines which experts handle each token. A poorly quantized router sends tokens to the wrong experts, cascading errors through the entire forward pass. The shared expert processes all tokens regardless of routing, making it equally critical.
By preserving the router, shared expert, and all attention layers at 8-bit, while quantizing the 64 routed experts to 6-bit, we maintain routing accuracy, shared representation quality, and full attention fidelity while achieving significant compression. The 6-bit variant provides the highest fidelity for the routed experts among the compressed variants, at the cost of additional size compared to 4-bit and 5-bit versions.
Quantization Details
| Metric | Value |
|---|
| Quantization type | Mixed 8/6-bit |
| Average | 6.646 bits per weight |
| Group size | 64 |
| Method | mlx_lm with custom quant_predicate |
| Total output size | ~23 GB (from ~62.5 GB BF16) |
| Compression ratio | ~2.7× |
Recommended Inference Parameters
| Parameter | Value |
|---|
temperature | 0.2 |
top_k | 50 |
top_p | 0.95 |
min_p | 0.01 |
LM Studio Jinja template or oMLX custom kwargs
add these flags to the top of the jinja template or as custom kwargs to use this model in the way it was intended by GLM:
1{%- set enable_thinking = true -%}
2{%- set clear_thinking = false -%}
Use with mlx-lm
1from mlx_lm import load, generate
2
3model, tokenizer = load("leonsarmiento/GLM-4.7-Flash-6bit-mlx")
4
5prompt = "Hello, how are you?"
6
7response = generate(model, tokenizer, prompt=prompt, temp=0.2, top_k=50, top_p=0.95)
8print(response)
