The same code path was validated on GLM-4.7-Flash (355B, same MoE architecture with 64 experts) where it loaded successfully and scored correctly on all test prompts with 13.3 GB GPU memory.

Architecture

GLM-5.1 uses the Glm4MoeLiteNaiveMoe architecture:

769B total parameters, 40B active per token
256 routed experts, 8 active per token, 1 shared expert
78 layers, hidden_size=6144
Multi-head Latent Attention (MLA)
First 3 layers are dense (not MoE)
200K context window

How it works

The WHT rotation + Gaussian Lloyd-Max codebook from TurboQuant (ICLR 2026). After a random Walsh-Hadamard rotation, weight distributions become near-Gaussian, making them efficiently quantizable with 8 centroids (3-bit) per 128-element group. Zero calibration data needed.

The checkpoint stores packed 3-bit indices + per-group norms. The loader handles:

Per-expert 2D → fused 3D regrouping (gate_proj + up_proj → gate_up_proj fusion)
Router/gate weight decompression in-place
Meta-device model creation for low-memory loading

Usage

pip install turboquant-plus-vllm@git+https://github.com/varjoranta/turboquant-vllm.git

python

from turboquant_vllm import load_tq3_model

model, tokenizer = load_tq3_model("varjosoft/GLM-5.1-Open-TQ3", device="cuda")
# Requires multi-GPU setup — see requirements below

GPU requirements for inference

Setup	Total VRAM	Per-GPU	Cost/hr (Verda)
8× A100 80GB	640 GB	45 GB	$10.32
4× H200 141GB	564 GB	90 GB	$13.56
2× B300 262GB	524 GB	180 GB	$13.98

Without TQ3, the BF16 model requires 1,510 GB VRAM (minimum 8× B300 at $55.92/hr).

Software requirements

transformers >= 5.5.0
turboquant-plus-vllm (GitHub)
PyTorch with CUDA

Comparison with other quantizations

Method	Size	Calibration	Format	Target
This (TQ3)	309 GB (4.9x)	None	Safetensors	GPU serving (vLLM/PyTorch)
Unsloth Dynamic 2-bit	236 GB (6.4x)	300K+ tokens	GGUF	Local/CPU (llama.cpp)
BF16 original	1,510 GB	N/A	Safetensors	8× B300+

License

MIT (same as base model). Created by Varjosoft Oy.

nwzjk/GLM-5.1-Open-TQ3

GLM-5.1 TQ3 (3-bit weight compression)

Compression

Status