Quantization: Affine integer quantization
Precision: 5-bit (~5.5 bits/weight avg)
Group size: 64
Disk size: 5502 MB
Quantized by:
sahilchachra
Affine quantization at 5-bit with group size 64. Good middle ground between 4-bit compression and 6-bit quality.
Evaluated on Apple M5 Pro with MLX. All metrics measured in a single pass (model loaded once).
1from mlx_lm import load, generate
2
3model, tokenizer = load("sahilchachra/granite-4.1-8b-5bit-mlx")
4response = generate(model, tokenizer, prompt="Your prompt here", max_tokens=512, verbose=True)
1from mlx_lm import load, stream_generate
2
3model, tokenizer = load("sahilchachra/granite-4.1-8b-5bit-mlx")
4for chunk in stream_generate(model, tokenizer, prompt="Your prompt here", max_tokens=512):
5 print(chunk.text, end="", flush=True)
See
ibm-granite/granite-4.1-8b for full model details, training information, and intended use.
