Quantization: Affine integer quantization
Precision: 8-bit (~8.5 bits/weight avg)
Group size: 64
Disk size: 8499 MB
Quantized by:
sahilchachra
Affine quantization at 8-bit with group size 64. Closest to FP16 quality. Recommended when memory allows and quality is the priority.
Evaluated on Apple M5 Pro with MLX. All metrics measured in a single pass (model loaded once).
1from mlx_lm import load, generate
2
3model, tokenizer = load("sahilchachra/granite-4.1-8b-8bit-mlx")
4response = generate(model, tokenizer, prompt="Your prompt here", max_tokens=512, verbose=True)
1from mlx_lm import load, stream_generate
2
3model, tokenizer = load("sahilchachra/granite-4.1-8b-8bit-mlx")
4for chunk in stream_generate(model, tokenizer, prompt="Your prompt here", max_tokens=512):
5 print(chunk.text, end="", flush=True)
See
ibm-granite/granite-4.1-8b for full model details, training information, and intended use.
