Quantization: Affine integer quantization
Precision: 6-bit (~6.5 bits/weight avg)
Group size: 64
Disk size: 6501 MB
Quantized by:
sahilchachra
Affine quantization at 6-bit with group size 64. Best quality among the smaller uniform variants, recommended for general use.
Evaluated on Apple M5 Pro with MLX. All metrics measured in a single pass (model loaded once).
1from mlx_lm import load, generate
2
3model, tokenizer = load("sahilchachra/granite-4.1-8b-6bit-mlx")
4response = generate(model, tokenizer, prompt="Your prompt here", max_tokens=512, verbose=True)
1from mlx_lm import load, stream_generate
2
3model, tokenizer = load("sahilchachra/granite-4.1-8b-6bit-mlx")
4for chunk in stream_generate(model, tokenizer, prompt="Your prompt here", max_tokens=512):
5 print(chunk.text, end="", flush=True)
See
ibm-granite/granite-4.1-8b for full model details, training information, and intended use.
