MLX 6-bit quant of
ibm-granite/granite-guardian-4.1-8b. Tighter quality than 4-bit at modest cost. Disk: ~6.3 GB. Active memory: ~7 GB plus KV cache.
For a classification model, 4-bit is usually fine. Pick 6-bit if you've measured 4-bit producing wrong calls in your specific risk categories.
1mlx_lm.convert --hf-path ibm-granite/granite-guardian-4.1-8b \
2 --mlx-path ./granite-guardian-4.1-8b-mlx-6bit -q --q-bits 6
Same as the 4-bit. Granite Guardian uses a structured prompt to classify a piece of input/output text. See the
base model card for the prompt template.
1temperature: 0.0
2max_tokens: 16
Apache-2.0 from base.
