Unlike uniform 4-bit quantization (which forces every layer onto the same bit grid and often collapses reasoning at low bit widths), this model was quantized with mlx-optiq using per-layer KL-sensitivity analysis:

A small calibration set (32 samples spanning prose, multi-step reasoning, code, and constraint-following instructions) is run through the FP16 reference and through trial quantizations of each layer.
The output drift per layer is measured. Layers whose outputs are most affected by quantization (typically the final attention projections, the lm_head, and a few middle blocks) get more bits; layers that tolerate aggressive quantization get fewer.
The final assignment hits the target average bits-per-weight while keeping the bits where they matter. This trades off precision unequally so the average comes out near the target (5.0 bits/weight), but the bits that matter most for output fidelity stay high.

Quantization config

Method: optiq_mixed_precision (mlx-optiq)
Target bits/weight: 5.0
Achieved bits/weight: 5.000
Candidate bits: [4, 6, 8]
Group size: 64
Sensitivity reference: bf16
Calibration: 32-sample 4-domain mix (prose + reasoning + code + constraints)

Per-layer bit allocation

84 quantizable components total. OptiQ allocated bits non-uniformly based on KL sensitivity:

Bits	Components	Share
8-bit	14	16.7%
6-bit	26	31.0%
4-bit	44	52.4%
Total	84	100.0%

Components kept at 8-bit (most sensitive to quantization):

model.layers.11.mlp.up_proj
model.layers.11.mlp.down_proj
model.layers.11.mlp.gate_proj
model.layers.11.self_attn.o_proj
model.layers.11.self_attn.v_proj
model.layers.11.self_attn.k_proj
model.layers.11.self_attn.q_proj
model.layers.0.mlp.up_proj
model.layers.0.mlp.down_proj
model.layers.0.mlp.gate_proj
model.layers.0.self_attn.o_proj
model.layers.0.self_attn.v_proj
model.layers.0.self_attn.k_proj
model.layers.0.self_attn.q_proj

Notice the pattern: the MoE feed-forward gates dominate the top tier. Gate weights pick which experts run for every token — small numerical drift here changes routing decisions and compounds through the rest of the network, so OptiQ pays the bits to keep them precise while compressing the experts themselves more aggressively.

Benchmark results

Evaluated on Apple M4 Pro with MLX. Model loaded once; performance and quality measured in a single pass.

Performance

	This model	Naive 8-bit	FP16 baseline
Decode tok/s (avg, long traces)	1587.34	1157.12	1270.13
Peak memory (GB)	0.141	0.154	0.223
Disk size (MB)	42	55	201

Quality

Benchmark	This model	Naive 8-bit	FP16 baseline	n
IFEval (instruction following)	13.6%	13.6%	15.9%	44
Alpaca-cleaned (instruct F1 vs reference)	34.6	37.5	36.2	50

Context scaling (decode tok/s)

Context length	Decode tok/s
~128 tokens	1586.2
~256 tokens	1584.5
~512 tokens	1586.1
~1024 tokens	1592.6

Usage

pip install mlx-lm

python

from mlx_lm import load, generate

model, tokenizer = load("sahilchachra/supra-50m-instruct-optiq-5bpw-mlx")
response = generate(model, tokenizer, prompt="Your prompt here", max_tokens=256, verbose=True)

All variants in this collection

Model	Variant
sahilchachra/supra-50m-instruct-8bit-mlx	Affine int8
sahilchachra/supra-50m-instruct-optiq-5bpw-mlx	OptiQ mixed-precision (target 5.0 bpw) ← this model

Notes

Requires Apple Silicon (M1 or later) with MLX
Benchmarks run on Apple M4 Pro, 24 GB unified memory
License: see SupraLabs/Supra-50M-Instruct for the original model's license

Original model

See SupraLabs/Supra-50M-Instruct for full model details and intended use.