BAAI/AREX-Turbo converted to MLX and quantized to 8-bit, for inference on Apple Silicon.

Converted with mlx-vlm 0.6.8 (mlx 0.32.0).

The effective figure exceeds 8 because mlx-vlm leaves the vision tower in bf16 by design and quantizes only the language model — the vision encoder is a small share of the weights but disproportionately sensitive to quantization error. Verified on this conversion:

language_model: QuantizedLinear x186, QuantizedEmbedding x1   <- 8-bit
vision_tower  : Linear x50, LayerNorm x25, Conv3d x1          <- bf16

All numbers below were measured against the unquantized bf16 source on an M2 Pro (32 GB), greedy decoding throughout.

Perplexity over 204 tokens of held-out text spanning prose, code, legal and scientific registers; top-1 agreement and KL are computed per position over the full next-token distribution. These are the numbers to judge quantization by — they involve no sampling and no decoding choices.

Greedy (temperature=0.0) continuations, scored against the bf16 output as reference.

Read these as agreement, not quality. bf16 is the reference here, not ground truth, so a divergence is only a defect if the quantized answer is worse. It is not always: on 17 * 23 the bf16 model started a long derivation while the 8-bit model answered 391 directly — a mismatch that counts against BLEU while being the better response. That is why the two sections below exist.

Pass rate on 8 short prompts with verifiable answers (arithmetic, factual recall, sorting). Deterministic — this is the only layer that measures correctness rather than similarity.

18 open-ended prompts graded by claude-sonnet-5. The judge never sees which model produced which answer. Pairwise comparisons are run twice with positions swapped; a verdict counts only if the judge picks the same model both times, and disagreements are reported as inconsistent rather than resolved silently. Absolute grades are averaged over 2 repeats per answer.

Gap of 0.028 against combined judge noise of 0.216 → indistinguishable from bf16.

Judge noise is not negligible and is reported rather than hidden: grading the same bf16 model across separate runs varied by ~0.17 on this scale. Differences smaller than the combined SEM should not be read as a ranking.

BLEU against bf16 falls from 82.7 (8-bit) to 49.5 (4-bit), while task accuracy and judged quality stay flat. Judged quality is statistically indistinguishable from bf16 at every bit width tested. KL divergence from bf16 drops ~76x across that same range.

Hardware-specific; will differ on other chips.

No standard task benchmarks (MMLU, GSM8K, agentic/tool-use evals) were run. The accuracy layer above is 8 short prompts, not a benchmark. The vision path was checked for coherence on sample images but not scored. The source model is agent/deep-research oriented, and none of its agentic capabilities were evaluated here — if that is your use case, measure on your own data.

With an image:

See the original model card for capabilities, intended use and limitations. All credit for the model belongs to its authors.