oQ4e quantization of Qwen/Qwen3.5-4B with a float16 base dtype, built with craquehouse/model-lab.

oQ defaults to a bfloat16 base. M1 and M2 have no native bf16 path in the GPU, so this build uses fp16 instead — worth roughly 20% of prefill on those parts by oMLX's own reckoning. Both are 16-bit, so it is not a size trade. On M3 and later, hardware bf16 closes the gap and this build has no advantage over a normal one.

fp16's exponent range is narrower than bf16's (max ~65504 against ~3.4e38). If you see inf or degenerate output on hardware where the bf16 build is fine, that is the thing to suspect.

The vision and audio towers are dropped entirely, which is what makes this build genuinely fp16: oQ holds towers at fp32 whenever the target dtype is float16, so a multimodal build of the same model is not the dtype its name claims.

Tensors written: F16 x651, F32 x24, U32 x249. The 24 fp32 tensors are all A_log, the SSM / linear-attention state parameters, which oQ never quantizes or narrows.

This is a thinking model and reasoning is enabled unless you turn it off — the reply arrives in reasoning_content, and content can come back empty if max_tokens runs out mid-thought.

That default comes from this repo's own chat template, which gates on the variable being explicitly false:

An unset enable_thinking takes the else branch and opens a thinking block, so silence means reasoning. Turn it off per request:

Note this costs 2 prompt tokens rather than saving any — the disabled form emits a pre-closed <think>\n\n</think>\n\n pair, which is longer than the bare opener. Measured on the same prompt: 16 prompt tokens and 27 output in 0.71s with reasoning off, against 14 prompt tokens and 224 output in 3.82s with it on.

A top-level "enable_thinking": false does not work — oMLX's request model does not declare that field, so it is silently dropped. reasoning_effort is accepted and then discarded, as this template has no such variable. Both were tested: each returned a full reasoning block.

Home Assistant, and most OpenAI-compatible integrations, give you no way to set chat_template_kwargs. Set the default on the server instead, per model, in ~/.omlx/model_settings.json:

The key is the API-visible model id, which is the model directory's basename — not an org-qualified path, even when the model is stored under one. oMLX reads this file at startup and does not rewrite it.

This sets a default, not a lock: a client that does send chat_template_kwargs can still turn reasoning back on, which the measurements above demonstrate — they were taken with this file in place. oMLX's ModelSettings also carries a forced_ct_kwargs field, described as keys that cannot be overridden per request; that path is untested here.

Built with oMLX 0.6.3rc2. The version is part of the recipe, not trivia: 0.5.0-rc1 shipped an oQ4/oQ4e regression worth ~37 points of GSM8K, and the checkpoints loaded and generated fluently throughout (#2172).

Base 4-bit affine at group 64, with 98 tensors held above it by oQ's sensitivity budget: 91 at 5-bit, 7 at 6-bit. The boosts are the point of oQ and they are invisible in the file size — build.json in this repo records them, and oq_imatrix_report.json records what the 128-sample calibration pass actually covered.

Measured by oMLX 0.6.3rc2's own benchmark (/admin/api/bench/start, corpus code_python, TG=128) on an Apple M2 Pro with 16 GB, reasoning disabled.

Prefill settles around 420-440 t/s once the GPU saturates; the PP1024 row is flattered by running at 92% utilisation against ~99.7% everywhere else. Decode falls 59.6 to 48.4 t/s across a 16x context increase — -19%, with TPOT rising 16.9 to 20.8 ms.

2.5x the aggregate throughput at 4x concurrency, with per-request latency degrading roughly in proportion. Peak memory is flat at 3.50 GiB across all three: KV for four 1024-token streams is negligible against the weights.

oMLX pins concurrent tests to PP1024 — it generates batch prompts at that length regardless of what else you ask for — so this says nothing about concurrency at long context.

One run per cell, on a machine also serving an embedding model and a separate audio stack, so the memory figures include a co-resident neighbour and none of these are best-case. Treat them as directional.

86.7% on assist-mini (170/196), verified by asserting entity-state changes inside Home Assistant 2026.7.4 — not by matching strings against expected output.

Divergence says how far this build's output distribution moved from the checkpoint it was quantized from. It does not say how the model scores on any task — there is no MMLU, no GSM8K, no HumanEval here, and none is implied. Those measure how good the model is; a quantization's claim is that it is the same model, which is what the number above tests.

Inherits the licence of the base model, Qwen/Qwen3.5-4B. Quantization does not create a new work; check the base model's card before redistributing.