Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Try the Lab · All OptiQ quants · Docs
A 4-bit mixed-precision MLX quant of deepreinforce-ai/Ornith-1.0-35B, built on the Qwen3.5-35B-A3B Mixture-of-Experts architecture (256 experts, 8 active per token). Sensitive layers are kept at 8-bit and robust ones at 4-bit.
65 GB of bf16 weights become 22 GB.
Image input works. The vision tower is kept at bf16 in a sidecar, so this quant takes images as well as text.
Running it on a 24 GB Mac
At 22 GB this does not fit comfortably in a 24 GB Mac's Metal working set. Serve it with SSD expert streaming, which reads only the active experts per token:
That brings resident memory down to 4.58 GB. Streaming is the default (auto) in optiq serve, so it engages by itself when a MoE will not fit; the flag above just makes it explicit. On a 32 GB+ Mac the model fits resident and streaming is unnecessary.
Quantization details
Property
Value
Predominant precision
4-bit
Layers at 8-bit (sensitive)
397
Layers at 4-bit (robust)
113
Total quantized layers
510
Achieved bits per weight
4.513
Group size
64
Experts
256 per layer, 8 active per token
Vision tower
bf16, 333 tensors, in optiq/optiq_vision.safetensors
Size on disk
22 GB, from a 65 GB bf16 base
We follow the same naming convention llama.cpp uses for Q4_K_M and similar mixed-precision quants: the "4-bit" label is the predominant precision, not the weighted average.
The base model ships no MTP head, so this quant has no speculative-decoding sidecar.
Ornith-1.0-35B shares the Qwen3.5-35B-A3B architecture unchanged (no shape or math field of the text config differs), so all 510 quantizable layers map across exactly and the allocation lands at the same 4.513 bits per weight when recomputed against Ornith's own tensors.
These are measured bit-widths, not a static rule-of-thumb recipe. But they were measured on the base architecture, not on this model. Training shifts weights, so Ornith's own per-layer sensitivities could differ somewhat. Which layers are fragile is mostly a property of the architecture, so the transfer is sound, but it is a transfer and you should know that.
Only the language tower is quantized. The vision tower stays at bf16, which is how every OptiQ VLM ships.
Note that mlx_lm.load holds the whole model resident, which is slow on a 24 GB Mac. Prefer optiq serve --stream-experts there.
This is a reasoning model: it thinks before answering, so give it enough max_tokens to finish.
Images
Send an image through the OpenAI-compatible endpoint:
python
1import base64, io, requests
2from PIL import Image
34buf = io.BytesIO(); Image.open("photo.jpg").save(buf,format="PNG")5uri ="data:image/png;base64,"+ base64.b64encode(buf.getvalue()).decode()67requests.post("http://127.0.0.1:8080/v1/chat/completions", json={8"model":"ornith","max_tokens":256,9"messages":[{"role":"user","content":[10{"type":"text","text":"What is in this image?"},11{"type":"image_url","image_url":{"url": uri}}]}]})
Verification
Text generation and arithmetic reasoning were exercised on the finished artifact before release, through expert streaming on a 24 GB M4.
The quantization was also checked numerically: dequantizing individual experts out of the artifact and comparing them against the corresponding experts in the bf16 checkpoint gives 0.74-0.76% mean relative error on the 8-bit layers and 10.0% on the 4-bit layers, which is what each bit-width should cost. Experts were sampled across layers 0, 20 and 39, including expert 255 of 256.
No task benchmarks were run on this quant; for measured quality numbers on the base architecture, see the Qwen3.5-35B-A3B OptiQ card.
Quantization does not change the behaviour or alignment of the base model. Use it under the same terms as the original.
