An OptiQ mixed-precision 8-bit quant of codelion/dhara-250m, the second member of OptiQ's Diffusion LLM family, for Apple Silicon.

dhara is a tri-mode 250M model: one set of weights that decodes three ways, standard autoregressive (left-to-right), block-diffusion (fill a block of tokens and iteratively un-mask it), and self-speculation (draft a block with the diffusion forward, verify with the AR forward). It is a custom architecture stock mlx-lm can't load (it adds Canon depthwise-conv layers, QK-norm after RoPE, and a logit soft-cap); OptiQ ships a vendored, mlx-native port that registers with mlx-lm and is bit-exact to the reference.

At 250M, dhara is a base to fine-tune, the way Google's Gemma-270M is, small enough to LoRA on-device for one task, not a general assistant.

Demo: dhara-chat Space — chat with it and compare all three decoding modes.

Block-diffusion and self-speculation are handled by the OptiQ runtime. optiq serve --model mlx-community/dhara-250m-OptiQ-8bit serves an OpenAI/Anthropic-compatible API; --mtp routes through the self-speculative path. LoRA fine-tuning uses the standard optiq lora train autoregressive trainer.

At 250M there is no redundancy to spend, so this quant is tuned for fidelity to the reference, not for the smallest possible file. OptiQ's mixed precision holds the layers that need it at bf16 and drops the rest to 8-bit.

Measured against the bf16 reference on the same calibration prompts:

Decoded with the reference implementation's settings (greedy, repetition penalty 1.3), autoregressive and self-speculative decode are byte-identical to bf16; block-diffusion matches at 0.87 similarity (its confidence-threshold un-masking is chaotic, and more precision does not tighten it).

A 4-bit dhara stays fluent but answers differently from the reference — coherent text, different model. On a 250M base that is meant to be fine-tuned, matching the reference matters more than the last 200 MB.

The full 6-benchmark Capability Score (MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop) is unchanged:

BFCL, HumanEval and HashHop sit at 0 for every variant including bf16; this is a genuine 250M ceiling, not a quantization or harness artifact. Note the Capability Score is flat across every quant of this model, 4-bit included, because a 250M base has no benchmark headroom for quantization damage to show up in, which is why the KL and output-fidelity numbers above are the ones that decide the bit budget.

dhara decodes three ways from one set of weights. The recommended default is self-speculation (--mtp): it drafts a block in one parallel forward and verifies it autoregressively (two forwards per round, no commit pass), so the emitted output is identical to plain AR decode while committing ~3–4 tokens per round, AR accuracy at ~1.4× the speed of token-by-token AR. The model is overhead-bound (a 32-token forward costs about the same as a 1-token forward), so quantization buys size, not throughput.

Self-spec and block-diffusion are prefix-cached (KV + Canon-conv state), so each step processes only the new block, O(block) per step, not O(sequence).

OptiQ measures each layer's quantization sensitivity (KL divergence vs the bf16 reference on calibration data) and assigns per-layer bit-widths under a target budget. This quant: 99 weight tensors at 8-bit + 125 held at bf16, 10.25 bits-per-weight.

Because dhara is tri-mode, the calibration probes both forwards it serves, causal AR and block-diffusion, across the mask trajectory a block actually travels (100% masked down to 25%). Self-speculation needs no separate calibration: it composes the other two forwards.

The Canon depthwise convs, QK-norm, and logit soft-cap are not Linear modules, so they stay at bf16 automatically, only the attention and MLP projections are quantized.

This quant was produced by mlx-optiq. Point it at any Hugging Face model to get the same sensitivity-aware mixed precision:

Derived from codelion/dhara-250m. See the Diffusion LLM family guide for details.