Status: Experimental / Untested at scale. This is a research checkpoint. It has passed basic generation and small-sample benchmarks but has not been comprehensively evaluated. Use at your own risk and please report issues.

4-bit quantized version of Qwen/Qwen3.6-35B-A3B for Apple Silicon, using Quaternion-Optimal Rounding (QOR) -- an experimental quantization method that uses quaternion geometry to make smarter rounding decisions than standard round-to-nearest (RTN).

No custom code, no extra dependencies. Just pip install mlx-lm and go.

Quick Start

pip install mlx-lm

Interactive Chat (recommended)

bash

mlx_lm.chat \
  --model jacob9706/Qwen3.6-35B-A3B-QOR-4bit \
  --max-tokens 2048

This is a thinking model -- it reasons inside <think> tags before responding. Use --max-tokens 2048 (or higher) to give it room for both thinking and the response.

Single Prompt

bash

mlx_lm.generate \
  --model jacob9706/Qwen3.6-35B-A3B-QOR-4bit \
  --prompt "Explain spherical k-means in three sentences." \
  --max-tokens 2048

Python API

python

from mlx_lm import load, generate

model, tokenizer = load("jacob9706/Qwen3.6-35B-A3B-QOR-4bit")

messages = [{"role": "user", "content": "What is quaternion geometry?"}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)

response = generate(model, tokenizer, prompt=prompt, max_tokens=2048)
print(response)

What is QOR?

Standard quantization (RTN) rounds each weight independently to the nearest integer grid point, minimizing per-element error. But for tasks like attention (Q-K similarity) and MoE gating (expert selection), what matters is preserving the direction of output vectors, not minimizing element-wise error.

QOR (Quaternion-Optimal Rounding) takes a different approach:

Angular-error-optimal rounding: For each group of 4 consecutive weights, QOR exhaustively searches all 16 floor/ceil combinations and picks the one that best preserves the direction of the layer's output (minimizes 1 - cos(W_q x, W x)). This is done offline during quantization -- zero cost at inference.
Quaternion pre-rotations (optional, not used in this checkpoint): Learned block-diagonal 4x4 rotations that transform weights into a more quantization-friendly basis before rounding.

The output is a completely standard MLX INT4 checkpoint. At runtime, it uses the same native mx.quantized_matmul kernel as any other 4-bit model. The improvement lives entirely in which integer values were chosen.

Potential benefits

~7% lower angular error vs RTN across all weight matrices, meaning output directions are better preserved after quantization
Directional fidelity matters most for attention similarity and MoE routing -- the components that determine which experts fire and how context is attended to
Same speed, same memory as standard INT4 -- the improvement is free at inference time
Benefits are expected to be larger at lower bit widths (2-bit, 3-bit) where rounding decisions matter more

What's NOT tested

This checkpoint has not been comprehensively evaluated. Specifically:

No full MMLU-Pro / MMLU-Redux run (only a 28-question smoketest)
No long-context evaluation
No coding benchmarks (HumanEval, MBPP)
No safety/alignment evaluation
No multi-turn conversation testing
The angular error improvement has not been definitively linked to downstream task gains at 4-bit

If you run evaluations on this model, please share your results!

Early Numbers

Tested on Apple M4 Max (48 GB). Take these with a grain of salt -- sample sizes are small.

Metric	QOR 4-bit	mlx-community RTN 4-bit	Unquantized (reported)
Throughput	88 tok/s	86 tok/s	N/A
Peak memory	19.6 GB	19.6 GB	~70 GB
Disk size	18 GB	18 GB	65 GB
MMLU-Pro (n=28)	85.7%	89.3%	85.2%

Note: The MMLU-Pro numbers are from a 28-question smoketest and are noisy (+/-3.6% per question). A full-dataset evaluation is needed for meaningful comparison.

Quantization Details

Parameter	Value
Bit width	4
Group size	64
Format	Affine (min-max per group)
Calibration	32 random normal vectors
Block search	Exhaustive 2^4 = 16 candidates per 4-block
Quantized	All linear projections, embeddings, lm_head, MoE gates, experts
Not quantized	Layer norms, biases, conv1d

Base Model

Qwen/Qwen3.6-35B-A3B -- Mixture-of-Experts, 35B total parameters, ~3B active per token. Hybrid architecture with Mamba-style linear attention on most layers and standard attention on select layers.

Disclaimer

This is a slopped out idea.

License

Apache 2.0 (inherited from the base model).