The original bf16 weights (~70 GB) are too large for most local setups. This conversion brings the model to ~22 GB with 5-bit affine quantization via Apple's MLX framework, making it practical to run entirely on-device on a Mac with 32 GB or more of unified memory.
Why this model matters
Claude-grade reasoning, open weights
Trained on ~8K reasoning traces from Claude Opus 4.7 with explicit <think>…</think> chain-of-thought
Sparse MoE efficiency
256 experts, 8 routed + 1 shared — only ~3B parameters active per token despite 35B total
Apple Silicon native
MLX quantized weights load directly into unified memory — no CPU↔GPU copies, no CUDA required
Long context
262K token context window (base architecture); 64K usable at inference
Quick Start
Install
pip install mlx-lm
Generate
python
1from mlx_lm import load, generate
23model, tokenizer = load("AntimatterAI/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-MLX-5bit")45messages =[{"role":"user","content":"How many positive integers less than 1000 have digits that sum to 20?"}]6prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)78response = generate(model, tokenizer, prompt=prompt, max_tokens=8192, verbose=True)9print(response)
40 (mixed linear + full attention, every 4th layer is full attention)
Hidden size
2048
Attention heads
16 (KV heads: 2)
Head dim
256
Expert FFN size
512 per expert
Context window
262,144 tokens (architecture max)
Vocabulary
248,320 tokens
Quantization Details
Method
MLX affine quantization
Precision
5-bit weights (5.502 effective bits per weight)
Group size
64
Router gates
Preserved at 8-bit (all 40 layers — mlp.gate and mlp.shared_expert_gate)
Vision tower
Stripped during conversion (text-only serving)
Disk size
~22 GB (5 sharded safetensors)
Conversion tool
mlx-lm v0.31.x (python -m mlx_lm convert)
Router gates are kept at 8-bit precision to preserve expert routing quality — the gates are small tensors but critical for MoE performance.
Benchmarks
Benchmarks below are from the original bf16 model as evaluated by lordx64. Quantization to 5-bit typically incurs minimal degradation on MoE architectures due to the sparse activation pattern.
The MoE architecture activates only ~3B parameters per token, so prefill memory per step is much smaller than a dense 35B model. Expect ~30–50 tokens/sec decode speed on M-series chips depending on memory bandwidth.
Reasoning Behavior
This model emits explicit <think>…</think> reasoning blocks before its final answer — this is by design, inherited from the Claude Opus 4.7 distillation. Example:
<think>
I need to count positive integers less than 1000 whose digits sum to 20.
Let me denote a 3-digit number as having digits a, b, c where...
[detailed step-by-step reasoning]
</think>
The answer is 75.
For applications where you only need the final answer, strip everything between <think> and </think> in post-processing.
Preventing Thought Loops
At 5-bit quantization, this model will degenerate inside <think> blocks and during extended generation. The degeneration takes multiple forms: paragraph-level repetition, single-word stutter, synonym chain runaway, word concatenation, and free-association word salad. Repetition penalties alone are insufficient — you must also disable extended thinking and cap token output.
Recommended: disable thinking mode at the server level:
Critical. Prevents the model from entering <think> blocks where degeneration is most severe
repetition_penalty: 2.0
Strong penalty needed — lower values (1.15–1.5) only change the degeneration pattern without stopping it
repetition_context_size: 1024
Wide lookback window to catch long-distance repetition
max_tokens: 2048
Hard cap — the model produces ~100–300 tokens of useful content before quality degrades
For production use, the included proxy (mlx_openai_input_limit_proxy.py) provides output-side loop detection that catches degeneration patterns and truncates responses at the last coherent sentence. This is the most reliable defense.
Why this happens: The 5-bit quantization reduces the model's ability to maintain coherent long-range generation. Combined with reasoning-distillation training that strongly biases toward extended output, the model enters self-reinforcing degeneration modes that escalate from word repetition to increasingly creative forms of incoherence.
Training Lineage
Qwen/Qwen3.6-35B-A3B (Apache 2.0)
└── lordx64/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled (LoRA SFT on Claude Opus 4.7 traces)
└── AntimatterAI/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-MLX-5bit (this model — MLX 5-bit conversion)
Reasoning over knowledge. Distillation transfers how to think, not new facts. The model's knowledge boundary is that of Qwen 3.6.
Long generation budgets. The model will use 5–30K tokens of <think> reasoning on hard problems. Set max_tokens accordingly.
Attention-only LoRA. Expert FFNs are untouched from the base — domains where Claude and Qwen diverge may show uneven results.
Distillation provenance. Training data was generated via Anthropic's Claude API. Downstream users should confirm compliance with Anthropic's usage policies.
Citation
If you use this model, please cite the original work and this conversion:
bibtex
1@misc{lordx64_qwen36_distill_2026,
2 title = {Qwen3.6-35B-A3B distilled from Claude Opus 4.7 reasoning},
3 author = {lordx64},
4 year = {2026},
5 url = {https://huggingface.co/lordx64/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled},
6}
78@misc{qwen36_a3b_2026,
9 title = {Qwen3.6-35B-A3B},
10 author = {Qwen Team},
11 year = {2026},
12 url = {https://huggingface.co/Qwen/Qwen3.6-35B-A3B},
13}
1415@misc{antimatterai_mlx_5bit_2026,
16 title = {Qwen3.6-35B-A3B Claude Opus 4.7 Reasoning — MLX 5-bit conversion},
17 author = {Antimatter AI},
18 year = {2026},
19 url = {https://huggingface.co/AntimatterAI/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-MLX-5bit},
20}
Acknowledgements
lordx64 — original model author; fine-tuned Qwen3.6 on Claude Opus 4.7 reasoning traces and published the weights under Apache 2.0.
Qwen Team — for the Qwen3.6-35B-A3B base model with a permissive license.
Anthropic — for Claude Opus 4.7, the teacher model.
Apple MLX Team — for the MLX framework enabling efficient on-device inference.
Unsloth — for accelerated LoRA training tooling used in the original fine-tune.
Antimatter AI — Building Digital Solutions That Matter
AI Development | Product Design | Healthcare Apps | IoT
