MLX (Apple Silicon) port of Solar Open 2 (250B-A15B Hybrid-Attention MoE) in three quantization tiers, produced with alis-dwq — data-aware quantization with teacher-guided distillation (DWQ). Both distilled builds are fine-tuned against the Q8 reference under a per-slice acceptance gate, with an EN sentinel gate as the final safety net.
✅ Status: all three tiers final. Q v3 (4-bit/g32 + DWQ, lr 5e-7 × 3 rounds): EN loop cleared with teacher-level diversity (distinct4 0.399 vs teacher 0.411), code KLD better than its own base. F v2 (2.34 bpw, 73.3 GB): KLD 0.42 (−76% vs the legacy 2-bit path), all loops clean, EN sentinel ACCEPT. A fourth tier (M, 4-bit/g64) was trained twice (Q-teacher and Q8-teacher) but is not shipped — it lost every metric to Q v3 at the same memory tier.
Solar Open 2 architecture & quantization map — hybrid attention stack [Softmax, Linear×3]×12, MoE 320 routed + 1 shared experts, routers pinned to 8-bit
Builds (branches)
One repo, three weights — pick a branch by machine memory:
Sizes are decimal GB (10⁹ bytes, as shown on the HF page) with binary GiB (2³⁰ bytes) in parentheses. Measured bpw is computed on the weight shards over 250.29B params; the t96 adapters (4.0 GB) ship separately and are not fused into the shards.
3-tier build lineup — disk size and measured bits per weight
Every build keeps all MoE router gates at 8-bit / g64 (48/48 layers) — router quantization was found to be the dominant quality risk, so only expert/attention weights are quantized per the tier recipe. Why no mid tier? An M build (4-bit/g64, 131–141 GB) was distilled twice (Q-teacher and Q8-teacher arms). The Q8 arm fixed its EN regression but still lost every metric to Q v3 at the same memory tier (≥160 GB machines), so it is retained locally as a control, not shipped.
Install & run
⚠️ Requires the solar_open2 architecture, which is not yet in PyPI mlx-lm (0.31.3). Stock pip install mlx-lm will fail with ValueError: Model type solar_open2 not supported. Two options:
bash
1# option A — the fork that adds solar_open2 (recommended; the fork's default branch is `solar-open2`)2pip install"git+https://github.com/avlp12/mlx-lm.git"
Option B — keep stock mlx-lm and drop the bundled solar_open2.py (shipped in every branch) into .../site-packages/mlx_lm/models/. Same file, same result.
bash
1# main = Q v3 (recommended)2mlx_lm.generate --model avlp12/Solar-Open2-250B-Alis-MLX-Dynamic \3 --prompt "안녕하세요, 간단히 자기소개를 해주세요." --max-tokens 512
Branch selection (t96 = F v2, t384 = T Q8) uses the Python API — the CLI has no --revision flag:
python
1from mlx_lm import load, generate
23model, tok = load("avlp12/Solar-Open2-250B-Alis-MLX-Dynamic", revision="t96")# F v2 (77 GB)4# model, tok = load("avlp12/Solar-Open2-250B-Alis-MLX-Dynamic", revision="t384") # T Q8 (266 GB)5print(generate(model, tok, prompt="Explain MoE routing in three sentences.", max_tokens=512, verbose=False))
CLI-only alternative: hf download avlp12/Solar-Open2-250B-Alis-MLX-Dynamic --revision t96, then point --model at the downloaded snapshot directory.
F v2 LoRA adapters (t96 branch only): the DWQ compensator adapters ship un-fused in alis_adapters/ (rank 8). Download that folder and pass --adapter-path <local alis_adapters dir> to use them. All numbers in this card are measured without the adapters (same convention as the control arm), so a fused deployment may do better.
Chat, reasoning & sampling. The chat template (chat_template.jinja) ships in every branch; mlx_lm applies it automatically in chat/server mode (raw mlx_lm.generate with a plain string prompt does not — apply the template yourself). Client-side settings follow the base model card: temperature 1.0 / top_p 1.0 for both reasoning modes — avoid ~0.7-class temperatures, which measurably raise loop/runaway-thinking risk on quantized builds. reasoning_effort defaults to "high" (thinking ON); use "none" for a direct answer (--chat-template-args '{"reasoning_effort":"none"}' on mlx_lm.server, or the apply_chat_template kwarg). Reasoning markers are dedicated tokens <|think:start|> / <|think:end|> (ids 130/131) — there is no</think> or "Final Answer" convention. Split reasoning from the final answer on <|think:end|>, and do not add it as a stop token: the answer comes after it, and termination is <|im:end|> (id 129, already in generation_config.json). Budget max_tokens for thinking + answer together — thinking typically runs 0.5–2K tokens on chat prompts, so ≥4K total is recommended; an empty answer or a reply cut off mid-thought means the budget ran out inside the reasoning block. Server + LoRA adapters (F v2): fork revisions ≤ 9d67a0c silently dropped --adapter-path in mlx_lm.server — the adapter lookup ran after the model-name remap, always missed, and the server then served the bare 2-bit build. Fixed in 7937fda (branch solar-open2); update the fork, or pass "adapters": "<local alis_adapters dir>" in the request body as a workaround.
Hardware compatibility (memory planning)
Weights are loaded fully into GPU-wired unified memory; context costs extra on top. KV cache ≈ 48 KiB/token fp16 — only 12 of 48 layers (the softmax GQA layers, every 4th) hold a KV cache; the 36 KDA linear-attention layers carry a fixed-size recurrent state (~150 MiB total) that does not grow with context. → ~4.8 GiB per 100K tokens · 256K ≈ 12 GiB · 1M ≈ 48 GiB.
Default GPU wired limit on macOS ≈ 80% of RAM; raise it with sudo sysctl iogpu.wired_limit_mb=<MB> (revert by lowering again; needs admin). Build footprints below use exact GiB: F v2 68.3 GiB (+3.9 GiB adapters) · Q v3 145.7 GiB · T 247.7 GiB.
machine RAM
default GPU limit (~80%)
what fits stock
with wired-limit bump
96 GiB
~77 GiB
F v2 — 한계 직전 (여유 ~5 GiB ≈ 100K tok ctx), bump 권장
~88 GiB: F v2 + ~16 GiB ctx (~330K tok)
128 GiB
~103 GiB
F v2 + ~31 GiB ctx (~640K tok)
—
192 GiB
~154 GiB
Q v3 — 한계 직전 (여유 ~8 GiB ≈ 170K tok ctx), bump 권장
~176 GiB: Q v3 + ~30 GiB ctx (~630K tok)
256 GiB
~206 GiB
Q v3 + ~60 GiB ctx (full 1M tok)
—
384 GiB
~309 GiB
T (Q8) + ~61 GiB ctx (full 1M tok)
—
512 GiB
~412 GiB
T (Q8) + full 1M ctx
—
Rule of thumb: pick the largest build whose weights fit your machine with the context you need — quality scales with bits (see the measured table below). DWQ training peak observed: 275 GB wired+compressed during Q v3 distillation (training-only; inference needs are far lower).
Note on the HF sidebar panel: the native Hardware compatibility widget indexes only the repo's root model (main = Q v3, shown as "4-bit · MLX · 156 GB") — unlike GGUF repos, where every quant file gets its own row, an MLX repo gets a single row and branch builds (t96, t384) are not indexed. Use the table above for F/T hardware sizing.
Q tier journey (disclosed): the 4-bit/g32 base shipped a greedy EN loop (len23). Attempt 1 (Q-dwq, legacy scalar gate) made it worse (len27) — the scalar gate's blind spot that motivated the per-slice gate. Attempt 2 (Q v2, gate + Q8 teacher, lr 1e-6 × 6 rounds) cleared the loop but collapsed EN diversity (distinct4 0.221) and regressed KLD +6.7% — over-distillation. Attempt 3 (Q v3, same gate/teacher but lr 5e-7 × 3 rounds) keeps the loop fix, restores EN diversity to 0.399 (97% of the Q8 teacher's 0.411), brings KLD EN back to base level (+0.7%), and improves code KLD −5.1% below its own base. The residual trade-off is Korean KLD +8.4% vs base (0.10553 → 0.11438) — see Known Limitations.
F tier: the legacy path (uniform 2-bit/g128 base + clip + DWQ) failed the usability gate (KLD 1.23, all slices LOOPED) — a uniform 2-bit predicate destroys the KDA chokepoints and embed/head beyond DWQ recovery. F v2 re-quantizes with quant_predicate_builder (chokepoint 8b / embed+head 6b / attn+shared 4b / experts 2b, 578/578 per-layer entries verified) and distills from the Q8 teacher under a per-slice acceptance gate with rank-8 LoRA compensators. Result: KLD 1.73 (legacy base) → 1.23 (legacy DWQ) → 0.42 (F v2 + DWQ), all loop probes clean, EN sentinel ACCEPT. The LoRA adapters ship separately (alis_adapters/); the numbers above are measured without them (same convention as the legacy arm), so a fused deployment may do better.
DWQ training detail: 200 calibration samples (180 train / 20 valid), expert-routing coverage 99.8%; teacher = Q8 reference for all shipped builds; deepest-first layer order, 3 rounds (Q v3) / 6 rounds (F v2); per-slice acceptance gate (EN ceiling ε=1%) + EN sentinel gate (paired, cap 1%).
Effective bit utilization (code-entropy audit, §2-2):
Metric
M (4-bit/g64)
F (2-bit/g128 clipped)
Effective bits/weight
3.72 (93% of nominal)
1.65 (82% of nominal)
Low-entropy groups
0.0%
0.5% (concentrated in L0, worst 31.7%)
Weak (<50% util) experts
0 / 144 stacks
0 / 144 stacks
Quantization efficiency — nominal vs measured vs effective bits per weight
F clip requantization gate (T-dequant clipping before 2-bit requant):
Held-out PPL
Before clip
After clip
Δ
wikitext
110.56
49.51
−55.2%
code
21.81
11.63
−46.7%
Korean
23.94
18.93
−20.9%
F build clip gate — held-out PPL before vs after clipping
Base model (bf16, official card numbers)
Solar Open 2 is Upstage's 250B-A15B open-weight model: Hybrid-Attention MoE — [Softmax, Linear×3] × 12 (only 12 of 48 layers hold a KV cache), NoPE on linear-attention layers, 321 experts (320 routed top-8 + 1 shared), vocab 196,608, 1M-token context, ~12T pre-training tokens, English/Korean/Japanese.
Benchmark
Solar Open 2 (bf16)
MMLU-Pro
86.2
GPQA-Diamond
86.3
SWE-Bench Verified
70.4
AIME 2026
95.7
KMMLU-Pro
78.4
(Official numbers from the base model card; quantized-build deltas are tracked in the KLD table above.)
Known Limitations
Q v3 carries a disclosed Korean KLD regression (+8.4% vs its 4-bit/g32 base: 0.10553 → 0.11438) — the price of clearing the base's EN greedy loop while restoring EN diversity. English and code KLD are at or below base level; if your workload is Korean-critical and you can tolerate a known EN loop, prefer the T (Q8) tier or the base model.
F v2 is a 2.34 bpw build: Korean KLD (0.75) is materially higher than the Q tier — expected at this bit budget; prefer Q v3 for Korean-heavy workloads. The DWQ LoRA adapters ship un-fused (alis_adapters/); all reported numbers are without them.
Reasoning-block runaway (root-caused, disclosed): an external integrator reported F v2 thinking that never closed (22.6K chars, no <|think:end|>). This was traced to a serving bug, not the weights: fork mlx_lm.server ≤ 9d67a0c silently dropped --adapter-path, so that serve ran the bare 2-bit build without its DWQ LoRA compensators (fixed in 7937fda, see Install & run). With adapters actually applied, thinking terminated in 5/5 direct-path runs across both teams — at the official temp 1.0 / top_p 1.0 and at 0.7/0.95. Guidance stands regardless: official sampling, generous max_tokens, reasoning_effort="none" for short direct answers — and treat the adapters as part of the deployed F v2 configuration (a bare serve can ramble).
Architecture support:solar_open2 is not yet in PyPI mlx-lm (0.31.3) — use the fork (default branch solar-open2; the adapter-drop fix is 7937fda) or the bundled solar_open2.py.
Calibration coverage saturates at 200 samples (99.8% expert routing), but median tokens/expert (4,314) is ~19% of the 753B reference run; very rare experts may be under-distilled.
Quantized builds inherit the base model's knowledge cutoff (2026-02) and language focus (EN/KO/JA).
License
Distributed under the Upstage Solar License (same as the base model). Per the license's derivative-model terms:
Model name carries the "Solar" prefix ✅
"Built with Solar" attribution displayed above ✅
A copy of the Upstage Solar License is included in this repo ✅
![Solar Open 2 architecture & quantization map — hybrid attention stack [Softmax, Linear×3]×12, MoE 320 routed + 1 shared experts, routers pinned to 8-bit](images/architecture_map.png)
