~103B-A8B hybrid MoE — 30 GB on disk (down from the 200 GB bf16 source) —
mixed-precision JANGTQ2 quantization on inclusionAI's Bailing-V2.5 hybrid
architecture. Runs on a 64 GB+ Apple Silicon Mac.
Quantization: JANGTQ2 — 2-bit MXTQ codebook (Hadamard-rotated,
Lloyd-Max optimized) on routed-expert weights + 8-bit affine on every
attention path (MLA q_a/q_b/kv_a/kv_b/dense, Linear-Attn
query_key_value/g_proj/dense) + 8-bit affine on shared experts /
layer-0 dense MLP / MTP eh_proj / embed / lm_head + fp16 passthrough on
every RMSNorm / GroupRMSNorm / router gate / expert_bias / slope
Bundle size:30 GB on-disk across 31 shards (24,576 routed-expert
tensors at 2-bit + 211 affine 8-bit + 228 passthrough)
Runs on: M2 Pro 64 GB / M3 Max 64 GB+ / M4 Max 128 GB / M5 Max 128 GB
Architecture (bailing_hybrid)
This is a hybrid attention model — every 8th layer is full softmax MLA,
the other 28 layers (of 32) are Lightning-Linear-Attention (Gated Linear
Attention with per-head learned slopes). Plus a single Multi-Token
Prediction head for speculative decoding.
Layer block
Count
Attention
MLP
Layer 0
1
Linear (GLA)
Dense MLP (intermediate=9216)
Layers 1–6, 8–14, 16–22, 24–30
27
Linear (GLA)
MoE (256+1)
Layers 7, 15, 23, 31
4
MLA (full softmax)
MoE (256+1)
MTP head (32)
1
MLA
MoE (256+1)
MLA: DeepSeek-V2-flavored — q_lora_rank=1536, kv_lora_rank=512,
qk_nope_head_dim=128, qk_rope_head_dim=64 (interleaved, theta=6M),
v_head_dim=128. Materializes K and V (no absorb optimization).
Linear attention: Lightning-Attention-2 / Gated Linear Attention — fused
query_key_value projection, per-head RMSNorm on Q/K, partial RoPE (50% of
head_dim, adjacent-half rotation), per-head ALiBi-style slope scaled by
layer_idx, sigmoid-gated output through g_proj + GroupRMSNorm (groups=4).
MoE routing: sigmoid scoring + grouped routing (8 groups × top-4 groups
top-8 within), e_score_correction_bias for noaux_tc — selection uses
biased scores, weighting uses original sigmoid scores. Identical math to
DeepSeek-V3 / Kimi-K2.6.
The bundle ships configuration_bailing_moe_v2_5.py and
modeling_bailing_moe_v2_5.py for HF compatibility, plus an mlx_lm
model class (bailing_hybrid.py) that handles the hybrid attention
dispatch, MLA, GLA recurrence, MoE routing, and MTP.
Reasoning + tools
The chat template defaults to detailed thinking off. To enable
chain-of-thought:
The model emits <think>...</think> blocks before its answer when thinking
is on; runtimes binding to the deepseek_r1 reasoning parser will extract
them automatically. Tool calls follow the DeepSeek tool-call block format.
Credits
Quantization + MLX runtime: Jinho Jang (eric@jangq.ai)
