Views
No views yet
com.microsoft::MoE op for the Mixture-of-Experts blocks
and serialize weights in the standard ONNX external-data format (model.onnx + model.onnx.data).| Path | Precision | EP | Size (decoder) |
|---|---|---|---|
f16/default/ | fp16 | portable ONNX | ~48 GB |
f16/cuda/ | fp16 | CUDA (with GroupQueryAttention etc.) | ~48 GB |
f16/onnx-standard/ | fp16 | strict ONNX-only (functions inlined) | ~48 GB |
bf16/default/ | bf16 | portable ONNX | ~48 GB |
bf16/cuda/ | bf16 | CUDA | ~48 GB |
bf16/onnx-standard/ | bf16 | strict ONNX-only | ~48 GB |
Q4_K_M/default/ | mixed (4-bit k-quant + fp16) | portable ONNX | ~45 GB |
NF4/default/ | mixed (4-bit NF4 + fp16) | portable ONNX | ~45 GB |
Note on Q4_K_M / NF4 size: Only ~6% size reduction relative to fp16. Most weights in this MoE model are per-expert tensors held as inputs to the fusedcom.microsoft::MoEop rather than asMatMulinitializers, so the standard Olive weight-quantization passes only hit the non-MoE layers (attention, per-layer projections,lm_head). Real MoE compression requires the QMoE pipeline, which is not yet supported by mobius.
com.microsoft::MoE op is invoked with activation_type=swiglu and
non-default attributes that correspond to standard SwiGLU semantics
(y = silu(gate) * up):swiglu_fusion = 1 (interleaved layout)
activation_alpha = 1.0 (no GPT-OSS 1.702 multiplier)
activation_beta = 0.0 (no GPT-OSS "+1" bias on the up branch)
swiglu_limit = inf (no clipping)alpha=1.702, beta=1.0, limit=7.0, interleaved-only) and will produce incorrect output.
Background: microsoft/onnxruntime-genai#2062.1import onnxruntime_genai as og
2
3model = og.Model("path/to/f16/cuda")
4tokenizer = og.Tokenizer(model)
5params = og.GeneratorParams(model)
6params.set_search_options(max_length=200, do_sample=False)
7params.input_ids = tokenizer.encode(
8 "<start_of_turn>user\nHello!<end_of_turn>\n<start_of_turn>model\n"
9)
10print(tokenizer.decode(model.generate(params)[0]))