gemma-4-E2B-it — ONNX / OpenVINO
ONNX and OpenVINO exports of
google/gemma-4-E2B-it, built with
mobius and packaged for
ONNX Runtime GenAI. INT4
variants are quantized with
Olive.
Gemma 4 E2B is an any-to-any multimodal model, so every variant is a
multi-component package with four ONNX models — decoder, embedding,
vision_encoder, audio_encoder — plus a shared genai_config.json,
tokenizer, and image / audio processor configs.
Variants
Each variant folder is <precision>/<target> (or a top-level folder for the
OpenVINO packages). Pick the precision for your quality/size budget and the
target for your runtime/EP.
Precisions
| Precision | Folder prefix | Notes |
|---|
| FP16 | f16/ | Full-precision half. |
| BF16 | bf16/ | BFloat16 (no INT4). |
| INT4 · K-Quant (Q4_K_M) | Q4_K_M/ | Olive K-Quant, block 32. Recommended INT4. |
| INT4 · NF4 | NF4/ | NF4 4-bit weights. |
| INT4 · OpenVINO | openvino-*/ | K-Quant + MatMulNBitsToQDQ, compiled for OpenVINO. |
Targets (execution-provider builds)
| Target (subfolder) | Runtime / EP | Provider in genai_config |
|---|
cuda | ONNX Runtime GenAI + CUDA | cuda (fused GroupQueryAttention) |
default | ONNX Runtime GenAI (CPU / DML) | empty → portable ONNX |
onnx-standard | any conformant ONNX runtime | empty → pure ai.onnx, no com.microsoft ops |
webgpu | ONNX Runtime Web / WebGPU | webgpu |
Full matrix (present in this repo)
| Folder | Precision | Target | Size |
|---|
f16/cuda | FP16 | CUDA | ~11.3 GB |
f16/default | FP16 | CPU / DML | ~11.3 GB |
f16/onnx-standard | FP16 | portable ONNX | ~11.3 GB |
f16/webgpu | FP16 | WebGPU | ~11.3 GB |
bf16/cuda | BF16 | CUDA | ~11.3 GB |
bf16/default | BF16 | CPU / DML | ~11.3 GB |
bf16/onnx-standard | BF16 | portable ONNX | ~11.3 GB |
Q4_K_M/cuda | INT4 (K-Quant) | CUDA | ~8.1 GB |
Q4_K_M/default | INT4 (K-Quant) | CPU / DML | ~8.1 GB |
NF4/cuda | INT4 (NF4) | CUDA | ~7.9 GB |
NF4/default | INT4 (NF4) | CPU / DML | ~7.9 GB |
openvino-epcontext/ | INT4 | OpenVINO EP (ORT) | ~7.4 GB |
openvino-ir/ | INT4 | OpenVINO runtime | ~7.4 GB |
Usage — ONNX Runtime GenAI (the f16 / bf16 / Q4_K_M / NF4 variants)
1pip install onnxruntime-genai # CPU / WebGPU / DML
2pip install onnxruntime-genai-cuda # for the */cuda variants
1import onnxruntime_genai as og
2model = og.Model("Q4_K_M/cuda") # or f16/default, NF4/cuda, ...
3# ... standard ORT GenAI generation loop ...
OpenVINO
Both OpenVINO packages are INT4 and require
OpenVINO ≥ 2026.3 (the release
that ships the opset-24
RMSNormalization /
RotaryEmbedding ONNX-frontend
translators,
openvinotoolkit/openvino#35545).
openvino-epcontext/ — compiled graph for ONNX Runtime ✅ verified
Each component's model.onnx is a single com.microsoft::EPContext node
(source="OpenVINOExecutionProvider") that points at the compiled OpenVINO IR
(model.xml / model.bin) next to it, so ONNX Runtime's OpenVINO EP loads the
precompiled graph instead of recompiling.
Verified end-to-end: the decoder loads via the OpenVINO EP and a forward
pass returns logits (1, seq, 262144) fp16 (all finite), using ONNX Runtime
1.28 built against OpenVINO 2026.3.
1import onnxruntime as ort
2sess = ort.InferenceSession(
3 "openvino-epcontext/decoder/model.onnx",
4 providers=[("OpenVINOExecutionProvider", {"device_type": "NPU"})], # or GPU / CPU
5)
openvino-ir/ — OpenVINO IR
Native OpenVINO IR (openvino_model.xml / .bin) per component:
1import openvino as ov
2decoder = ov.Core().compile_model("openvino-ir/decoder/openvino_model.xml", "NPU")
runtime-wheels/
The exact wheels used to verify the OpenVINO packages (Linux, cp312) — OpenVINO
2026.3 + ONNX Runtime 1.28 with the OpenVINO EP — since no official release
ships OpenVINO ≥ 2026.3 yet:
1pip install runtime-wheels/openvino-2026.3.0-*.whl \
2 runtime-wheels/onnxruntime_openvino-1.28.0-*.whl
Build recipe
Built via
microsoft/olive-recipes google-gemma-4-E2B-it:
MobiusBuilder for the FP16 / BF16 exports;
MobiusBuilder →
OnnxKQuantQuantization for INT4; the OpenVINO packages additionally run
MatMulNBitsToQDQ →
OpenVINOConversion →
OpenVINOEncapsulation.
License
Inherits the
Gemma license.
