OpenVINO INT8 export of google/gemma-4-E4B-it, re-exported so that prefix (KV) caching works.

Intel's published export, OpenVINO/gemma-4-E4B-it-int8-ov, cannot use openvino_genai's continuous-batching backend, so it gets no prefix caching at all:

No ScaledDotProductAttention operation observed in the graph,
cannot perform the SDPAToPagedAttention transformation.

That backend is built by rewriting fused ScaledDotProductAttention nodes into PagedAttention. Intel's language model was traced with decomposed attention, so there is nothing to rewrite. The tell is static — no GPU, no model load:

Its sibling exports (gemma-4-E2B-it-int4-ov, gemma-4-26b-a4b-it-int4-ov) are fine, so this is a per-build defect rather than anything about Gemma 4. optimum-intel pins the attention implementation only for models listed in FORCE_ATTN_MODEL_CLASSES (phi3_v, gemma2, llama4); gemma4 is not among them, so the export environment decides.

Quality is unchanged. Against Intel's export on an identical probe set (OCR of small text incl. an arbitrary serial, a 3x3 letter-grid read, object counting, digits, colours/shapes, a text control), this export answered 7/7 byte-identically — including both of Intel's wrong answers. It adds a capability; it changes nothing else. Same 7.8 GB, same 42 layers, same 84 KB/token KV, same INT8 asymmetric channel-wise weights.

What you gain, measured on an Intel Arc Pro B60 (24 GB, OpenVINO 2026.3 release) with a ~7.9k-token repeated prefix — the shape of an agent's fixed system prompt:

Note the honest shape: Intel's export is faster cold, because the continuous-batching path prefills more slowly. This one pays a ~3.3 s premium once and is then ~2.6x faster per turn, breaking even by turn 3. If your client prewarms, the cold turn is paid before the first request and every user-visible turn is the fast one. For single-shot vision, Intel's export is the better choice; for agent loops, this one.

Recorded in the IR: OpenVINO 2026.3.0-22451, optimum-intel 2.2.0.dev0+e1de5be, transformers 5.5.4. Export took ~3 minutes once the weights were local.

One gotcha if you reproduce this. The chat template is baked into openvino_tokenizer.xml at export time, so editing chat_template.jinja afterwards does nothing. Google's template uses Python-style implicit string concatenation inside raise_exception(...) — valid in Jinja2, rejected by openvino_genai's C++ Jinja parser with "Expected closing parenthesis in call args" at warmup. Replace the template in the source directory before exporting. The template here is the runtime-parseable one from Intel's export, with thanks.

Served here through NoLlama, where prefix caching is on by default for GPU/CPU VLM slots.

Gemma. Use of this model is governed by the Gemma Terms of Use. This repository redistributes a quantized conversion of google/gemma-4-E4B-it; all rights and restrictions of the original apply unchanged.