Officially, this model does not run on OpenVINO. Its architecture (gemma4_unified) is unsupported by the exporter (optimum-intel #1764) and rejected by the GenAI runtime ("Unsupported VLM model type"). This repo is it running anyway: needle retrieval verified at 40,000 tokens of context on a single Intel Arc card, at roughly 55 tok/s decode. And as of the latest update it is no longer text-only: vision and audio both work through the custom pipeline shipped in this repo (av_pipeline.py), which as far as I can find makes this the first gemma4_unified anywhere with sight and hearing on OpenVINO.

It took four stacked fixes, each useless without the others. All of them are baked into this repo already, so you only need two things at runtime (see How to run):

The model_type is spoofed from gemma4_unified to gemma4. The runtime rejects the original name with a literal string comparison, but the 12B text graph is the 26B's graph minus one input, so the gemma4 pipeline drives it correctly.
The RoPE lookup-table graph patch (fp16 rope execution collapses this model from about 8K; see the toolkit repo for the full writeup). Tables ship to 65K positions, p-RoPE zeros preserved.
You must run the OpenVINO GenAI NIGHTLY (2026.3-dev). The 2026.2 release decodes this graph wrong (first token fine, then garbage). This is a runtime bug that nightly already fixed.
For text-only VLMPipeline, pass DYNAMIC_QUANTIZATION_GROUP_SIZE: 128. An earlier default group size garbled long context, so this card previously required DQGS 0. The specific 128 setting has now passed matched coherence tests at 6,620 and 30,000 input tokens with the exact same output as DQGS 0. The separate vision/audio paths remain validated with DQGS 0.

This is the abliterated ("heretic") Gemma-4 12B, refusals removed, a general instruction model. I run it for roleplay (Skyrim companion AI, Discord character bots), and it is the little sibling of my 26B MoE and 31B: at 7.5 GB it is the one that fits 12 GB and 16 GB Arc cards (B580, A770 class), not just the big workstation parts.

Toolkit: patches, OpenAI-compatible server, quickstart, and the full bug catalog live at github.com/Wondernuttz/OpenVino-For-Gemma-4.

Measured performance (single Arc Pro B70, OpenVINO GenAI 2026.3 nightly)

Metric	Value
Decode, short context	56.1 tok/s
Decode at 6K context	~26 tok/s
Prefill, 512 tokens	2,808 tok/s
Prefill, 2K tokens	3,835 tok/s
Prefill, 6K tokens	2,956 tok/s
Prefill, 30K coherence prompt	851 tok/s
Model load	~6-7 s with a warm OpenVINO cache
Weights	7.5 GB

The main Intel Arc B70 benchmark repo (PMZFX) does not list a 12B, so there is no published same-card figure to compare against. As far as I can find these are the only OpenVINO numbers for this model anywhere, since the toolchain officially cannot produce them.

Matched DQGS sweep

Fresh process per shape, one Arc Pro B70, deterministic generation, text-only VLMPipeline.

Input	DQGS 0	DQGS 128	Gain
512	2,118 tok/s	2,808 tok/s	+32.6%
2,048	2,957 tok/s	3,835 tok/s	+29.7%
6,144	2,406 tok/s	2,956 tok/s	+22.9%
6,620 coherence	2,310 tok/s	2,754 tok/s	+19.2%
30,000 coherence	805 tok/s	851 tok/s	+5.7%

Short decode stayed at 56 tok/s. Both 6,620 and 30,000 token retrieval tests passed all four checks and produced byte-identical answers between DQGS 0 and 128. The smaller gain at 30K is expected because attention, not the INT4 projection work, dominates that far into context.

Long-context capability, verified by needle retrieval

A password fact planted early in the prompt, retrieved at the end. Pass means the exact password.

Context	Thinking OFF	Thinking ON
4K	PASS	not tested
8K	PASS	PASS (12.9 s, clean structured reasoning)
16K	PASS	PASS (26.9 s)
24K	NULL	not tested
32K	NULL (60 s)	not tested
40K	NULL (104 s)	not tested

48K and beyond hit host-RAM limits on my test box, not a model limit. The shipped position tables go to 65K.

Vision and audio, working (an OpenVINO first)

The base model is encoder-free multimodal: no vision tower, no audio tower. Images become up to 280 soft tokens through a small projection stack, and audio is raw 16 kHz waveform chopped into 640-sample frames (40 ms per token) fed through an RMSNorm and one Linear layer. That design is why this works at all: the exported vision IR was in this repo from day one, and the entire audio encoder is a single 5 MB matrix (audio_projection.npy, extracted from the heretic checkpoint) that this repo now ships.

What GenAI gets wrong is preprocessing: its gemma4 pipeline patchifies at 16x16 (768 values per patch) where the unified tower wants 16x16 patches merged 3x3 into 6912-wide model patches. av_pipeline.py does the correct preprocessing with the real Gemma4UnifiedImageProcessor from transformers main, runs the vision IR, splices the soft tokens into the embedding stream, and drives the language model with a manual stateful loop. Audio needs no IR at all: two lines of numpy reimplement the projection exactly (the RMSNorm has no learned scale, so waveform normalization cancels out).

Verified results, single Arc Pro B70:

Test	Result
Solid red square, "what color?"	"Red"
Synthetic scene (sky, grass, sun)	Correct two-sentence description with correct spatial layout
Real 2752x1536 game cover art	Detailed, accurate description: both characters, clothing, pose, the glowing shield, the palette
6 s TTS clip	Transcribed VERBATIM, every word
18 s real microphone recording	Transcribed word for word; the speaker verified the transcript as perfect

Speed: multimodal vs text-only, and the two ways to run it

There are two working AV paths, plus the text-only baseline. All numbers measured on the same single Arc Pro B70, cache-clean single-stream runs:

Metric	Text-only DQGS 0 (comparison run)	Native C++ AV (toolkit patch)	Python (`av_pipeline.py`)
Prefill 512	2,301 tok/s (0.22 s)	0.63 s TTFT	0.40 s TTFT
Prefill 2K	3,170 tok/s (0.65 s)	0.93 s (~2,200 tok/s)	1.18 s
Prefill 6K	2,120 tok/s (2.9 s)	2.09 s (~2,940 tok/s)	3.0-5.4 s
Image request, end to end	n/a	0.73 s to first token	~0.44 s
18 s audio request, end to end	n/a	0.62 s to first token	~0.39 s
Decode, short context	~55 tok/s	~49 tok/s	50-54 tok/s
Decode at 6K context	~26 tok/s	16.8 tok/s (12.1 with an image in context)	25.9 tok/s

Why they differ: the architecture is encoder-free, so multimodality itself costs almost nothing; an image is simply 264 extra context tokens and 18 s of audio is 458, and the Python pipeline's decode at 6K (25.9 tok/s) exactly matches the text-only figure (26). The native path pays a fixed ~0.4-0.5 s first-request initialization on short prompts, wins time-to-first-token at depth, but its decode at deep context currently runs slower (16.8 vs 26 tok/s; per-step overhead in GenAI's unified branch, measured honestly, cause not yet isolated). The Python path is quickest to first token on short requests and holds full decode speed at depth, but its deep prefill is slower because it feeds f32 embeddings from host RAM every request.

Rule of thumb as of today: the Python pipeline is the best all-rounder; take the native patch when you want one C++ process with GenAI's serving machinery, or the fastest first token on long prompts.

Way 1: Python pipeline (no build, files in this repo)

bash

python av_pipeline.py --model-dir ./gemma4-12b-heretic-ov --gate 2 --image photo.jpg
python av_pipeline.py --model-dir ./gemma4-12b-heretic-ov --gate 4 --audio speech.wav

Audio input must be 16 kHz mono WAV. The script needs torch+torchvision (CPU build is fine, they only do preprocessing) and the gemma4_unified processor module from transformers main; if your installed transformers predates it, copy image_processing_gemma4_unified.py from the transformers GitHub repo into transformers/models/gemma4_unified/ with an empty __init__.py.

Way 2: native C++ (patched OpenVINO GenAI, full walkthrough in the toolkit repo)

bash

git clone --recurse-submodules https://github.com/openvinotoolkit/openvino.genai
cd openvino.genai
git apply gemma4-unified-audio.patch   # from the toolkit repo: adds audio input support
cmake -DCMAKE_BUILD_TYPE=Release -S . -B build && cmake --build build -j

Then generate openvino_audio_embeddings_model.xml with the toolkit's make_audio_ir.py (it wraps this repo's audio_projection.npy), and mind the three usage rules that make or break native vision: model_type must be the real gemma4_unified (NOT the gemma4 spoof; the spoofed path crashes with images), DYNAMIC_QUANTIZATION_GROUP_SIZE: 0 for this AV path, and the prompt must contain <|image|> (or <|audio|>) where the media belongs; a missing tag makes GenAI prepend the block before <bos>, which scrambles color and identity binding while shapes still work (a solid red square answers "Green"). Waveforms go in as f32 tensors shaped [nsamples, 1, 1] through the ordinary images API.

Honest limitations

Vision and audio need the bundled Python pipeline. OpenVINO GenAI's VLMPipeline still cannot drive this model's multimodal side (wrong patchify, no audio path), so av_pipeline.py is the way in. Decode through it is ~95% of the C++ pipeline; text-only prefill is slower, so use VLMPipeline for pure text serving.
Nightly runtime required until OpenVINO GenAI 2026.3 releases. pip install --pre --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly openvino openvino-tokenizers openvino-genai
Use DYNAMIC_QUANTIZATION_GROUP_SIZE: 128 for the text-only pipeline. Keep DQGS 0 for the bundled Python and native vision/audio paths until those paths receive the same sweep.

How to run

bash

pip install --pre --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly openvino openvino-tokenizers openvino-genai
pip install huggingface_hub
huggingface-cli download Wondernutts/gemma-4-12B-it-qat-q4_0-unquantized-uncensored-heretic-int4-ov --local-dir ./gemma4-12b-heretic-ov

python

import openvino_genai as g

pipe = g.VLMPipeline("./gemma4-12b-heretic-ov", "GPU",
                     **{"DYNAMIC_QUANTIZATION_GROUP_SIZE": 128})

def chat(system, user, think=False):
    p = "<bos><|turn>system\n" + system + ("\n<|think|>" if think else "") + "<turn|>\n"
    p += "<|turn>user\n" + user + "<turn|>\n<|turn>model\n"
    if not think:
        p += "<|channel>thought\n<channel|>"   # pre-closed thought channel = fast direct replies
    c = g.GenerationConfig()
    c.max_new_tokens = 512 if not think else 1536
    c.do_sample = True; c.temperature = 0.9; c.top_p = 0.95
    try: c.repetition_penalty = 1.2
    except Exception: pass
    try: c.apply_chat_template = False
    except Exception: pass
    return str(pipe.generate(p, generation_config=c))

print(chat("You are Lydia, housecarl to the Dragonborn. Dry wit, fiercely loyal.",
           "We have been walking this frozen pass for six hours. Say something."))

Same rules as the siblings: <|turn> prompt format (not classic Gemma), repetition penalty around 1.2, never use JSON grammar mode (google-deepmind/gemma#622), thinking needs at least 1024 max_new_tokens.

Provenance

google/gemma-4-12B-it (QAT q4_0 unquantized), heretic abliteration by llmfan46, OpenVINO INT4 export, then the model-type spoof and RoPE lookup-table patch (this repo).

Intended use and content notice

Uncensored general model, built and tested for roleplay and creative writing on local Intel hardware. The abliteration removes refusal behavior and outputs are unfiltered; you are responsible for lawful and appropriate use. Licensed under Apache 2.0, same as the upstream Gemma 4 release.