You can deploy this model and start earning money today!

shailesh83/
gemma-4-31b-it-int4-ov

Name: gemma-4-31b-it-int4-ov
Availability: InStock

openvino

gemma4

Views

No views yet

gemma-4-31b-it-int4-ov

Gemma4-31B-it INT4 OpenVINO — OVMS 2026.3 Deployment (Legacy VLM Pipeline)

This repository contains google/gemma-4-31B-it (dense 31B) exported to OpenVINO IR format with INT4 weight compression, ready to serve via OpenVINO Model Server (OVMS) 2026.3 as an OpenAI-compatible endpoint.

Validated on: Intel Arc Pro B60 (24 GB), Ubuntu 24.04, OVMS 2026.3.0 native CLI binary.

Read this before you deploy: this model must be served with OVMS's legacy VLM pipeline, not VLM_CB (continuous batching / paged attention). VLM_CB produces corrupted output for this model — see Known Challenges for how this was isolated. This is not a quantization bug; the INT4 export itself is clean.

Model Files

File	Size	Description
`openvino_language_model.bin`	~16 GB	INT4 (asymmetric) language model weights
`openvino_vision_embeddings_model.bin`	~551 MB	SigLIP2 vision encoder (unused for text-only inference, present because Gemma 4 is architecturally a VLM)
`openvino_text_embeddings_model.bin`	~1.4 GB	Text embedding model
`openvino_tokenizer.xml/.bin`	~17 MB	OV tokenizer
`openvino_detokenizer.xml/.bin`	~4.3 MB	OV detokenizer
`graph.pbtxt`	~1 KB	OVMS MediaPipe pipeline config (legacy VLM, not VLM_CB)

Quantization parameters: INT4_ASYM, group_size=128, ratio=1.0 (no mixed precision).

Quick Start

Prerequisites

OVMS 2026.3 native CLI binary (tarball, not a Docker image — see below)
Intel GPU with Level Zero / OpenCL drivers installed
At least 18 GB GPU VRAM (Arc Pro B60 or equivalent)
/dev/dri render device accessible
Python 3.12 with the pyovms extension on PYTHONPATH (bundled with the OVMS tarball)

Step 1 — Get OVMS 2026.3

bash

mkdir -p ~/ovms_2026.3
wget https://storage.openvinotoolkit.org/repositories/openvino_model_server/packages/weekly/2026.3.0.39717a69b/ovms_ubuntu24_2026.3.0_python_on.tar.gz
tar -xzf ovms_ubuntu24_2026.3.0_python_on.tar.gz -C ~/ovms_2026.3
~/ovms_2026.3/ovms/bin/ovms --version

This is the standalone binary release, not a docker save archive — don't docker load it.

Step 2 — Download the model

bash

mkdir -p ~/ovms_models
cd ~/ovms_models
pip install -U "huggingface_hub[hf_xet]" hf_xet
export HF_XET_HIGH_PERFORMANCE=1
export HF_XET_NUM_CONCURRENT_RANGE_GETS=32

hf download shailesh83/gemma-4-31b-it-int4-ov \
    --local-dir google/gemma-4-31B-it \
    --exclude "*.md" ".gitattributes"

Step 3 — Create config.json

bash

cat > config.json << 'EOF'
{
    "model_config_list": [
        {
            "config": {
                "name": "google/gemma-4-31B-it",
                "base_path": "google/gemma-4-31B-it"
            }
        }
    ]
}
EOF

Step 4 — Start OVMS (native CLI, GPU)

bash

export PYTHONPATH=~/ovms_2026.3/ovms/lib/python:$PYTHONPATH

~/ovms_2026.3/ovms/bin/ovms \
  --config_path ~/ovms_models/config.json \
  --rest_port 9001 \
  --allowed_local_media_path ~/ovms_models \
  --allowed_media_domains all \
  --log_level INFO

The PYTHONPATH export is required — without it, OVMS's Python-Mediapipe backend can't find its bundled pyovms extension module and the server will not come up.

CPU-only fallback: change device: "GPU" to device: "CPU" in graph.pbtxt, then drop --device/--group-add if you're also using Docker.

Step 5 — Wait for the model to load

curl -s http://localhost:9001/v3/models

Wait for google/gemma-4-31B-it to show up in the response.

Testing

Text-only inference

bash

curl http://localhost:9001/v3/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-31B-it",
    "messages": [{"role": "user", "content": "Explain how a transformer attention mechanism works, step by step."}],
    "max_tokens": 400,
    "temperature": 0.1
  }'

Streaming

bash

curl http://localhost:9001/v3/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-31B-it",
    "messages": [{"role": "user", "content": "Explain photosynthesis briefly."}],
    "max_tokens": 200,
    "stream": true
  }'

Automated benchmark + corruption check

A benchmark script is included in this repo (benchmark_gemma4.py) that runs a fixed set of prompts, reports tokens/sec, and flags known corruption signatures (useful as a regression check after any OVMS/optimum-intel upgrade):

bash

python benchmark_gemma4.py \
  --url http://localhost:9001/v3/chat/completions \
  --model google/gemma-4-31B-it \
  --runs 3

Benchmark Results

Single Intel Arc Pro B60, legacy VLM pipeline, single-stream, temperature=0.1. 4 prompt types x 3 runs = 12 requests.

Test	Max Tokens	Avg Elapsed (s)	Avg Tok/s	Result
attention_mechanism	400	20.41	19.60	3/3 clean
backprop_long	800	41.20	19.42	3/3 clean
math_check	100	5.25	19.04	3/3 clean
summary	300	15.27	19.64	3/3 clean

12/12 clean, 0 corrupted, 0 failed. Average 19.42 tok/s (range 18.84–19.66).

Known Challenges

This deployment went through a fairly involved debugging process. Summarized here so you don't have to repeat it:

INT4 corruption (red herring). Early attempts at exporting this model with INT4 compression (various group-size/ratio/symmetric combinations, across multiple optimum-intel branches) produced output that looked corrupted — garbled sub-tokens collapsing into repeated-token loops after roughly 100 tokens of otherwise coherent generation. This pattern held across CPU and GPU, across export methods, and across several quantization recipes, strongly suggesting an INT4-specific bug.
The actual root cause: OVMS's continuous-batching pipeline, not quantization. The same INT4 export produces clean output when tested directly via optimum-intel in Python and clean output when served through OVMS's legacy VLM pipeline — but corrupts when served through VLM_CB (continuous batching / paged attention), even with identical weights. This isolates the bug to VLM_CB's interaction with Gemma 4's hybrid attention head dimensions (256 sliding-window / 512 global) — likely a similar issue to one that required mainline vLLM to force a specific attention backend for this same model.
Gemma 4 is registered strictly as a VLM task, even for text-only use — exports must use --task image-text-to-text (direct optimum-cli) or --pipeline_type VLM/VLM_CB (export_model.py); text-generation-with-past is rejected outright.
Dependency conflicts are the biggest time sink. optimum-intel's Gemma 4 support currently lives on a fork (rkazants/optimum-intel@support_gemma_4), not mainline, and it has a narrow compatible window with transformers/safetensors/optimum-onnx versions. Installing OVMS's own export_model.py requirements.txt will silently install a different optimum-intel on top of the fork. Install requirements.txt first (full dependency resolution), then the fork second, unconstrained — this way the fork wins the conflict and pulls in compatible dependency versions.
OVMS's native GGUF loader does not support this model. If you already have a GGUF quant of Gemma 4 (e.g. from Unsloth), OVMS's built-in --pull --gguf_filename path will reject it — its recognized-architecture list includes gemma3 but not gemma4 as of this OVMS version. Use the IR export path in this repo instead.

Export Command (for reference)

bash

git clone https://github.com/openvinotoolkit/model_server.git
cd model_server/demos/common/export_models
pip install -r requirements.txt

pip install "git+https://github.com/rkazants/optimum-intel.git@support_gemma_4" \
    --extra-index-url https://download.pytorch.org/whl/cpu

python export_model.py text_generation \
    --source_model google/gemma-4-31B-it \
    --pipeline_type VLM \
    --weight-format int4 \
    --extra_quantization_params "--group-size 128 --ratio 1.0" \
    --target_device GPU \
    --model_repository_path ./ovms_models \
    --config_file_path ./ovms_models/config.json

Do not substitute --pipeline_type VLM_CB here — see Known Challenges.

Troubleshooting

Symptom	Cause	Fix
`ModuleNotFoundError: No module named 'pyovms'`	OVMS's Python-Mediapipe backend can't find its bundled extension module.	`export PYTHONPATH=<ovms_dir>/ovms/lib/python:$PYTHONPATH` before starting `ovms`.
`KeyError: 'gemma4'` in transformers	Active transformers version doesn't recognize the `gemma4` model_type.	Install a transformers version with gemma4 support (5.13.0 confirmed working).
`ValueError: ... only supports the tasks image-text-to-text for gemma4`	This optimum-intel branch registers Gemma 4 strictly as a VLM task.	Use `--task image-text-to-text` or `--pipeline_type VLM`/`VLM_CB`, never `text-generation-with-past`.
`ModuleNotFoundError: No module named 'optimum.exporters.onnx'`	The `optimum-onnx` package got uninstalled or never installed.	`pip install --no-deps optimum-onnx`
`ImportError: safetensors>=0.8.0 is required ... found safetensors==0.7.0`	Version constraint wasn't quoted in shell, or `--no-deps` skipped the bump.	`pip install "safetensors>=0.8.0"` — quote it, or `>` gets parsed as a shell redirect.
Clean output via raw Python, corrupted via OVMS serving	`VLM_CB` pipeline bug with this model's hybrid attention heads.	Export and serve with `--pipeline_type VLM` (legacy) instead.
`Invalid request URL`	Using `/v1/` instead of `/v3/`.	Change URL to `/v3/chat/completions`.

Hardware Requirements

Component	Minimum	Tested
GPU VRAM	18 GB	Intel Arc Pro B60 (24 GB)
System RAM	32 GB	256 GB
Storage	25 GB free	NVMe SSD
OS	Ubuntu 22.04+	Ubuntu 24.04
Driver	Intel GPU driver with Level Zero	Latest production

Known Limitations

No continuous batching. The legacy VLM pipeline serves one request at a time; this configuration is validated for single-request serving only. Concurrent throughput is unvalidated.
VLM_CB should not be used with this model until the underlying paged-attention issue with Gemma 4's hybrid head dimensions is fixed upstream in OpenVINO GenAI.
Vision tower is present (SigLIP2) but unexercised by the text-only testing in this repo — image+text inference is architecturally supported but not validated here.

License

Gemma model weights are subject to the Gemma Terms of Use. This repository contains only the OpenVINO-converted weights with no modifications to model parameters.