This repository contains Gemma4-26B-A4B exported to OpenVINO IR format with INT4 weight compression, ready to serve via OpenVINO Model Server (OVMS) 2026.2 as an OpenAI-compatible VLM endpoint.

Validated on: Intel Arc Pro B60 (2× 48 GB), Ubuntu 24.04, OVMS 2026.2-gpu Docker image.

Model Files

File	Size	Description
`openvino_language_model.bin`	~13 GB	INT4 language model weights
`openvino_vision_embeddings_model.bin`	~547 MB	SigLIP2 vision encoder
`openvino_text_embeddings_model.bin`	~705 MB	Text embedding model
`openvino_tokenizer.xml/.bin`	~16 MB	OV tokenizer
`openvino_detokenizer.xml/.bin`	~4 MB	OV detokenizer
`graph.pbtxt`	~1 KB	OVMS MediaPipe VLM pipeline config

Quick Start

Prerequisites

Docker with GPU support
Intel GPU with OpenCL/Level Zero drivers installed
At least 16 GB GPU VRAM (Arc Pro B60 or equivalent)
/dev/dri render device accessible

Step 1 — Download the model

bash

mkdir -p ~/ovms_models
cd ~/ovms_models

huggingface-cli download spandey2/gemma-4-26b-a4b-int4-ov \
    --local-dir google/gemma-4-26B-A4B \
    --exclude "*.md" ".gitattributes"

Step 2 — Create config.json

bash

# Single model (Gemma4 only)
cat > config.json << 'EOF'
{
    "model_config_list": [],
    "mediapipe_config_list": [
        {
            "name": "gemma4-26b",
            "base_path": "google/gemma-4-26B-A4B"
        }
    ]
}
EOF

For two models in one server (Gemma4 + GPT-OSS-20B, selectable from OpenWebUI):

bash

cat > config.json << 'EOF'
{
    "model_config_list": [],
    "mediapipe_config_list": [
        {
            "name": "gemma4-26b",
            "base_path": "google/gemma-4-26B-A4B"
        },
        {
            "name": "gpt-oss-20b",
            "base_path": "gpt-oss-20b"
        }
    ]
}
EOF

Note: For the dual-model setup, export gpt-oss-20b separately using export_model.py --pipeline_type LM and place it under ~/ovms_models/gpt-oss-20b/.

Step 3 — Start OVMS with Docker (GPU)

bash

cd ~/ovms_models

docker run -d -p 9001:9001 --rm \
  --name ovms-gemma4 \
  --user $(id -u):$(id -g) \
  -v $(pwd):/models:rw \
  --device /dev/dri \
  --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) \
  openvino/model_server:2026.2-gpu \
  --config_path /models/config.json \
  --rest_port 9001 \
  --allowed_local_media_path /models \
  --allowed_media_domains all \
  --log_level INFO

CPU-only fallback (no --device /dev/dri, no --group-add): Change device: "GPU" to device: "CPU" in google/gemma-4-26B-A4B/graph.pbtxt, then remove the --device and --group-add flags.

Step 4 — Wait for model to load (~20–30 seconds)

bash

docker logs -f ovms-gemma4
# Wait for: Mediapipe: gemma4-26b state changed to: AVAILABLE

Or poll the status endpoint:

watch -n 5 'curl -s http://localhost:9001/v3/models | python3 -m json.tool'

Testing

Text-only inference

bash

curl http://localhost:9001/v3/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4-26b",
    "messages": [{"role": "user", "content": "What is 7 times 8?"}],
    "max_tokens": 50,
    "temperature": 0.1,
    "skip_special_tokens": true
  }'

Expected response:

{"choices":[{"finish_reason":"stop","index":0,"message":{"content":"56","role":"assistant",...}}],...}

Image + text inference (VLM)

Important: The image must be placed inside the mounted /models directory so the container can read it. Use image_url first in the content array, followed by text. Do not include <image> tokens in the text.

bash

# Copy your image into the models directory (container maps /models → ovms_models/)
cp /path/to/your/image.jpg ~/ovms_models/test_image.jpg

curl http://localhost:9001/v3/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4-26b",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "/models/test_image.jpg"}},
        {"type": "text", "text": "Respond in English only. What is in this image?"}
      ]
    }],
    "max_tokens": 150,
    "temperature": 0.1
  }'

Expected response:

{"choices":[{"finish_reason":"stop","index":0,"message":{"content":"A cat.\n","role":"assistant",...}}],...}

Note: The response may contain a trailing \n. Strip it client-side: content = response["choices"][0]["message"]["content"].strip()

Base64 image

bash

IMAGE_B64=$(base64 -w0 /path/to/image.jpg)

# image_url first, then text — same ordering rule applies
curl http://localhost:9001/v3/chat/completions \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"gemma4-26b\",
    \"messages\": [{
      \"role\": \"user\",
      \"content\": [
        {\"type\": \"image_url\", \"image_url\": {\"url\": \"data:image/jpeg;base64,${IMAGE_B64}\"}},
        {\"type\": \"text\", \"text\": \"Respond in English only. Describe this image.\"}
      ]
    }],
    \"max_tokens\": 150,
    \"temperature\": 0.1
  }"

Streaming

bash

curl http://localhost:9001/v3/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4-26b",
    "messages": [{"role": "user", "content": "Explain photosynthesis briefly."}],
    "max_tokens": 200,
    "stream": true,
    "skip_special_tokens": true
  }'

List available models

bash

curl http://localhost:9001/v3/models
# Returns: {"data":[{"id":"gemma4-26b","object":"model",...}],...}

Connect to OpenWebUI

OpenWebUI can use this server directly as an OpenAI-compatible backend.

Open OpenWebUI → Settings → Connections → OpenAI API
Set URL: http://<your-server-ip>:9001/v3
Set API Key: ovms (any non-empty string)
Click Save

Both gemma4-26b and gpt-oss-20b (if configured) will appear in the model dropdown.

Important: Use /v3 not /v1 — OVMS 2026.2 LLM/VLM endpoints are under /v3/.

For image uploads via OpenWebUI to work, OVMS receives the image as base64 in the request body — this works automatically when you attach an image in the chat interface.

Export Command (for reference)

This model was exported using:

bash

# Clone OVMS export tools
git clone --branch releases/2026/2 \
    https://github.com/openvinotoolkit/model_server.git
cd model_server/demos/common/export_models

pip install -r requirements.txt

python export_model.py text_generation \
    --source_model google/gemma-4-26B-A4B \
    --pipeline_type VLM \
    --weight-format int4 \
    --target_device GPU \
    --model_repository_path ./ovms_models \
    --config_file_path ./ovms_models/config.json

Known Limitations (VLM Legacy Pipeline)

Limitation	Details	Workaround
`<end_of_turn>` in output	VLM Legacy pipeline does not strip EOS turn tokens	`content.strip()` removes trailing `\n`; ignore `<end_of_turn>` if present
`skip_special_tokens` ignored	Not supported in stateful/legacy VLM pipeline	Strip client-side
Image must be first in content array	Text-first ordering causes image to be ignored	Always put `image_url` before `text` in content array
No `<image>` token in text	Adding `<image>\n` causes language drift	Use plain English text prompt only
Single-turn only for image	Multi-turn conversations with image history unreliable	Send image fresh in each request
`gpt-oss-20b` needs separate export	Model files not included in this repo	Export separately with `export_model.py --pipeline_type LM`

Troubleshooting

Symptom	Cause	Fix
`Invalid request URL`	Using `/v1/` instead of `/v3/`	Change URL to `/v3/chat/completions`
`Chat template was not found`	Missing `chat_template` in `tokenizer_config.json`	Already fixed in this repo
`Image parsing failed`	Corrupted image or wrong path	Ensure image is inside mounted `/models` dir
`Given url does not match`	External URL without `--allowed_media_domains`	Add `--allowed_media_domains all` to docker run
Container exits immediately	conda env polluting `PYTHONPATH`	Run `conda deactivate` before docker commands
`Model with requested name is not found` on `/v1/models`	MediaPipe servables don't list at `/v1/models`	Use `/v3/models` instead
Port connection refused	Port mismatch between `-p` and `--rest_port`	Both must use the same port number

Hardware Requirements

Component	Minimum	Tested
GPU VRAM	14 GB	2× Intel Arc Pro B60 (48 GB each)
System RAM	32 GB	256 GB
Storage	20 GB free	NVMe SSD
OS	Ubuntu 22.04+	Ubuntu 24.04
Driver	Intel GPU driver with Level Zero	Latest production

License

Gemma model weights are subject to the Gemma Terms of Use. This repository contains only the OpenVINO-converted weights with no modifications to model parameters.

shailesh83/gemma-4-26b-a4b-int4-ov

Gemma4-26B-A4B INT4 OpenVINO — OVMS 2026.2 Deployment

Model Files

Quick Start

Prerequisites

Step 1 — Download the model

Step 2 — Create config.json

Step 3 — Start OVMS with Docker (GPU)

Step 4 — Wait for model to load (~20–30 seconds)

Testing

Text-only inference

Image + text inference (VLM)

Base64 image

Streaming

List available models

Connect to OpenWebUI

Export Command (for reference)

Known Limitations (VLM Legacy Pipeline)

Troubleshooting

Hardware Requirements

License

shailesh83/
gemma-4-26b-a4b-int4-ov