intel_google-gemma-3-12b-it-int4
- Model creator: google
- Original model: google-gemma-3-12b-it
This repository contains the Gemma-3-12b-it model optimized for Intel hardware using the OpenVINO™ Toolkit and quantized to INT4 precision.
It is designed for high-performance inference on edge devices, including Intel Core Ultra (iGPU), Xeon, and Intel Arc graphics.
Model Details
- Developed by: Advantech-EIOT
- Architecture: Gemma-3 (12B)
- Task: Text Generation (Chat/Instruction)
- Precision: INT4 (Weight Compression)
- Optimization: OpenVINO™ Toolkit
Deployment with OpenVINO Model Server (OVMS)
OpenVINO Model Server (OVMS) provides a high-performance, scalable solution for serving this model via OpenAI-compatible APIs.
1. Prerequisite: Verify Model Files
Before launching the server, ensure your local directory contains the following OpenVINO IR files:
openvino_model.xml (Model topology)openvino_model.bin (Model weights)tokenizer_config.json and related tokenizer files
2. Launch with Docker
Use the official OVMS container to serve the 12B model. Replace $(pwd) with the path to your model directory.
1docker run -d --rm -p 8000:8000 \
2 -v $(pwd):/workspace/model:ro \
3 openvino/model_server:latest \
4 --rest_port 8000 \
5 --model_path /workspace/model \
6 --model_name gemma-3-12b-it \
7 --plugin_config '{"PERFORMANCE_HINT": "LATENCY"}'
3. Usage via OpenAI API
Once the server is running, you can interact with it using the standard OpenAI Python client:
1from openai import OpenAI
2
3# Initialize client pointing to your OVMS instance
4client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
5
6response = client.chat.completions.create(
7 model="gemma-3-12b-it",
8 messages=[{"role": "user", "content": "Explain the advantages of INT4 quantization for edge AI."}]
9)
10
11print(response.choices[0].message.content)
Hardware Compatibility
This 12B INT4 model is highly optimized for:
- Intel Core Ultra (CPU/iGPU): Ideal for local AI PC deployments.
- Advantech Edge AI Platforms: Such as those used in industrial or IoT environments.
- Intel Xeon Scalable Processors: Efficient for high-throughput inference.
- Intel Arc Discrete Graphics: Accelerated LLM performance.
Limitations and Disclaimer
Gemma-3 is a powerful model but may exhibit hallucinations.
Users should validate outputs for critical applications.
Please refer to the Google Gemma License for usage restrictions.
