You can deploy this model and start earning money today!

CelesteImperia/
Gemma-2-2B-IT-OpenVINO-INT4

Name: Gemma-2-2B-IT-OpenVINO-INT4
Availability: InStock

Views

No views yet

Gemma-2-2B-IT-OpenVINO-INT4

This repository contains an optimized OpenVINO™ IR version of Google's Gemma-2-2B-IT, quantized to INT4 precision using NNCF. It is designed for high-efficiency local inference on Intel hardware.

🐍 Usage with Optimum-Intel

To run this model, ensure you have the required dependencies installed: bash pip install "optimum-intel[openvino,nncf]" transformers

Python Inference Example

python

from optimum.intel import OVModelForCausalLM
from transformers import AutoTokenizer

model_id = "Your_Username/Gemma-2-2B-IT-OpenVINO-INT4"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = OVModelForCausalLM.from_pretrained(model_id)

# Gemma 2 Chat Template Handshake
messages = [
    {"role": "user", "content": "Explain the advantages of INT4 quantization."},
]
input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

outputs = model.generate(input_ids, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

🏛️ Chat Template

This model uses the standard Gemma 2 instruction format:

text

<bos><start_of_turn>user
{prompt}<end_of_turn>
<start_of_turn>model

🏗️ Technical Details

Optimization Tool: NNCF (Neural Network Compression Framework)
Quantization: INT4 Asymmetric
Format: OpenVINO IR (.xml / .bin)
VRAM/RAM Usage: ~1.6GB

📜 License

This model is released under the Gemma Terms of Use. Users must adhere to the Google Prohibited Use Policy.

☕ Support the Forge

Maintaining a dual-GPU AI workstation and hosting high-bandwidth models requires significant resources. If our open-source tools power your projects, consider supporting our development:

Platform	Support Link
Global & India	Support via Razorpay

Scan to support via UPI (India Only):

Connect with the architect: Abhishek Jaiswal on LinkedIn

CelesteImperia/Gemma-2-2B-IT-OpenVINO-INT4