This repository contains a highly optimized, 4-bit quantized OpenVINO Intermediate Representation (IR) version of Google's MedGemma-1.5-4b-it.

It was quantized using Intel's Neural Network Compression Framework (NNCF) via Hugging Face optimum-intel to enable ultra-lightweight local deployment exclusively on CPUs. By leveraging OpenVINO, this version drastically minimizes RAM footprints and executes inference smoothly on modern CPU architectures without risking Out-Of-Memory (OOM) memory crashes.

Key Features

CPU Optimized: Tuned natively for Intel, AMD, and ARM processors via the OpenVINO C++ execution engine.
Asymmetric 4-bit Quantization (int4_asym): Offers a fantastic balance between medical description correctness and token generation latency.
Low Memory Overhead: Fits easily into standard consumer systems and lightweight cloud instances.
Live Text Streaming: Configured out of the box to pipe tokens to your terminal in real-time as they process.

Installation & Environment Setup

Before using the model, ensure your Python environment has the optimized CPU runtimes installed. It is highly recommended to run this setup in a fresh block to prevent memory collision issues:

pip install --upgrade huggingface_hub optimum[intel] openvino nncf pillow --quiet

Python Inference Example

The following example demonstrates how to run medical image analysis locally using the quantized OpenVINO INT4 model on CPU.

python

from PIL import Image
import torch
from optimum.intel.openvino import OVModelForVisualCausalLM
from transformers import AutoProcessor, TextStreamer

MODEL_ID = "vishnu0967/medgemma-1.5-4b-it-openvino-int4"

print("Loading optimized 4-bit OpenVINO model directly from Hugging Face Hub...")
# Load model directly from Hugging Face onto CPU
model = OVModelForVisualCausalLM.from_pretrained(
    MODEL_ID,
    device="CPU",
    ov_config={
        "PERFORMANCE_HINT": "LATENCY",
        "INFERENCE_NUM_THREADS": ""
    }
)

# Load processor
processor = AutoProcessor.from_pretrained(MODEL_ID)

image = Image.open("/content/input.png").convert("RGB")

messages = [
    {
        "role": "system",
        "content": [
            {
                "type": "text",
                "text": "You are an expert radiologist."
            }
        ]
    },
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Analyze this medical image and explain any abnormalities."
            },
            {
                "type": "image",
                "image": image
            }
        ]
    }
]


prompt = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=False
)

inputs = processor(
    text=prompt,
    images=image,
    return_tensors="pt"
)

streamer = TextStreamer(
    processor,
    skip_prompt=True,
    skip_special_tokens=True
)

print("\nGenerating response on CPU...")
print("--- Radiologist Report ---")

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=520,
        do_sample=False,
        streamer=streamer
    )

response = processor.decode(
    outputs[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True
)

print("\n--- Inference Finished ---")

Notes

Designed for CPU-only inference using OpenVINO Runtime.
Uses 4-bit asymmetric quantization (int4_asym) generated with NNCF.
Supports multimodal image + text medical reasoning.
Works on Intel, AMD, and ARM CPUs supported by OpenVINO.
PERFORMANCE_HINT="LATENCY" is recommended for interactive usage.
Increase max_new_tokens for longer medical reports.

Quick Start

pip install --upgrade huggingface_hub optimum[intel] openvino nncf pillow transformers --quiet

python

from optimum.intel.openvino import OVModelForVisualCausalLM
from transformers import AutoProcessor

model = OVModelForVisualCausalLM.from_pretrained(
    "vishnu0967/medgemma-1.5-4b-it-openvino-int4",
    device="CPU"
)

processor = AutoProcessor.from_pretrained(
    "vishnu0967/medgemma-1.5-4b-it-openvino-int4"
)

vishnu0967/medgemma-1.5-4b-it-openvino-int4

MedGemma 1.5 4B IT - 4-bit OpenVINO INT4 (CPU Optimized)

Key Features

Installation & Environment Setup

Python Inference Example

Notes

Quick Start

license: mit

vishnu0967/
medgemma-1.5-4b-it-openvino-int4