rivitti/paligemma2-10b-mix-224-int8
INT8-ready release of PaliGemma2 10B for NVIDIA GPU inference with Transformers + BitsAndBytes.
This repository is intended to be loaded with 8-bit quantization at inference time.
Model Details
- Base model: google/paligemma2-10b-mix-224
- Task type: Vision-Language (VQA / image-text generation)
- Inference target: NVIDIA GPU (tested workflow for A40)
- Quantization runtime: BitsAndBytes (load_in_8bit=True)
Requirements
- Python 3.10+
- torch with CUDA support
- transformers 4.58.0 or newer
- bitsandbytes 0.44.0 or newer
- accelerate
- pillow
Quick Start (INT8 inference)
from transformers import AutoProcessor, PaliGemmaForConditionalGeneration, BitsAndBytesConfig
from PIL import Image
import torch
repo_id = "rivitti/paligemma2-10b-mix-224-int8"
processor = AutoProcessor.from_pretrained(repo_id)
model = PaliGemmaForConditionalGeneration.from_pretrained(
repo_id,
quantization_config=BitsAndBytesConfig(load_in_8bit=True),
device_map="cuda:0",
torch_dtype=torch.float16,
)
image = Image.open("your_image.jpg").convert("RGB")
prompt = "<image> Answer briefly: What is in this image?"
inputs = processor(text=prompt, images=image, return_tensors="pt")
inputs = {k: v.to("cuda:0") for k, v in inputs.items()}
with torch.no_grad():
output = model.generate(**inputs, max_new_tokens=32, do_sample=False)
text = processor.batch_decode(output[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]
print(text.strip())
Notes
- Add the <image> token at the beginning of the prompt when using both text and images.
- If you run out of GPU memory, keep CUDA_VISIBLE_DEVICES=0 and close other GPU workloads.
- If loading fails, verify transformers/bitsandbytes versions and CUDA wheel compatibility.
Limitations
- Accuracy can vary depending on prompt format and decoding parameters.
- Short-answer prompting is recommended for VQA benchmarking.
Intended Use
- Research and benchmarking for VLM inference.
- VQA experiments with reproducible 8-bit loading settings.
