Audio, video, and image inputs are fully preserved. Only the LLM backbone linear layers are quantized. Audio and image encoders remain at bfloat16.

Quantization details

bnb_4bit_quant_type: nf4
bnb_4bit_use_double_quant: True
bnb_4bit_compute_dtype: bfloat16

Usage

python

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

model = AutoModelForImageTextToText.from_pretrained(
    "derkar00/gemma-4-E2B-it-4bit-nf4",
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
processor = AutoProcessor.from_pretrained("derkar00/gemma-4-E2B-it-4bit-nf4")