Ships a custom handler.py (Inference Toolkit) because the Gemma 4 arch is too new for the default container. requirements.txt pins transformers>=5.12.1. Deploy on a GPU instance with >=24 GB VRAM (model is ~16 GB bf16).

Request body:

{"inputs": "What is a LoRA adapter?", "parameters": {"max_new_tokens": 256}}

Local use

python

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

m = AutoModelForImageTextToText.from_pretrained(
    "dxv2k/gemma-4-E4B-it-merged", dtype=torch.bfloat16, device_map="auto")
p = AutoProcessor.from_pretrained("dxv2k/gemma-4-E4B-it-merged")
msgs = [{"role": "user", "content": [{"type": "text", "text": "Hello"}]}]
enc = p.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True,
                            return_dict=True, return_tensors="pt").to(m.device)
out = m.generate(**enc, max_new_tokens=128)
print(p.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))