You can deploy this model and start earning money today!

tomg-group-umd/
zero-model-checkpoints

Name: zero-model-checkpoints
Availability: InStock

transformers

pytorch

llama-3

zero-shot-vision-encoder-grafting

image-text-to-text

2505.22664

meta-llama/Llama-3.1-70B-Instruct

finetune

cc-by-nc-4.0

endpoints_compatible

Views

No views yet

Zero-Shot Vision Encoder Grafting via LLM Surrogates

International Conference on Computer Vision (ICCV), 2025

This repository contains vision encoders and surrogate models, which are described in Zero-Shot Vision Encoder Grafting via LLM Surrogates.

Paper Abstract

Vision language models (VLMs) typically pair a modestly sized vision encoder with a large language model (LLM), e.g., Llama-70B, making the decoder the primary computational burden during training. To reduce costs, a potential promising strategy is to first train the vision encoder using a small language model before transferring it to the large one. We construct small "surrogate models" that share the same embedding space and representation language as the large target LLM by directly inheriting its shallow layers. Vision encoders trained on the surrogate can then be directly transferred to the larger model, a process we call zero-shot grafting -- when plugged directly into the full-size target LLM, the grafted pair surpasses the encoder-surrogate pair and, on some benchmarks, even performs on par with full decoder training with the target LLM. Furthermore, our surrogate training approach reduces overall VLM training costs by ~45% when using Llama-70B as the decoder.

Sample Usage

This model is a full Vision-Language Model (LlavaLlamaForCausalLM) incorporating a surrogate-trained vision encoder, and can be used directly with the transformers library.

python

from transformers import AutoProcessor, AutoModelForCausalLM
import torch
from PIL import Image
import requests
from io import BytesIO

# Load model and processor
# Example model ID for a surrogate-trained 8B Llama-3.1 encoder
model_id = "tomg-group-umd/llama3.1-8b_surrogate-trained-encoder" 
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True # Required as it uses a custom LlavaLlamaForCausalLM architecture
)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

# Prepare inputs
image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/bird_sized.jpg"
image = Image.open(BytesIO(requests.get(image_url).content))

messages = [
    {"role": "user", "content": "What is in this image?"},
]

# Apply chat template and process inputs
prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=prompt, images=image, return_tensors="pt").to(model.device)

# Generate response
output_ids = model.generate(**inputs, max_new_tokens=50)
output = processor.decode(output_ids[0], skip_special_tokens=True)

print(output)
# Expected output (may vary slightly):
# "A bird with a blue head and green body is perched on a branch. The bird has a long tail and is facing to the right."

Citation

If you find our work helpful in your research, please cite it as:

bibtex

@inproceedings{yue2025zero,
  title     = {Zero-Shot Vision Encoder Grafting via LLM Surrogates},
  author    = {Yue, Kaiyu and Singla, Vasu and Jia, Menglin and Kirchenbauer, John and Qadri, Rifaa and Cai, Zikui and Bhatele, Abhinav and Huang, Furong and Goldstein, Tom},
  booktitle = {International Conference on Computer Vision (ICCV)},
  year      = {2025}
}

tomg-group-umd/zero-model-checkpoints

Zero-Shot Vision Encoder Grafting via LLM Surrogates

Paper Abstract

Sample Usage

Citation

tomg-group-umd/
zero-model-checkpoints