PenguinVL is a compact Vision-Language Model, designed to explore the efficiency limits of small-scale VLMs.
Unlike most existing VLMs that rely on contrastive-pretrained vision encoders (e.g., CLIP/SigLIP), Penguin-VL initializes its vision encoder directly from a text-only LLM. This design avoids the objective mismatch between contrastive learning and autoregressive language modeling, enabling tighter alignment between visual representations and the language backbone.
1import torch
2from transformers import AutoModel, AutoImageProcessor
3from transformers.image_utils import load_image
4
5model_name = "tencent/Penguin-Encoder"
6image_path = "your_img.jpg"
7images = load_image(image_path)
8
9model = AutoModel.from_pretrained(
10 model_name,
11 trust_remote_code=True,
12 device_map="auto",
13 torch_dtype=torch.bfloat16,
14 attn_implementation="flash_attention_2",
15)
16processor = AutoImageProcessor.from_pretrained(model_name, trust_remote_code=True)
17
18inputs = processor(images=images, merge_size=1)
19inputs = {k: torch.tensor(v).cuda() for k, v in inputs.items()}
20if "pixel_values" in inputs:
21 inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16)
22image_features = model(**inputs)
Main Results can see the ablation section in our paper.
If you find Penguin-VL useful for your research and applications, please cite using this BibTeX:
1@article{Penguin-VL,
2 title={Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders},
3 author={Boqiang Zhang and Lei Ke and Ruihan Yang and Qi Gao and Tianyuan Qu and Rossell Chen and Dong Yu and Leoweiliang},
4 journal={arXiv preprint arXiv:2603.06569},
5 year={2026}
6}
