PenguinVL is a compact Vision-Language Model designed to explore the efficiency limits of small-scale VLMs. Rather than being only an instruction-tuned model, PenguinVL is built from the ground up through LLM-based vision encoder construction, multimodal pretraining, and subsequent instruction tuning.
Unlike most existing VLMs that rely on contrastive-pretrained vision encoders (e.g., CLIP/SigLIP), PenguinVL initializes its vision encoder directly from a text-only LLM. This design avoids the objective mismatch between contrastive learning and autoregressive language modeling, enabling tighter alignment between visual representations and the language backbone.
-
🧠 LLM-based Vision Encoder
The vision encoder is adapted from a pretrained text LLM (Qwen3-0.6B), modified with bidirectional attention and 2D-RoPE for spatial modeling.
This provides strong semantic priors and native compatibility with the downstream LLM.
-
🎥 Efficient Video Understanding
A Temporal Redundancy-Aware (TRA) token compression strategy dynamically allocates token budgets across frames, enabling long-video reasoning within a limited context window.
-
🏗 Unified Architecture
The model consists of:
- LLM-initialized vision encoder
- Lightweight MLP projector
- Qwen3 language backbone
-
📊 Compact but Strong
At 2B scale, Penguin-VL achieves competitive performance across image, document, OCR, math, and video benchmarks while remaining deployment-friendly.
1import torch
2from transformers import AutoModelForCausalLM, AutoProcessor
3
4model_name = "tencent/Penguin-VL-2B"
5
6model = AutoModelForCausalLM.from_pretrained(
7 model_name,
8 trust_remote_code=True,
9 device_map="auto",
10 torch_dtype=torch.bfloat16,
11)
12
13processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)
14
15# Example: Image + Text
16inputs = processor(
17 conversation=[
18 {"role": "system", "content": "You are a helpful assistant."},
19 {
20 "role": "user",
21 "content": [
22 {"type": "image", "image": {"image_path": "assets/example.jpg"}},
23 {"type": "text", "text": "Describe this image."}
24 ],
25 },
26 ],
27 return_tensors="pt",
28)
29
30inputs = {k: v.cuda() if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
31if "pixel_values" in inputs:
32 inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16)
33
34output_ids = model.generate(**inputs, max_new_tokens=128)
35response = processor.decode(output_ids[0], skip_special_tokens=True)
36
37print(response)
If you find Penguin-VL useful for your research and applications, please cite using this BibTeX:
1@article{Penguin-VL,
2 title={Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders},
3 author={Boqiang Zhang and Lei Ke and Ruihan Yang and Qi Gao and Tianyuan Qu and Rossell Chen and Dong Yu and Leoweiliang},
4 journal={arXiv preprint arXiv:2603.06569},
5 year={2026}
6}
