-
What fundamental constraints set native VLMs apart from modular ones, and to what extent can these barriers be overcome?
-
How to make research in native VLMs more accessible and democratized, thereby accelerating progress in the field.
-
effectively align pixel and word representations within a shared semantic space;
-
seamlessly integrate the strengths of separate vision and language modules;
-
inherently embody various cross-modal properties that support unified vision-language encoding, aligning, and reasoning.
-
With only 390M image-text examples, NEO develops strong visual perception from scratch inside a dense and monolithic model via elaborate primitives.
-
NEO serves as a cornerstone for scalable and powerful native VLMs, paired with reusable components that foster a cost-effective and extensible ecosystem.
-
Model Type: Native Vision-Language Models
-
Model Mode: Mixed Native-Attn & Native-RoPE
-
Layer Parameters: 214M vs. 193M (Qwen3-8B)
-
Model Parameters: 9B (Non-Embedding)
-
Number of Layers: 42 (6 for Pre-Buffer & 36 for Post-LLM)
-
Number of Heads: 32 for Q and 8 for KV (GQA)
-
Head Dimensions: 128 * 2 for QK and 128 for V
1@article{Diao2025NEO,
2 title = {From Pixels to Words--Towards Native Vision-Language Primitives at Scale},
3 author = {Diao, Haiwen and Li, Mingxuan and Wu, Silei and Dai, Linjun and Wang, Xiaohua and Deng, Hanming and Lu, Lewei and Lin, Dahua and Liu, Ziwei},
4 journal = {arXiv preprint arXiv:2510.14979},
5 year = {2025}
6}
