LaViT (Latent Visual Thoughts) addresses a critical Perception Gap in multimodal latent reasoning: student models often mimic a teacher's textual output while attending to fundamentally divergent visual regions, effectively relying on language priors rather than grounded perception.
To bridge this gap, LaViT compels the student to autoregressively reconstruct the teacher's visual semantics and attention trajectories prior to text generation, employing a curriculum sensory gating mechanism to prevent shortcut learning.
Abstract: Current multimodal latent reasoning often relies on external supervision (e.g., auxiliary images), ignoring intrinsic visual attention dynamics. In this work, we identify a critical Perception Gap in distillation: student models frequently mimic a teacher's textual output while attending to fundamentally divergent visual regions, effectively relying on language priors rather than grounded perception. To bridge this, we propose LaViT, a framework that aligns latent visual thoughts rather than static embeddings. LaViT compels the student to autoregressively reconstruct the teacher's visual semantics and attention trajectories prior to text generation, employing a curriculum sensory gating mechanism to prevent shortcut learning. Extensive experiments show that LaViT significantly enhances visual grounding, achieving up to +16.9% gains on complex reasoning tasks and enabling a compact 3B model to outperform larger open-source variants and proprietary models like GPT-4o.
🚀 Usage
Installation
pip install transformers torch pillow
Basic Usage
python
1from transformers import AutoProcessor, AutoModelForCausalLM
2from PIL import Image
3import requests
45# Load model and processor6processor = AutoProcessor.from_pretrained("Svard/LaViT-3B")7model = AutoModelForCausalLM.from_pretrained("Svard/LaViT-3B")89# Load image10url ="https://example.com/image.jpg"11image = Image.open(requests.get(url, stream=True).raw)1213# Prepare prompt14prompt ="What is in this image?"1516# Process inputs17inputs = processor(images=image, text=prompt, return_tensors="pt")1819# Generate response20outputs = model.generate(**inputs, max_new_tokens=512)21response = processor.decode(outputs[0], skip_special_tokens=True)22print(response)
Advanced Usage with Visual Reasoning
For tasks requiring visual reasoning, you can use the <lvr> (Latent Visual Reasoning) tokens:
prompt = "Analyze this image step by step: <lvr> What objects are present? <lvr> What are their spatial relationships? <lvr>"
📊 Performance
LaViT-3B achieves significant improvements on various benchmarks:
MMVP: Enhanced performance on multi-modal visual perception tasks
BLINK: Improved results on visual reasoning benchmarks
Visual Grounding: Up to +16.9% gains on complex reasoning tasks
🏗️ Model Architecture
Base Model: Qwen2.5-VL-3B-Instruct
Parameters: 3B
Training Method: Visual thought trajectory supervision
Key Innovation: Latent visual thought alignment with curriculum sensory gating
📝 Citation
If you find this model useful in your research, please cite:
bibtex
1@misc{wu2026lavitaligninglatentvisual,
2 title={LaViT: Aligning Latent Visual Thoughts for Multi-modal Reasoning},
3 author={Linquan Wu and Tianxiang Jiang and Yifei Dong and Haoyu Yang and Fengji Zhang and Shichaang Meng and Ai Xuan and Linqi Song and Jacky Keung},
4 year={2026},
5 eprint={2601.10129},
6 archivePrefix={arXiv},
7 primaryClass={cs.CV},
8 url={https://arxiv.org/abs/2601.10129},
9}
📄 License
This model is licensed under the Apache-2.0 License.
🙏 Acknowledgments
This model is built upon Qwen2.5-VL and inspired by the LVR (Latent Visual Reasoning) framework. We thank the open-source community for their valuable contributions.
