This repository hosts the
Extract-0.6B† model, which serves as the perception module for the two-stage
Extract+Think† framework. This model was presented in the paper
Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models.
Extract+Think is an approach designed to address perception and reasoning bottlenecks in small multimodal models. It focuses on visual extraction tuning, explicitly training the model to consistently extract instruction-relevant visual details across tasks, which then feeds into a separate reasoning stage.
In this variant, we train from scratch under the visual extraction tuning paradigm, without previous visual instruction tuning or captioning.
Extract-0.6B
† is used as the perception module for the two-stage Extract+Think
† framework. For the reasoning stage, the authors primarily utilize Qwen3 models (
1.7B and
4B).
To use this model, particularly for evaluation, the authors utilize the
lmms-eval framework. The setup and evaluation instructions are detailed in the
GitHub repository. This involves cloning the repository, installing dependencies, and integrating custom evaluation files with
lmms-eval.
1cd lmms-eval
2model_name=markendo/llava-extract-from-scratch-qwen3-0.6B
3python -m lmms_eval \
4 --model=llava_onevision \
5 --model_args=pretrained=$model_name,conv_template=qwen_1_5,device_map=auto \
6 --tasks=mmstar_prism_stage_1 \
7 --batch_size=1 \
8 --output_path results \
9 --log_samples
Please refer to the
GitHub repository for full setup instructions, including the second stage of reasoning.
This repository is built on top of
LLaVA-OneVision and
lmms-eval.
1@article{endo2025downscalingintelligence,
2 author = {Endo, Mark and Yeung-Levy, Serena},
3 title = {Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models},
4 journal = {arXiv preprint},
5 year = {2025},
6}
