Q-Zoom-Qwen3VL-4B
Q-Zoom is a query-aware adaptive high-resolution perception framework
for Multimodal Large Language Models that operates in an efficient
coarse-to-fine manner. Instead of indiscriminately flooding the
quadratic self-attention with redundant high-resolution tokens, Q-Zoom
adds two lightweight modules on top of a pretrained MLLM:
- A Dynamic Gating Network (TWIG) that safely bypasses high-resolution
processing whenever the coarse global features already suffice.
- A Self-Distilled Region Proposal Network (SD-RPN) that, when
high-resolution perception is needed, precisely localizes the
task-relevant Region-of-Interest (RoI) directly from the MLLM's own
intermediate feature space — no extra annotation, no external detector.
This checkpoint is the Stage-3 Q-Zoom finetune of Qwen3-VL-4B-Instruct.
Configuration
- K is the LLM layer index at which the gating head reads hidden
states to decide whether the high-res RoI re-decode should fire.
- T is the TWIG threshold expressed as the gate-score percentile
used during training (lower → more aggressive RoI use at eval).
Highlights
- Q-Zoom accelerates inference at matched accuracy on both Doc/OCR
and high-resolution vision benchmarks, and configured for maximum
perceptual fidelity it surpasses the parent backbone's peak accuracy.
See the project page for the
full per-backbone Pareto curves and the paper for the headline numbers
(e.g. 2.52× Doc/OCR / 4.39× HR speedups, +1.1% / +8.1% over peak on
the Qwen2.5-VL-7B backbone).
- The same recipe transfers across Qwen2.5-VL (3B / 7B), Qwen3-VL,
LLaVA-1.5 (7B / 13B) and emerging RL-based thinking-with-image models.
All evaluation results in the paper are reported under a
per-single-image constraint of either 576 (Doc/OCR) or 4,096 (HR/Vision)
maximum visual tokens.
Quick start
1. Install the matching environment
Q-Zoom touches model-private internals of the backbones, so the
required transformers version differs per family:
| Backbone family | transformers pin | Conda env |
|---|
| Qwen2.5-VL | transformers==4.51.3 | qzoom-q25 |
| Qwen3-VL | transformers==4.57.1 | qzoom-q3 |
The repo's install.sh handles both pins automatically:
1git clone https://github.com/YuHengsss/Q-Zoom.git
2cd Q-Zoom
3bash install.sh qwen3vl
4conda activate qzoom-q3
2. Download the checkpoint
1huggingface-cli download YuhengSSS/Q-Zoom-Qwen3VL-4B \
2 --local-dir ./checkpoints/Q-Zoom-Qwen3VL-4B \
3 --local-dir-use-symlinks False
3. Run the standard Q-Zoom evaluation suite
1CHECKPOINT_PATH=./checkpoints/Q-Zoom-Qwen3VL-4B \
2NUM_GPUS=4 \
3bash examples/eval_only/eval_qwen3vl_stage3.sh
This runs the standard Q-Zoom benchmark suite (TextVQA, InfoVQA, ChartQA,
OCRBench, DocVQA, V*Bench, MME-RealWorld-Lite, HRBench) with the
gating-aware decoding loop. Set TWO_STAGE_ROI=False to disable Q-Zoom and
fall back to vanilla decoding.
At inference time, Q-Zoom always produces a direct response from the
low-resolution pass; the high-res gating head decides per sample whether
to also produce a RoI-based response by re-decoding the cropped
region predicted by the SD-RPN attention map.
Training data
This checkpoint was finetuned with the data hosted at
YuhengSSS/Q-Zoom-Training:
- Stage-1 SD-RPN pseudo-labels (per token attention maps)
- Stage-2 judged Post-SFT JSONLs (consistency-aware sample generation)
- Stage-3 ROI re-decode pickles (per-image RoI boxes + answer pairs)
See DATASETS.md in the GitHub repo for the per-stage filenames.
Citation
1@article{qzoom,
2 title = {Q-Zoom: Query-Aware Adaptive Perception for Efficient
3 Multimodal Large Language Models},
4 author = {Shi, Yuheng and Pei, Xiaohuan and Wen, Linfeng and
5 Dong, Minjing and Xu, Chang},
6 journal= {arXiv preprint arXiv:2604.06912},
7 year = {2026}
8}
You may also be interested in our earlier work that introduced the
self-distilled RoI predictor used by Q-Zoom's SD-RPN branch:
1@article{shi2025catching,
2 title = {Catching the Details: Self-Distilled RoI Predictors for
3 Fine-Grained MLLM Perception},
4 author = {Shi, Yuheng and Pei, Xiaohuan and Dong, Minjing and Xu, Chang},
5 journal= {arXiv preprint arXiv:2509.16944},
6 year = {2025}
7}
License
Apache 2.0. The checkpoint inherits the license of the base model
Qwen/Qwen3-VL-4B-Instruct; please respect both.
Links
