You can deploy this model and start earning money today!

iLearn-Lab/
CVPRW26-ViSAGE

pytorch

apache-2.0

Views

No views yet

🚀 ViSAGE @ CVPR-NTIRE Video Saliency Prediction Challenge 2026

Kun Wang¹ Yupeng Hu¹ Zhiran Li¹ Hao Liu¹ Qianlong Xiang^2,3,4 Liqiang Nie²

¹Shandong University
²Harbin Institute of Technology
³City University of Hong Kong
⁴Shenzhen Loop Area Institute

These are the official implementation, pre-trained model weights, and configuration files for ViSAGE, designed for the NTIRE 2026 Challenge on Video Saliency Prediction (CVPRW 2026).

🔗 Paper: Accepted by CVPRW 2026 🔗 GitHub Repository: iLearn-Lab/CVPRW26-ViSAGE 🔗 Challenge Page: NTIRE 2026 VSP Challenge

📌 Model Information

1. Model Name

ViSAGE(Video Saliency with Adaptive Gated Experts)

2. Task Type & Applicable Tasks

Task Type: Video Saliency Prediction (VSP) / Computer Vision
Applicable Tasks: Robust and adaptive prediction of human visual attention (saliency maps) in dynamic video sequences.

3. Project Introduction

Video Saliency Prediction requires capturing complex spatio-temporal dynamics and human visual priors. ViSAGE tackles this by leveraging a powerful multi-expert ensemble framework.

💡 Method Highlight: The framework consists of a shared InternVideo2 backbone adapted via two-stage LoRA fine-tuning, alongside dual specialized experts utilizing Temporal Modulation (for explicit spatial priors) and Multi-Scale Fusion (for adaptive data-driven perception). For robust performance, the Ensemble Fusion Module obtains the final prediction by converting the expert outputs to logit space before averaging, which provides significantly more accurate estimation than simple saliency map averaging.

4. Training Data Source

Dataset provided by the NTIRE 2026 Video Saliency Prediction Challenge (Private Test and Validation sets).

🚀 Usage & Basic Inference

Step 1: Prepare the Environment

Clone the GitHub repository and set up the Conda environment:

bash

git clone https://github.com/iLearn-Lab/CVPRW26-ViSAGE.git
cd ViSAGE

bash

conda create -n visage python=3.10 -y
conda activate visage
pip install -r requirements.txt

Step 2: Data & Pre-trained Weights Preparation

Challenge Data: Use the provided scripts to extract frames from the source videos. The extracted frames will be automatically saved to derived_fullfps. (⚠️ Important: Do not modify the output directory name derived_fullfps unless you manually update the path configs in all inference scripts.)

python video_to_frames.py

ViSAGE Checkpoints: Download our model checkpoints(https://huggingface.co/iLearn-Lab/CVPRW26-ViSAGE).
InternVideo2 Backbone: Download the pre-trained InternVideo2-Stage2_6B-224p-f4 model from Hugging Face and clone the InternVideo repo:

bash

 git clone https://github.com/OpenGVLab/InternVideo.git
 *(Update the pre-trained weight paths in `Expert1/inference.py` and `Expert2/inference.py` to match your local directory).*

Step 3: Run Inference & Ensemble

1. Inference: Generate predictions for both experts.

bash

python Expert1/inference.py
python Expert2/inference.py

2. Ensemble: Merge the inference results from Expert 1 and Expert 2 in logit space.

python ensemble.py

3. Format Check & Video Generation: Validate your submission format and render the predicted saliency outputs onto the source video frames.

bash

python check.py
python makevideos.py

Step 4: Training (Optional)

If you wish to train the model from scratch, run the two-stage LoRA fine-tuning pipeline:

bash

python trainnew.py   # Stage 1
python trainnew2.py  # Stage 2

⚠️ Limitations & Notes

Disclaimer: This framework and its pre-trained weights are intended for academic research purposes only.

The model relies heavily on the InternVideo2 backbone; out-of-memory (OOM) errors may occur on GPUs with less than 24GB VRAM.
Inference speed and performance may fluctuate depending on the hardware utilized.

🤝 Acknowledgements & Contact

Contact: If you have any questions or encounter issues, feel free to open an issue or contact the author Kun Wang at khylon.kun.wang@gmail.com.

📝⭐️ Citation

If you find this project useful for your research, please consider citing:

@inproceedings{ntire26visage, title={{ViSAGE @ NTIRE 2026 Challenge on Video Saliency Prediction: Methods and Results}}, author={Wang, Kun and Hu, Yupeng and Li, Zhiran and Liu, Hao and Xiang, Qianlong and Nie, Liqiang},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
year={2026} }