xGen-MM-Vid (BLIP-3-Video) is an efficient compact vision-language model (VLM) with an explicit temporal encoder, specifically designed to understand videos. It is developed by Salesforce AI Research. Incorporation of a learanable temporal encoder modules within the original (image-based) BLIP-3 architecture is its key aspect.
Here, we are sharing the 128 token version trained to take 8-frame video inputs. In principle, it is able to take any number of frames, but it was trained with 8-frame videos.
For more details, check out our tech report. More detailed explanation could also be found in the blog article.
Results
Tokens vs. accuracy
The above figure shows the number of visual tokens vs. accuracy trade-off of various video models including xGen-MM-Vid (BLIP-3-Video) on the MSVD-QA dataset.
Examples
How to use
Please check out our inference script as an example to use our model. This codebase is based on the xGen-MM.
Bias, Risks, Limitations, and Ethical Considerations
The main data sources are from the internet, including webpages, video stock sites, and curated datasets released by the research community.
The model may be subject to bias from the original data source, as well as bias from LLMs and commercial APIs.
We strongly recommend users assess safety and fairness before applying to downstream applications.
License
Our code and weights are released under the CC by-NC 4.0 license.
This release is for research purposes only in support of an academic paper. Our models, datasets, and code are not specifically designed or evaluated for all downstream purposes. We strongly recommend users evaluate and address potential concerns related to accuracy, safety, and fairness before deploying this model. We encourage users to consider the common limitations of AI, comply with applicable laws, and leverage best practices when selecting use cases, particularly for high-risk scenarios where errors or misuse could significantly impact people’s lives, rights, or safety. For further guidance on use cases, refer to our AUP and AI AUP.
Citation
@misc{blip3video-xgenmmvid,
author = {Michael S. Ryoo and Honglu Zhou and Shrikant Kendre and Can Qin and Le Xue and Manli Shu and Silvio Savarese and Ran Xu and Caiming Xiong and Juan Carlos Niebles},
title = {xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMs},
year = {2024},
eprint = {2410.16267},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2410.16267},
}
Troubleshoot
If you missed any packages, please consider the following
