[2024.8.27] The core code for the evaluation and fine-tuning of VISTA can be obtained from this link. This includes Stage2 training, downstream task fine-tuning, as well as the datasets we used for evaluation.
[2024.6.13] We have released VISTA-S2 dataset, a hybrid multi-modal dataset consisting of over 500,000 instances for multi-modal training (Stage-2 training in our paper).
In this project, we introduce Visualized-BGE, a universal multi-modal embedding model. By incorporating image token embedding into the BGE Text Embedding framework, Visualized-BGE gains the flexibility to process multi-modal data that goes beyond just text. Visualized-BGE is mainly used for hybrid modal retrieval tasks, including but not limited to:
Multi-Modal Knowledge Retrieval (query: text; candidate: image-text pairs, text, or image) e.g. WebQA
We have generated a hybrid multi-modal dataset consisting of over 500,000 instances for multi-modal training (Stage-2 training in our paper). You can download our dataset from this 🤗 HF Link.
Process the image compression package with the following commands:
If you obtain the following directory structure. You can then use the annotation information (json files) for your own training:
images
|__coco
|__edit_image
Usage
Installation:
Install FlagEmbedding:
git clone https://github.com/FlagOpen/FlagEmbedding.git
cd FlagEmbedding/research/visual_bge
pip install -e .
Another Core Packages:
pip install torchvision timm einops ftfy
You don't need to install xformer and apex. They are not essential for inference and can often cause issues.
Generate Embedding for Multi-Modal Data:
Visualized-BGE provides the versatility to encode multi-modal data in a variety of formats, whether it's purely text, solely image-based, or a combination of both.
1####### Use Visualized BGE doing composed image retrieval2import torch
3from visual_bge.modeling import Visualized_BGE
45model = Visualized_BGE(model_name_bge ="BAAI/bge-base-en-v1.5", model_weight="path: Visualized_base_en_v1.5.pth")6model.eval()7with torch.no_grad():8 query_emb = model.encode(image="./imgs/cir_query.png", text="Make the background dark, as if the camera has taken the photo at night")9 candi_emb_1 = model.encode(image="./imgs/cir_candi_1.png")10 candi_emb_2 = model.encode(image="./imgs/cir_candi_2.png")1112sim_1 = query_emb @ candi_emb_1.T
13sim_2 = query_emb @ candi_emb_2.T
14print(sim_1, sim_2)# tensor([[0.8750]]) tensor([[0.7816]])
Multi-Modal Knowledge Retrieval
python
1####### Use Visualized BGE doing multi-modal knowledge retrieval2import torch
3from visual_bge.modeling import Visualized_BGE
45model = Visualized_BGE(model_name_bge ="BAAI/bge-base-en-v1.5", model_weight="path: Visualized_base_en_v1.5.pth")6model.eval()7with torch.no_grad():8 query_emb = model.encode(text="Are there sidewalks on both sides of the Mid-Hudson Bridge?")9 candi_emb_1 = model.encode(text="The Mid-Hudson Bridge, spanning the Hudson River between Poughkeepsie and Highland.", image="./imgs/wiki_candi_1.jpg")10 candi_emb_2 = model.encode(text="Golden_Gate_Bridge", image="./imgs/wiki_candi_2.jpg")11 candi_emb_3 = model.encode(text="The Mid-Hudson Bridge was designated as a New York State Historic Civil Engineering Landmark by the American Society of Civil Engineers in 1983. The bridge was renamed the \"Franklin Delano Roosevelt Mid-Hudson Bridge\" in 1994.")1213sim_1 = query_emb @ candi_emb_1.T
14sim_2 = query_emb @ candi_emb_2.T
15sim_3 = query_emb @ candi_emb_3.T
16print(sim_1, sim_2, sim_3)# tensor([[0.6932]]) tensor([[0.4441]]) tensor([[0.6415]])
Huixiangdou: Using Visualized BGE for the group chat assistant.
Evaluation Result
Visualized BGE delivers outstanding zero-shot performance across multiple hybrid modal retrieval tasks. It can also serve as a base model for downstream fine-tuning for hybrid modal retrieval tasks.
Zero-shot Performance
Statistical information of the zero-shot multi-modal retrieval benchmark datasets. During the zero-shot evaluation, we utilize the queries from the validation or test set of each dataset to perform retrieval assessments within the entire corpus of the respective dataset.
Statistical information for the zero-shot multi-modal retrieval benchmark datasets.
Zero-shot evaluation results with Recall@5 on various hybrid multi-modal retrieval benchmarks. The -MM notation indicates baseline models that have undergone multi-modal training on our generated data.
Zero-shot evaluation results with Recall@5 on various hybrid multi-modal retrieval benchmarks.
Fine-tuning on Downstream Tasks
Supervised fine-tuning performance on the WebQA dataset. All retrievals are performed on the entire deduplicated corpus.
image.png
Supervised fine-tuning performance on the CIRR test set.
image.png
Supervised fine-tuning performance on the ReMuQ test set.
image.png
FAQ
Q1: Can Visualized BGE be used for cross-modal retrieval (text to image)?
A1: While it is technically possible, it's not the recommended use case. Our model focus on augmenting hybrid modal retrieval tasks with visual capabilities.
Acknowledgement
The image token embedding model in this project is built upon the foundations laid by EVA-CLIP.
Citation
If you find this repository useful, please consider giving a star ⭐ and citation
@article{zhou2024vista,
title={VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval},
author={Zhou, Junjie and Liu, Zheng and Xiao, Shitao and Zhao, Bo and Xiong, Yongping},
journal={arXiv preprint arXiv:2406.04292},
year={2024}
}
