We propose GENIUS, a universal generative retrieval framework that supports diverse tasks across multiple modalities. By learning discrete, modality‐decoupled IDs via semantic quantization, GENIUS encodes multimodal data into compact identifiers and performs constant‐time retrieval with competitive accuracy.
GENIUS Overview
✨ Key Advantages
Universal Retrieval
Single model handles various retrieval tasks including image‐to‐image, text‐to‐text, image‐to‐text, text‐to‐image, and their combinations.
Fast Retrieval
Constant‐time lookup via discrete ID matching, independent of candidate pool size.
Competitive Accuracy
Comparable to—and sometimes better than—embedding‐based methods, while significatnly reducing inference cost.
Overview
GENIUS consists of three key components that work together in a three-stage training pipeline:
Multimodal Encoder (CLIP-SF)
Extracts joint image/text features using a shared backbone. We leverage UniIR's score‐fusion CLIP model to learn cross‐modal relations without extra pretraining, with pretrained checkpoints available on Hugging Face.
Modality-Decoupled Quantizer
Compresses continuous embeddings into discrete, layered ID codes including modality and semantic information. Through residual quantization training, it learns to encode both images and text into layered, discrete IDs:
Subsequent codes: semantic features (objects → attributes → context)
Semantic Quantization
ID Generator Sequence‐to‐sequence decoder that generates target IDs based on query embeddings. During training, it learns to predict discrete IDs from various input queries (images, text, or pairs with instruction). We employ query augmentation (query-target mixing) to improve generalization and incorporate constrained beam search during decoding to enforce valid ID sequences.
Installation
Clone the repository and create the Conda environment:
Note: All three models are required for full functionality.
📈 Performance
The results in parentheses denote scores from our reimplemented checkpoints, as the originals were lost during server migration. While close to the paper, slight variations may occur due to retraining randomness.
Universal Information Retrieval
Task
Dataset
CLIP_SF
BLIP_FF
GENIUS (checkpoint)
GENIUSᴿ (checkpoint)
T→I
VisualNews
42.6
23.0
18.5 (18.5)
27.3 (27.3)
MSCOCO
77.9
75.6
55.1 (55.3)
68.0 (68.0)
Fashion200K
17.8
25.4
13.7 (14.0)
16.2 (15.9)
T→T
WebQA
84.7
79.5
31.1 (31.9)
42.9 (43.6)
T→(I,T)
EDIS
59.4
50.3
36.6 (37.0)
44.1 (44.1)
WebQA
78.8
79.7
49.0 (49.0)
59.7 (59.3)
I→T
VisualNews
42.8
21.1
18.4 (18.2)
26.8 (26.8)
MSCOCO
92.3
88.8
82.7 (83.0)
90.6 (90.7)
Fashion200K
17.9
27.6
12.8 (12.9)
16.2 (16.6)
I→I
NIGHTS
33.4
33.0
8.1 (8.1)
30.2 (30.0)
OVEN
39.2
34.7
34.6 (34.5)
38.0 (38.0)
(I,T)→T
InfoSeek
24.0
19.7
10.4 (10.5)
18.0 (18.0)
(I,T)→I
FashionIQ
26.2
28.5
13.1 (13.1)
19.2 (19.3)
CIRR
43.0
51.4
20.1 (20.1)
38.3 (38.1)
(I,T)→(I,T)
OVEN
60.2
57.8
36.5 (36.6)
48.6 (48.3)
InfoSeek
44.6
27.7
14.2 (14.3)
28.6 (28.7)
⚡ Efficiency
When the candidate pool grows, embedding‐based retrieval (e.g., CLIP + nearest neighbors) slows down dramatically. In contrast, GENIUS's discrete ID generation is nearly constant time. Empirically, GENIUS is roughly 4× faster than competing generative methods like GRACE.
cvpr25_genius
Citation
If you find this work useful, please cite:
bibtex
1@inproceedings{kim2024genius,
2 title={GENIUS: A Generative Framework for Universal Multimodal Search},
3 author={Kim, Sungyeon and Zhu, Xinliang and Lin, Xiaofan and Bastan, Muhammet and Gray, Douglas and Kwak, Suha},
4 journal={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
5 year={2025}
6}
License
This project is licensed under the MIT License - see the LICENSE file for details.
