1School of Software, Shandong University 2School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), 2School of Data Science, City University of Hong Kong
✉ Corresponding author
This repository hosts the official pre-trained model weights for ENCODER, a novel network designed to explicitly mine visual entities and modification actions, and securely bind implicit modification relations in Composed Image Retrieval (CIR).
📌 Model Information
1. Model Name
ENCODER (Entity miNing and modifiCation relation binDing nEtwoRk) Checkpoints.
2. Task Type & Applicable Tasks
Task Type: Composed Image Retrieval (CIR).
Applicable Tasks: Retrieving a target image based on a reference image and a corresponding modification text. The model excels at capturing fine-grained modification relations through multimodal semantic alignment.
3. Project Introduction
Existing CIR approaches often struggle with the modification relation between visual entities and modification actions due to irrelevant factor perturbation, vague semantic boundaries, and implicit modification relations.
ENCODER introduces three innovative modules to achieve precise multimodal semantic alignment:
🔍 Latent Factor Filter (LFF): Filters out irrelevant visual and textual factors.
🔗 Entity-Action Binding (EAB): Employs modality-shared Learnable Relation Queries (LRQ) to mine visual entities and actions, learning their implicit relations to bind them effectively.
🧩 Multi-scale Composition (MSC): Performs multi-scale feature composition to precisely push the retrieved feature closer to the target image.
4. Training Data Source & Hosted Weights
The models were trained across four widely-used CIR datasets: FashionIQ, Shoes, Fashion200K, and CIRR. This Hugging Face repository provides the pre-trained .pt checkpoint files for each corresponding dataset:
📄 cirr.pt: Checkpoint trained on the open-domain CIRR dataset.
📄 fashion200k.pt: Checkpoint trained on the Fashion200K dataset.
📄 fashioniq.pt: Checkpoint trained on the FashionIQ dataset.
📄 shoes.pt: Checkpoint trained on the Shoes dataset.
Download the specific .pt files you wish to evaluate from this Hugging Face repository. Place them into a checkpoints/ directory within your cloned GitHub repo.
Step 3: Run Evaluation
To test your trained model on the validation set, use the evaluate_model.py script and point it to the downloaded weights:
Version Compatibility: Different versions of open_clip can impact model performance. To ensure consistent State-of-the-Art performance as reported in the paper, please strictly adhere to the environment dependencies specified in the requirements.txt file of the official repository.
State Dict Version: These hosted weights are the updated "state_dict" version for stable evaluation.
📝⭐️ Citation
If you find this code or our paper useful for your research, please consider leaving a Star ⭐️ on our GitHub repository and citing our AAAI 2025 paper:
bibtex
1@inproceedings{ENCODER,
2 title={Encoder: Entity mining and modification relation binding for composed image retrieval},
3 author={Li, Zixu and Chen, Zhiwei and Wen, Haokun and Fu, Zhiheng and Hu, Yupeng and Guan, Weili},
4 booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
5 volume={39},
6 number={5},
7 pages={5101--5109},
8 year={2025}
9}
