Multimodal embedding distributions across various distillation methods : We extract image and text embeddings from a finetuned CLIP and project them into a shared representation space using DOSNES. Red triangles and blue circles denote image and text embeddings, respectively. Left: Embeddings from randomly sampled data in the original dataset exhibit a well-spread and modality-aligned distribution. Middle: The distilled dataset generated by a sota MDD method (LoRS) leads to Modality Collapse, where image and text embeddings are poorly aligned and concentrated in distinct regions. Right: Our method effectively mitigates modality collapse, yielding a distribution that better preserves cross-modal alignment and exhibits greater representational diversity.

To get started, follow these instructions to set up the environment and install dependencies.

Here’s how to use RepBlend for Multimodal Dataset Distillation:

First, download the pretrained weights and datasets and place them into their respective folders.

The checkpoints for all experimental networks are available from their respective official repositories. For convenience, we have also provided them together 🤗 here. Once downloaded, put them in distill_utils/checkpoints/.

The dataset hase been validated on various benchmarks, you can download from their respective links. Once downloaded, put them in distill_utils/data/.

You can generate expert trajectories by running the scripts/buffer.sh, or alternatively, download our [pre-generated trajectories](🤗 https://huggingface.co/xinxin66/RepBlend) for faster reproduction.

You can distill multimodal datasets with RepBlend by running scripts/distill_coco_repblend.sh and scripts/distill_flickr_repblend.sh.

bash scripts/distill_coco_repblend.sh
bash scripts/distill_flickr_repblend.sh

Our experiments demonstrate the effectiveness of the proposed approach across various benchmarks.

For detailed experimental results and further analysis, please refer to the full paper.

If you find this code useful in your research, please consider citing our work:

Our code has referred to previous works: