Official implementation of paper: Zero Shot Domain Adaptive Semantic Segmentation by Synthetic Data Generation and Progressive Adaptation (IROS 25').

Code: https://github.com/roujin/SDGPA

Deep learning-based semantic segmentation models achieve impressive results yet remain limited in handling distribution shifts between training and test data. In this paper, we present SDGPA (Synthetic Data Generation and Progressive Adaptation), a novel method that tackles zero-shot domain adaptive semantic segmentation, in which no target images are available, but only a text description of the target domain's style is provided. To compensate for the lack of target domain training data, we utilize a pretrained off-the-shelf text-to-image diffusion model, which generates training images by transferring source domain images to target style. Directly editing source domain images introduces noise that harms segmentation because the layout of source images cannot be precisely maintained. To address inaccurate layouts in synthetic data, we propose a method that crops the source image, edits small patches individually, and then merges them back together, which helps improve spatial precision. Recognizing the large domain gap, SDGPA constructs an augmented intermediate domain, leveraging easier adaptation subtasks to enable more stable model adaptation to the target domain. Additionally, to mitigate the impact of noise in synthetic data, we design a progressive adaptation strategy, ensuring robust learning throughout the training process. Extensive experiments demonstrate that our method achieves state-of-the-art performance in zero-shot semantic segmentation.

Environment setting:

All of our experiments are conducted on NVIDIA RTX 3090 with cuda 11.8

You can find all the training scripts in the scripts/ folder.

We use day $\to$ snow setting as an example.

First, you should decide where you want to put the datasets. Let's denote it as <data_root> (for example:/data3/roujin). By default, the experimental logs are stored in <data_root>.

Then, organize the folder as follows:

<data_root>
└─ ACDC
   └─ gt
   └─ rgb_anon
└─ cityscapes
   └─ gtFine
   └─ leftImg8bit
└─ GTA5
   └─ images
   └─ labels

You can refer to cityscapes and ACDC's official websites for the datasets. For GTA5, as we only use a subset of them, we provide the following link to download the subset for your convenience: https://huggingface.co/datasets/roujin/GTA5subset

For synthetic data generation:

For progress model adaptation:

Evaluation:

<setting> can be "day", "fog", "rain", "snow", "night", "game"

We release the following results. See all logs and checkpoints during training from https://huggingface.co/roujin/SDGPA/tree/main

We recommend you to read the scripts and the paper for more details.

For hyperparameter selection of InstructPix2Pix, we recommend reading: https://huggingface.co/spaces/timbrooks/instruct-pix2pix/blob/main/README.md

This code is built upon the following repositories:

We thank them for their excellent work!