Envisioning Beyond the Few: Disentangled Semantics and Primitives for Few-Shot Atypical Layout-to-Image Generation

Environment Setup

1. Main environment for core training and inference.
   bash

   1conda create -n dsp python=3.10.20 2conda activate dsp 3pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu126 4pip install datasets==4.8.5 pillow==12.2.0 accelerate==1.13.0 transformers==5.8.1 diffusers==0.38.0 safetensors==0.8.0rc0 tensorboard==2.20.0 opencv-python==4.13.0.92 einops==0.8.2 imagesize==2.0.0 peft==0.19.1 ttach==0.0.3 ftfy==6.3.1 albumentations==2.0.8
2. Evaluation environment for MMDetection/MMEngine compatibility. It is used for evaluation with MMDetection/MMEngine due to strict version constraints, and also supports YOLO-based evaluation.
   bash

   1conda create -n dsp-eval python=3.10.20 2conda activate dsp-eval 3conda install mkl==2023.1.0 numpy==1.26.4 4conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=12.1 -c pytorch -c nvidia 5pip install mmengine==0.10.7 tqdm==4.67.3 shapely==2.1.2 scipy==1.15.3 terminaltables==3.1.10 ultralytics==8.4.50 pycocotools==2.0.11 https://download.openmmlab.com/mmcv/dist/cu121/torch2.1.0/mmcv-2.1.0-cp310-cp310-manylinux1_x86_64.whl "numpy<2.0.0" "setuptools<70.0.0"

Set Environment Variables

Pretrained Models Preparation

1. We use several pretrained models as external dependencies. Please download them manually from the following sources:
   • stable-diffusion-v1-5
   • clip-vit-large-patch14
   • dinov2_vitl14_pretrain.pth
   • ViT-B-16.pt
2. After downloading, organize the pretrained weights under ./pretrained as follows:
   bash

   1pretrained 2├── stable-diffusion-v1-5 3│ └── ... 4├── clip-vit-large-patch14 5│ └── ... 6├── dinov2_vitl14_pretrain.pth 7└── ViT-B-16.pt
   You may either copy or symlink the files. We recommend using symbolic links:
   bash

   1ln -s /path/to/stable-diffusion-v1-5 ./pretrained/stable-diffusion-v1-5 2ln -s /path/to/clip-vit-large-patch14 ./pretrained/clip-vit-large-patch14 3ln -s /path/to/dinov2_vitl14_pretrain.pth ./pretrained/dinov2_vitl14_pretrain.pth 4ln -s /path/to/ViT-B-16.pt ./pretrained/ViT-B-16.pt

Data Preparation

1. We use several public datasets. Please download them manually from the following sources:
   • DIOR
   • RUOD
   • ExDark
2. Unzip the downloaded datasets and organize the external dataset directories as follows:
   bash

   1DIOR-VOC 2├── Annotations 3│ ├── Horizontal_Bounding_Boxes 4│ └── Oriented_Bounding_Boxes 5└── VOC2007 6 ├── ImageSets 7 │ ├── Layout 8 │ ├── Main 9 │ └── Segmentation 10 └── JPEGImages
   bash

   1RUOD 2├── Environment_pic 3│ ├── blur 4│ ├── color 5│ └── light 6├── Environmet_ANN 7├── RUOD_ANN 8└── RUOD_pic 9 ├── test 10 └── train
   bash

   1ExDark 2├── annos 3├── imageclasslist.txt 4└── images
3. Run data preprocessing scripts located in ./scripts/data_process, after updating all hard-coded paths (e.g., /path/to/DIOR_VOC, /path/to/RUOD, /path/to/ExDark) in the scripts to match the local setup. Execute them in order.
   The preprocessing outputs will be generated under ./data with the following structure:
   bash

   1data 2├── DIOR 3│ ├── dior_emb.pt 4│ ├── images -> /path/to/DIOR-VOC/VOC2007/JPEGImages 5│ ├── metadatas 6│ └── patches 7├── EXDARK 8│ ├── exdark_emb.pt 9│ ├── images 10│ ├── metadatas 11│ └── patches 12└── RUOD 13 ├── images -> /path/to/RUOD/RUOD_pic 14 ├── metadatas 15 ├── patches 16 └── ruod_emb.pt

Training and Inference

Argument Description:

• config: configuration file for model and dataset setup.
• metaseed: seed generator identifier for deterministic sampling.
• num_seed: number of sampling seeds for few-shot evaluation.
• k_shot: number of samples per category in few-shot setting.
• run_id: identifier for different runs.
• gpu_ids: GPU device indices for execution.
• iter: number of bootstrap iterations for FID.

Base Phase Training

Novel Phase Training

Inference

Evaluation

Preparation

YOLO (mAP / AP50 / AP75)

Note: In yolo-wrapper-dior.sh, the --xml_folder path should be set to the DIOR annotation directory (/path/to/DIOR-VOC/Annotations/Horizontal_Bounding_Boxes).

Faster R-CNN (mAP / AP50 / AP75)

Bootstrap FID

Detection Metric Summarization

Acknowledgement

Citation