ProCap: Experiment Materials

Imagine How to Change: Explicit Procedure Modeling for Change Captioning

1. Explicit Procedure Modeling: Trains a procedure encoder to learn the change procedure from a sparse set of keyframes.
2. Implicit Procedure Captioning: Integrates the trained encoder within an encoder-decoder model for captioning using learnable procedure queries.

Contents

• Data
• Model Weights
• Evaluation
• Usage
• License

Data

Included Datasets

• CLEVR-data
  Processed pseudo-sequences for the CLEVR-Change dataset
• edit-data
  Processed pseudo-sequences for the Image-Editing-Request dataset
• spot-data
  Processed pseudo-sequences for the Spot-the-Diff dataset
• filter_files
  Confidence scores computed using CLIP4IDC
• filtered-spot-captions
  Refined captions for the Spot-the-Diff dataset

Model Weights

Explicit Procedure Modeling (Stage 1)

• pretrained_vqgan – VQGAN models for each dataset
• stage1_clevr_best
• stage1_edit_best
• stage1_spot_best

Implicit Procedure Captioning (Stage 2)

• clevr_best
• edit_best
• spot_best

Note: Stage 1 checkpoints can be directly reused to initialize Stage 2 training.

Evaluation

• densevid_eval
  Evaluation tools used for quantitative assessment

Usage

1. Data Preparation

1. Move caption files in filtered-spot-captions to the original caption directory of the Spot-the-Diff dataset.
2. Copy the processed data folders to the original dataset root and rename them as follows:

1. Place filter_files in the project root directory.

2. Model Weights

• Place pretrained_vqgan in the project root directory.
• To reuse Stage 1 weights during training, set symlink_path in training scripts as:

• To evaluate with pre-trained checkpoints, set resume_path in evaluation scripts as:

3. Evaluation Tool

Citation

License