An open-vocabulary dynamic scene graph generation model (DSGG) that integrates the closed-set DSGG OED with the open-vocabulary scene graph generation model (SGG) OvSGTR. We use OvSGTR as the spatial feature extractor and feed its outputs into OED's temporal routing module.
We use the dataset Action Genome to train and evaluate OvDSGG. Please process the downloaded dataset with the Toolkit and put the processed annotation files with COCO style into annotations folder. The directories of the dataset should look like:
Our baseline derives object pair predictions and predicate classifications directly from isolated query representations, without interaction modules. To train the baseline, run these commands:
To test open-vocabulary capabilities of OvDSGG, we train on only 70% of the categories in Action Genome. The rest are only seen during evaluation. To do this, we split the categories into Base (seen during training) and Novel (unseen during training)
The open-vocabulary training split is hardcoded in datasets/ag.py, and has been chosen such that:
~70% of objects are in Base (25 out of 36, 69.4%).
~70% of predicates are in Base (18 out of 26, 69.2%).
Common and rare categories are balanced across Base and Novel sets.
None of the Novel categories were seen by the pretrained open-vocabulary OvSGTR/GroundingDINO checkpoint.
See PRs #12 and #13 for details. To generate the split yourself, run:
python tools/propose_ag_split.py
Key Losses and Metrics
Metric / Loss
Type
Description
loss
Total Loss
Weighted sum of all scaled losses minimized by the optimizer.
loss_obj_ce
Object Loss
Cross Entropy loss for object classification.
loss_obj_bbox
Object Loss
L1 error for object bounding box center and size.
loss_obj_giou
Object Loss
Generalized IoU loss for object bounding box overlap.
loss_sub_bbox
Subject Loss
L1 error for subject bounding box center and size.
loss_sub_giou
Subject Loss
Generalized IoU loss for subject bounding box overlap.
loss_attn_ce
Relation Loss
Loss for classifying the "attention" or main interaction.
loss_spatial_ce
Relation Loss
Loss for classifying spatial relationships (e.g., "in front of").
loss_contacting_ce
Relation Loss
Loss for classifying contacting relationships (e.g., "holding").
obj_class_error_unscaled
Key Metric
The raw percentage of objects misclassified (Error Rate %).
References
Please note that most of the code in this repository was adapted from OED and OvSGTR. We thank the authors for their excellent work.
