OvDSGG Readme

Accepted at the ECCV 2026 Contextus Workshop. Read the paper on arXiv.

An open-vocabulary dynamic scene graph generation model (DSGG) that integrates the closed-set DSGG OED with the open-vocabulary scene graph generation model (SGG) OvSGTR. We use OvSGTR as the spatial feature extractor and feed its outputs into OED's temporal routing module.

The open-vocabulary setting was benchmarked against an open-vocabulary adaptation of OED and a version of OvSGTR adapted for DSGG.

Contents

• Setup
• Prepare Data
• Train
• Evaluate
• Checkpoints
• Open-Vocabulary Data Split
• Key Losses and Metrics

Setup

You must use Python 3.9 for this repo to work, due to the dependencies of OvSGTR, which this repo is built upon.

conda create -n myenv python=3.9
conda activate myenv
./install.sh

Prepare Data

We use the dataset Action Genome to train and evaluate OvDSGG. Please process the downloaded dataset with the Toolkit and put the processed annotation files with COCO style into annotations folder. The directories of the dataset should look like:

|-- action_genome
    |-- annotations   # gt annotations
        |-- ag_train_coco_style.json
        |-- ag_test_coco_style.json
        |-- ...
    |-- frames        # sampled frames
    |-- videos        # original videos

Zero-Shot Recall (zR@K) for Action Genome is calculated using the file datasets/ov_zeroshot_triplet.pytorch, in this repo. This file was generated using the following script:

python tools/generate_ag_zeroshot.py --ann_file data/action_genome/annotations/ag_train_coco_style.json

Train

We train the spatial module first, then the temporal module on the best spatial checkpoint.

Closed-set training

To run closed-set training, run these commands in order:

1. Spatial Training

Prior to training, download the pretrained OvSGTR closed-set SGG Swin-T checkpoint from HuggingFace and save it to ./checkpoints/vg-swint-full.pth.

python scripts/train_spatial_sgdet_ovdsgg_closed_set.py

2. Temporal Training

python scripts/train_temporal_sgdet_ovdsgg_closed_set.py

Open-vocabulary training

To run open-vocabulary training, run these commands in order:

1. Spatial Training

Prior to training, download the pretrained OvSGTR OvD+R-SGG Swin-T checkpoint from HuggingFace and save it to ./checkpoints/vg-ovdr-swint.pth.

python scripts/train_spatial_sgdet_ovdsgg_ovdr.py

2. Temporal Training

python scripts/train_temporal_sgdet_ovdsgg_ovdr.py

Ablations

Our baseline derives object pair predictions and predicate classifications directly from isolated query representations, without interaction modules. To train the baseline, run these commands:

Closed-set Baseline

python scripts/train_ablation_baseline_sgdet_ovdsgg_closed_set.py

Open-vocabulary Baseline

python scripts/train_ablation_baseline_sgdet_ovdsgg_ovdr.py

Evaluate

Closed-set evaluation

To evaluate closed-set models, run these commands:

1. Spatial Evaluation

python scripts/eval_spatial_sgdet_ovdsgg_closed_set.py

2. Temporal Evaluation

python scripts/eval_temporal_sgdet_ovdsgg_closed_set.py

Open-vocabulary evaluation

To evaluate open-vocabulary models, run these commands:

1. Spatial Evaluation

python scripts/eval_spatial_sgdet_ovdsgg_ovdr.py

2. Temporal Evaluation

python scripts/eval_temporal_sgdet_ovdsgg_ovdr.py

Ablation evaluation

To evaluate the ablation baseline models, run these commands:

Closed-set Baseline

python scripts/eval_ablation_baseline_sgdet_ovdsgg_closed_set.py

Open-vocabulary Baseline

python scripts/eval_ablation_baseline_sgdet_ovdsgg_ovdr.py

Checkpoints

The trained OvDSGG checkpoints are available from Hugging Face and Google Drive:

• Hugging Face: jhelsby/OvDSGG - includes checkpoints and matching config files
• Google Drive: link

To download the Hugging Face checkpoints directly into the expected local paths, run:

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="jhelsby/OvDSGG",
    local_dir=".",
    allow_patterns=["exps/*/checkpoint.pth", "config/*.py"],
)

If downloading from Google Drive manually, place each checkpoint file at the corresponding default path:

• Spatial Open-Vocabulary: exps/spatial_sgdet_ovdsgg_ovdr/checkpoint.pth
• Temporal Open-Vocabulary: exps/temporal_sgdet_ovdsgg_ovdr/checkpoint.pth
• Spatial Closed-Set: exps/spatial_sgdet_ovdsgg_closed_set/checkpoint.pth
• Temporal Closed-Set: exps/temporal_sgdet_ovdsgg_closed_set/checkpoint.pth
• Ablation Open-Vocabulary: exps/ablation_baseline_ovdr/checkpoint.pth
• Ablation Closed-Set: exps/ablation_baseline_closed_set/checkpoint.pth

Open-Vocabulary Data Split

To test open-vocabulary capabilities of OvDSGG, we train on only 70% of the categories in Action Genome. The rest are only seen during evaluation. To do this, we split the categories into Base (seen during training) and Novel (unseen during training)

The open-vocabulary training split is hardcoded in datasets/ag.py, and has been chosen such that:

• ~70% of objects are in Base (25 out of 36, 69.4%).
• ~70% of predicates are in Base (18 out of 26, 69.2%).
• Common and rare categories are balanced across Base and Novel sets.
• None of the Novel categories were seen by the pretrained open-vocabulary OvSGTR/GroundingDINO checkpoint.

See PRs #12 and #13 for details. To generate the split yourself, run:

python tools/propose_ag_split.py

Key Losses and Metrics

Metric / Loss	Type	Description
loss	Total Loss	Weighted sum of all scaled losses minimized by the optimizer.
loss_obj_ce	Object Loss	Cross Entropy loss for object classification.
loss_obj_bbox	Object Loss	L1 error for object bounding box center and size.
loss_obj_giou	Object Loss	Generalized IoU loss for object bounding box overlap.
loss_sub_bbox	Subject Loss	L1 error for subject bounding box center and size.
loss_sub_giou	Subject Loss	Generalized IoU loss for subject bounding box overlap.
loss_attn_ce	Relation Loss	Loss for classifying the "attention" or main interaction.
loss_spatial_ce	Relation Loss	Loss for classifying spatial relationships (e.g., "in front of").
loss_contacting_ce	Relation Loss	Loss for classifying contacting relationships (e.g., "holding").
obj_class_error_unscaled	Key Metric	The raw percentage of objects misclassified (Error Rate %).

References

Please note that most of the code in this repository was adapted from OED and OvSGTR. We thank the authors for their excellent work.