Initial commit

data/README.md

@@ -0,0 +1,11 @@
+# Datasets and Evaluation
+
+## SSv2-ST (SSv2 Spatio-Temporal dataset)
+
+### Pre-processing
+Our pre-processing pipeline is described here. We first extract the first noun chunk of the caption using Spacy. Then this subject is fed into Owl-ViT-L to obtain bounding boxes. If there are 0 bounding boxes corresponding to a subject, we use the next caption from the dataset. If there are atleast two bounding boxes, we interpolate bounding boxes for the missing frames linearly. The dataset downloading is a bit complex, you need to follow the instructions [here](https://github.com/MikeWangWZHL/Paxion#dataset-setup). Download the dataset and run `generate_ssv2_st.py`.
+
+## Interactive Motion Control - IMC
+We generate bounding boxes for this dataset using the `generate_imc.py` file. The prompts are in `custom_prompts.csv` and `filtered_prompts.csv`.
+
+For more details regarding the datasets and evaluation strategy, please refer to the [Peekaboo paper](https://arxiv.org/abs/2312.07509).

data/custom_prompts.csv

@@ -0,0 +1,100 @@
+A butterfly resting on a flower.,Butterfly,Small,Square,Stationary
+A koala clinging to a eucalyptus tree.,Koala,Medium,Rectangle Vertical,Stationary
+A peacock displaying its feathers in a garden.,Peacock,Medium,Rectangle Horizontal,Stationary
+A frog sitting on a lily pad in a pond.,Frog,Small,Square,Stationary
+A deer standing in a snowy field.,Deer,Medium,Rectangle Horizontal,Stationary
+A horse grazing in a meadow.,Horse,Medium,Rectangle Horizontal,Stationary
+A squirrel holding an acorn in a park.,Squirrel,Small,Rectangle Vertical,Stationary
+A parrot perched on a branch in the rainforest.,Parrot,Small,Rectangle Vertical,Stationary
+A fox sitting in a forest clearing.,Fox,Medium,Rectangle Horizontal,Stationary
+A swan floating gracefully on a lake.,Swan,Medium,Rectangle Horizontal,Stationary
+A panda munching bamboo in a bamboo forest.,Panda,Large,Rectangle Vertical,Stationary
+A hummingbird hovering near a flower.,Hummingbird,Small,Rectangle Vertical,Stationary
+A penguin standing on an iceberg.,Penguin,Medium,Rectangle Vertical,Stationary
+A lion lying in the savanna grass.,Lion,Large,Rectangle Horizontal,Stationary
+An owl perched silently in a tree at night.,Owl,Medium,Rectangle Vertical,Stationary
+A goat standing on a rocky hillside.,Goat,Medium,Rectangle Horizontal,Stationary
+A dolphin just breaking the ocean surface.,Dolphin,Medium,Rectangle Horizontal,Stationary
+A camel resting in a desert landscape.,Camel,Large,Rectangle Horizontal,Stationary
+A kangaroo standing in the Australian outback.,Kangaroo,Medium,Rectangle Vertical,Stationary
+An eagle sitting atop a mountain cliff.,Eagle,Small,Rectangle Vertical,Stationary
+An ancient clock tower in a historic city square.,Clock Tower,Large,Rectangle Vertical,Stationary
+A rustic wooden bridge over a tranquil stream.,Wooden Bridge,Medium,Rectangle Horizontal,Stationary
+A grand piano in an elegant concert hall.,Grand Piano,Medium,Rectangle Horizontal,Stationary
+A vintage car parked in front of a classic diner.,Vintage Car,Medium,Rectangle Horizontal,Stationary
+A majestic lighthouse on a rocky coastline.,Lighthouse,Large,Rectangle Vertical,Stationary
+A colorful hot air balloon tethered to the ground.,Hot Air Balloon,Large,Rectangle Vertical,Stationary
+A medieval castle overlooking a scenic valley.,Castle,Large,Rectangle Horizontal,Stationary
+A traditional windmill in a field of tulips.,Windmill,Large,Rectangle Vertical,Stationary
+An intricate sculpture in a modern art museum.,Sculpture,Medium,Rectangle Vertical,Stationary
+A red British telephone box on a city street.,Telephone Box,Medium,Rectangle Vertical,Stationary
+A classic steam train stationed at an old railway platform.,Steam Train,Large,Rectangle Horizontal,Stationary
+An old-fashioned street lamp on a foggy night.,Street Lamp,Medium,Rectangle Vertical,Stationary
+A snow-covered cabin in a winter landscape.,Cabin,Medium,Rectangle Horizontal,Stationary
+A beautifully crafted sundial in a botanical garden.,Sundial,Small,Square,Stationary
+An ornate fountain in a public park.,Fountain,Medium,Square,Stationary
+A weathered rowboat on a peaceful lakeshore.,Rowboat,Small,Rectangle Horizontal,Stationary
+A detailed mural on the side of an urban building.,Mural,Large,Rectangle Horizontal,Stationary
+A historical monument in a busy city center.,Monument,Large,Rectangle Vertical,Stationary
+A charming gazebo in a lush garden.,Gazebo,Medium,Rectangle Horizontal,Stationary
+A striking skyscraper against a city skyline.,Skyscraper,Large,Rectangle Vertical,Stationary
+A cheetah sprinting across the savanna.,Cheetah,Medium,Rectangle Horizontal,Left to right
+A school of fish swimming in a coral reef.,School of Fish,Medium,Rectangle Horizontal,Zig-zag
+A hummingbird darting around a flower garden.,Hummingbird,Small,Square,Zig-zag
+A horse galloping through a meadow.,Horse,Large,Rectangle Horizontal,Left to right
+A squirrel scampering up a tree trunk.,Squirrel,Small,Rectangle Vertical,Up to down
+A flock of geese flying in a V-formation.,Flock of Geese,Large,Rectangle Horizontal,Left to right
+A rabbit hopping through a grassy field.,Rabbit,Small,Rectangle Horizontal,Zig-zag
+A dolphin leaping out of the ocean waves.,Dolphin,Medium,Rectangle Horizontal,Up to down
+A bee buzzing around a blooming sunflower.,Bee,Small,Square,Zig-zag
+A butterfly fluttering over a meadow of wildflowers.,Butterfly,Small,Square,Zig-zag
+A kangaroo bounding across the Australian outback.,Kangaroo,Medium,Rectangle Horizontal,Left to right
+A hawk soaring in the sky above a mountain range.,Hawk,Medium,Rectangle Horizontal,Left to right
+A spider spinning a web in the morning light.,Spider,Small,Square,Zig-zag
+A snake slithering through a tropical rainforest.,Snake,Medium,Rectangle Horizontal,Zig-zag
+A dog running to catch a frisbee in a park.,Dog,Medium,Rectangle Horizontal,Left to right
+A cat playfully chasing a ball of yarn.,Cat,Small,Square,Zig-zag
+A herd of elephants migrating across the African plains.,Herd of Elephants,Large,Rectangle Horizontal,Left to right
+A bat flitting through the night sky.,Bat,Small,Rectangle Horizontal,Zig-zag
+A group of penguins waddling on an Antarctic ice sheet.,Group of Penguins,Medium,Rectangle Horizontal,Left to right
+A monkey swinging from branch to branch in the jungle.,Monkey,Medium,Rectangle Horizontal,Zig-zag
+A woodpecker climbing up a tree trunk.,Woodpecker,Small,Rectangle Vertical,Up to down
+A squirrel descending a tree after gathering nuts.,Squirrel,Small,Rectangle Vertical,Up to down
+A snake slithering down a rocky hill.,Snake,Medium,Rectangle Vertical,Up to down
+A bird diving towards the water to catch fish.,Bird,Small,Rectangle Vertical,Up to down
+A monkey climbing up a vine in the rainforest.,Monkey,Medium,Rectangle Vertical,Up to down
+A cat jumping down from a high fence.,Cat,Small,Rectangle Vertical,Up to down
+An eagle descending from the sky to its nest.,Eagle,Medium,Rectangle Vertical,Up to down
+A frog leaping up to catch a fly.,Frog,Small,Rectangle Vertical,Up to down
+A spider descending on its web from a branch.,Spider,Small,Rectangle Vertical,Up to down
+A mountain goat scaling a steep cliff.,Mountain Goat,Medium,Rectangle Vertical,Up to down
+A koala climbing up a eucalyptus tree.,Koala,Small,Rectangle Vertical,Up to down
+A bear climbing down a tree after spotting a threat.,Bear,Large,Rectangle Vertical,Up to down
+A parrot flying upwards towards the treetops.,Parrot,Small,Rectangle Vertical,Up to down
+A squirrel jumping from one tree to another,Squirrel,Small,Rectangle Vertical,Up to down
+A bat swooping down from a cave's ceiling.,Bat,Small,Rectangle Vertical,Up to down
+A duck diving underwater in search of food.,Duck,Medium,Rectangle Vertical,Up to down
+A kangaroo hopping down a gentle slope.,Kangaroo,Medium,Rectangle Vertical,Up to down
+A rabbit burrowing downwards into its warren.,Rabbit,Small,Rectangle Vertical,Up to down
+A raccoon climbing up a city lamppost.,Raccoon,Medium,Rectangle Vertical,Up to down
+An owl swooping down on its prey during the night.,Owl,Medium,Rectangle Vertical,Up to down
+A train chugging along a mountainous landscape.,Train,Large,Rectangle Horizontal,Left to right
+A hot air balloon drifting across a clear sky.,Hot Air Balloon,Large,Rectangle Vertical,Up to down
+A sailboat gliding over the ocean waves.,Sailboat,Medium,Rectangle Horizontal,Left to right
+A vintage car cruising down a coastal highway.,Vintage Car,Medium,Rectangle Horizontal,Left to right
+A windmill turning its blades in a gentle breeze.,Windmill,Large,Rectangle Vertical,Left to right
+A Ferris wheel rotating at a lively carnival.,Ferris Wheel,Large,Rectangle Vertical,Left to right
+A satellite orbiting Earth in outer space.,Satellite,Small,Rectangle Horizontal,Left to right
+A red double-decker bus moving through London streets.,Double-Decker Bus,Large,Rectangle Vertical,Left to right
+A bicycle rider pedaling through a city park.,Bicycle Rider,Medium,Rectangle Horizontal,Left to right
+A skateboarder performing tricks at a skate park.,Skateboarder,Small,Rectangle Horizontal,Left to right
+A gondola floating down a Venetian canal.,Gondola,Medium,Rectangle Horizontal,Left to right
+A jet plane flying high in the sky.,Jet Plane,Large,Rectangle Horizontal,Left to right
+A helicopter hovering above a cityscape.,Helicopter,Medium,Rectangle Horizontal,Left to right
+A roller coaster looping in an amusement park.,Roller Coaster,Large,Rectangle Horizontal,Left to right
+A leaf falling gently from a tree.,Leaf,Small,Rectangle Vertical,Up to down
+A mechanical clock's hands ticking forward.,Clock,Medium,Square,Left to right
+A streetcar trundling down tracks in a historic district.,Streetcar,Medium,Rectangle Horizontal,Left to right
+A rocket launching into space from a launchpad.,Rocket,Large,Rectangle Vertical,Up to down
+A paper plane gliding in the air.,Paper Plane,Small,Rectangle Horizontal,Left to right
+An escalator carrying people up in a shopping mall.,Escalator,Large,Rectangle Vertical,Up to down

data/filtered_prompts.txt

@@ -0,0 +1,125 @@
+A woodpecker climbing up a tree trunk.
+A woodpecker climbing up a tree trunk.
+A woodpecker climbing up a tree trunk.
+A squirrel descending a tree after gathering nuts.
+A squirrel descending a tree after gathering nuts.
+A squirrel descending a tree after gathering nuts.
+A bird diving towards the water to catch fish.
+A bird diving towards the water to catch fish.
+A bird diving towards the water to catch fish.
+A frog leaping up to catch a fly.
+A frog leaping up to catch a fly.
+A frog leaping up to catch a fly.
+A spider descending on its web from a branch.
+A spider descending on its web from a branch.
+A spider descending on its web from a branch.
+A parrot flying upwards towards the treetops.
+A parrot flying upwards towards the treetops.
+A parrot flying upwards towards the treetops.
+A squirrel jumping from one tree to another
+A squirrel jumping from one tree to another
+A squirrel jumping from one tree to another
+A rabbit burrowing downwards into its warren.
+A rabbit burrowing downwards into its warren.
+A rabbit burrowing downwards into its warren.
+A satellite orbiting Earth in outer space.
+A satellite orbiting Earth in outer space.
+A satellite orbiting Earth in outer space.
+A skateboarder performing tricks at a skate park.
+A skateboarder performing tricks at a skate park.
+A skateboarder performing tricks at a skate park.
+A leaf falling gently from a tree.
+A leaf falling gently from a tree.
+A leaf falling gently from a tree.
+A paper plane gliding in the air.
+A paper plane gliding in the air.
+A bear climbing down a tree after spotting a threat.
+A bear climbing down a tree after spotting a threat.
+A bear climbing down a tree after spotting a threat.
+A duck diving underwater in search of food.
+A duck diving underwater in search of food.
+A duck diving underwater in search of food.
+A kangaroo hopping down a gentle slope.
+A kangaroo hopping down a gentle slope.
+A kangaroo hopping down a gentle slope.
+An owl swooping down on its prey during the night.
+An owl swooping down on its prey during the night.
+An owl swooping down on its prey during the night.
+A hot air balloon drifting across a clear sky.
+A hot air balloon drifting across a clear sky.
+A hot air balloon drifting across a clear sky.
+A sailboat gliding over the ocean waves.
+A sailboat gliding over the ocean waves.
+A sailboat gliding over the ocean waves.
+A vintage car cruising down a coastal highway.
+A vintage car cruising down a coastal highway.
+A vintage car cruising down a coastal highway.
+A red double-decker bus moving through London streets.
+A red double-decker bus moving through London streets.
+A red double-decker bus moving through London streets.
+A jet plane flying high in the sky.
+A jet plane flying high in the sky.
+A jet plane flying high in the sky.
+A helicopter hovering above a cityscape.
+A helicopter hovering above a cityscape.
+A helicopter hovering above a cityscape.
+A roller coaster looping in an amusement park.
+A roller coaster looping in an amusement park.
+A roller coaster looping in an amusement park.
+A streetcar trundling down tracks in a historic district.
+A streetcar trundling down tracks in a historic district.
+A streetcar trundling down tracks in a historic district.
+A rocket launching into space from a launchpad.
+A rocket launching into space from a launchpad.
+A rocket launching into space from a launchpad.
+A deer standing in a snowy field.
+A deer standing in a snowy field.
+A deer standing in a snowy field.
+A horse grazing in a meadow.
+A horse grazing in a meadow.
+A horse grazing in a meadow.
+A fox sitting in a forest clearing.
+A fox sitting in a forest clearing.
+A fox sitting in a forest clearing.
+A swan floating gracefully on a lake.
+A swan floating gracefully on a lake.
+A swan floating gracefully on a lake.
+A panda munching bamboo in a bamboo forest.
+A panda munching bamboo in a bamboo forest.
+A panda munching bamboo in a bamboo forest.
+A penguin standing on an iceberg.
+A penguin standing on an iceberg.
+A penguin standing on an iceberg.
+A lion lying in the savanna grass.
+A lion lying in the savanna grass.
+A lion lying in the savanna grass.
+An owl perched silently in a tree at night.
+An owl perched silently in a tree at night.
+An owl perched silently in a tree at night.
+A dolphin just breaking the ocean surface.
+A dolphin just breaking the ocean surface.
+A dolphin just breaking the ocean surface.
+A camel resting in a desert landscape.
+A camel resting in a desert landscape.
+A camel resting in a desert landscape.
+A kangaroo standing in the Australian outback.
+A kangaroo standing in the Australian outback.
+A kangaroo standing in the Australian outback.
+A grand piano in an elegant concert hall.
+A grand piano in an elegant concert hall.
+A grand piano in an elegant concert hall.
+A vintage car parked in front of a classic diner.
+A vintage car parked in front of a classic diner.
+A vintage car parked in front of a classic diner.
+A colorful hot air balloon tethered to the ground.
+A colorful hot air balloon tethered to the ground.
+A colorful hot air balloon tethered to the ground.
+A red British telephone box on a city street.
+A red British telephone box on a city street.
+A red British telephone box on a city street.
+A classic steam train stationed at an old railway platform.
+A classic steam train stationed at an old railway platform.
+A classic steam train stationed at an old railway platform.
+An old-fashioned street lamp on a foggy night.
+An old-fashioned street lamp on a foggy night.
+An old-fashioned street lamp on a foggy night.