Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models

HM-World

HM-World is a dataset for hybrid memory in dynamic video world models. It provides video sequences, foreground masks, camera pose annotations, character pose annotations, event timestamps, and text captions for each sample.

Usage

Merge the split archives and extract the dataset with:

cat HM-World_* | tar -xzvf -

Dataset Structure

HM-World/
├── sample1/
│   ├── cond.mp4
│   ├── tgt.mp4
│   ├── cond_mask.mp4
│   ├── tgt_mask.mp4
│   ├── camera.json
│   ├── character.json
│   └── check.json
├── sample2/
│   └── ...
├── ...
└── caption.txt

File Description

• cond.mp4: condition video.
• tgt.mp4: target video.
• cond_mask.mp4: foreground mask video for the condition video.
• tgt_mask.mp4: foreground mask video for the target video.
• camera.json: camera pose information.
• character.json: character pose information.
• check.json: timestamps that record when the subjects enters or leaves the frame.
• caption.txt: captions for all samples in the dataset.