annotations/README.md

Assembly101 annotations

The annotations are divided into 2 granularities:

• fine-grained-annotations: used for Action Recognition and Action Anticipation benchmarks
• coarse-annotations: used for Temporal Action Segmentation benchmark

Raw videos are 60fps but the frame numbers in all the annotations are provided after extracting them from the raw video at 30fps.

If you use our dataset, kindly cite:

@article{sener2022assembly101,
    title = {Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities},
    author = {F. Sener and D. Chatterjee and D. Shelepov and K. He and D. Singhania and R. Wang and A. Yao},
    journal = {CVPR 2022},
}

Fine-Grained Annotations

We have 1380 fine-grained actions composed of a combination of 90 objects and 24 verbs.

The files present under the fine-grained-annotations folder are:

• actions.csv
• train.csv
• validation.csv
• test.csv
• head_actions.txt

File structure

actions.csv

This csv provides the list of actions and its mapping to the corresponding verbs and nouns. The columns are:

Column name	Type	example	Description
id	int	12	row index
action_id	int	380	numeric action ID
verb_id	int	7	corresponding numeric verb ID
noun_id	int	22	corresponding numeric noun ID
action_cls	str	remove excavator arm	action name
verb_cls	str	remove	verb name
noun_cls	str	excavator arm	object name

---

train.csv / validation.csv / test.csv

This csv provides the list of fine-grained action segments with their corresponding annotations that are split between training, validation and test. The columns are:

Column name	Type	example	Description
id	int	7	row (segment) index
video	str	nusar-2021_action_both_9033-a30_9033_user_id_2021-02-04_131528/C10404_rgb.mp4	the video name in the format of {sequence_name}/{view_name}.mp4
start_frame	int	84	#frame (@30fps) when the fine-grained action starts
end_frame	int	111	#frame (@30fps) when the fine-grained action ends
action_id	int	10	numeric action ID of the segment
verb_id	int	18	numeric verb ID of the segment
noun_id	int	27	numeric noun ID of the segment
action_cls	str	clap hand	action name of the segment
verb_cls	str	clap	verb name of the segment
noun_cls	str	hand	object name of the segment
toy_id	str	a30	numeric ID of the toy in the segment
toy_name	str	suv	name of the toy in the segment
is_shared	bool	0	is the toy shared between the training and evaluation (validation/test) set? 0 : not shared 1 : shared not shared corresponds to zero-shot classes
is_RGB	bool	1	is the view fixed (RGB) or monochrome? 0 : monochrome 1 : fixed (RGB)

---

head_actions.txt

This text file provides the 142 head action classes among a total of 1380 fine-grained action classes. The rest are all tail classes.

Coarse Annotations

We have 202 coarse actions composed of a combination of 61 objects and 11 verbs.

The files/folders present under the coarse-annotations folder are:

• actions.csv
• coarse_seq_views.txt
• tail_actions.txt
• coarse_labels (folder)
• coarse_splits (folder)

Folder structure

coarse_labels

coarse_labels
├── assembly_nusar-2021_action_both_9011-a01_9011_user_id_2021-02-01_153724.txt
├── assembly_nusar-2021_action_both_9011-b06b_9011_user_id_2021-02-01_154253.txt
├── .
├── .
├── .
├── disassembly_nusar-2021_action_both_9011-a01_9011_user_id_2021-02-01_153724.txt
├── disassembly_nusar-2021_action_both_9011-b06b_9011_user_id_2021-02-01_154253.txt
├── .
├── .
├── .
├── disassembly_nusar-2021_action_both_9086-c14a_9086_user_id_2021-02-16_153910.txt

coarse_splits

coarse_splits
├── train_coarse_assembly.txt
├── train_coarse_disassembly.txt
├── val_coarse_assembly.txt
├── val_coarse_disassembly.txt
├── test_coarse_assembly.txt
├── test_coarse_disassembly.txt

File structure

actions.csv

This csv provides the list of actions and its mapping to the corresponding verbs and nouns. The columns are:

Column name	Type	example	Description
action_id	int	21	numeric action ID
verb_id	int	0	corresponding numeric verb ID
noun_id	int	11	corresponding numeric noun ID
action_cls	str	attach arm connector	action name
verb_cls	str	attach	verb name
noun_cls	str	arm connector	object name

---

coarse_splits/{train/validation}_coarse_{assembly/disassembly}.txt

This text provides the list of coarse action segments with their corresponding annotations that are split between training, validation and test. The columns are:

Column Number	Column name	Type	example	Description
0	sequence	str	assembly_nusar-2021_action_both_9033-a30_9033_user_id_2021-02-04_132852.txt	the video name in the format of {assembly/disassembly}_{sequence_name}.txt
1	is_shared	str	notshared	is the toy shared between the training and evaluation (validation/test) set? not shared corresponds to zero-shot classes
2	toy_id	str	a30	numeric ID of the toy in the segment
3	toy_name	str	suv	name of the toy in the segment

---

coarse_splits/test_coarse_{assembly/disassembly}.txt

This text provides the list of coarse action segments for each sequence. The columns are:

Column Number	Column name	Type	example	Description
0	sequence	str	assembly_nusar-2021_action_both_9033-a30_9033_user_id_2021-02-04_132852.txt	the video name in the format of {assembly/disassembly}_{sequence_name}.txt
1	is_shared	str	notshared	is the toy shared between the training and evaluation (validation/test) set? not shared corresponds to zero-shot classes

---

coarse_labels/{assembly/disassembly}_{sequence}.txt

This text provides the list of coarse action segments with their corresponding annotations that are split between training, validation and test. The columns are:

Column Number	Column name	Type	example	Description
0	start_frame	str	000006833	#frame (@30fps) when the coarse action starts
1	end_frame	str	000007554	#frame (@30fps) when the coarse action ends
2	action_cls	str	attempt to screw chassis	action name of the segment

---

coarse_seq_views.txt

A list of all the videos that exist for a sequence in the format of {assembly/disassembly}_{sequence_name}/{view_name}.mp4
Most of the sequences have 12 views (8 fixed and 4 egocentric), hence most of the sequences have 12 videos associated to it.

---

tail_actions.txt

This text file provides the 171 tail action classes among a total of 202 coarse action classes. The rest are all head classes.

License

Assembly101 is licensed by us under the Creative Commons Attribution-NonCommercial 4.0 International License, found here. The terms are :

• Attribution : You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
• NonCommercial : You may not use the material for commercial purposes.