Buckets:

RyanWW
/

assembly101-bucket

Files

xet

RyanWW/assembly101-bucket / annotations /README.md

RyanWW

9 days ago

preview code

download

raw

7.85 kB

# Assembly101 annotations

The annotations are divided into 2 granularities:
- `fine-grained-annotations`: used for Action Recognition and Action Anticipation benchmarks
- `coarse-annotations`: used for Temporal Action Segmentation benchmark

```
Raw videos are 60fps but the frame numbers in all the annotations are provided after extracting them from the raw video at 30fps.
```

If you use our dataset, kindly cite:
```
@article{sener2022assembly101,
    title = {Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities},
    author = {F. Sener and D. Chatterjee and D. Shelepov and K. He and D. Singhania and R. Wang and A. Yao},
    journal = {CVPR 2022},
}
```

## Fine-Grained Annotations
We have 1380 fine-grained actions composed of a combination of 90 objects and 24 verbs.

The files present under the `fine-grained-annotations` folder are:
- `actions.csv`
- `train.csv`
- `validation.csv`
- `test.csv`
- `head_actions.txt`

### File structure

### actions.csv
This csv provides the list of actions and its mapping to the corresponding verbs and nouns. The columns are:
| Column name | Type | example | Description | 
|-------------|------|---------|-------------|
| id | int | 12 | row index |
| action_id | int | 380 | numeric action ID |
| verb_id | int | 7 | corresponding numeric verb ID |
| noun_id | int | 22 | corresponding numeric noun ID |
| action_cls | str | remove excavator arm | action name |
| verb_cls | str | remove | verb name |
| noun_cls | str | excavator arm | object name |

- - -

### train.csv / validation.csv / test.csv
This csv provides the list of fine-grained action segments with their corresponding annotations that are split between training, validation and test. The columns are:
| Column name | Type | example | Description | 
|-------------|------|---------|-------------|
| id | int | 7 | row (segment) index |
| video | str | nusar-2021_action_both_9033-a30_9033_user_id_2021-02-04_131528/C10404_rgb.mp4 | the video name in the format of {sequence_name}/{view_name}.mp4 |
| start_frame | int | 84 | #frame **(@30fps)** when the fine-grained action starts |
| end_frame | int | 111 | #frame **(@30fps)** when the fine-grained action ends |
| action_id | int | 10 | numeric action ID of the segment |
| verb_id | int | 18 | numeric verb ID of the segment |
| noun_id | int | 27 | numeric noun ID of the segment |
| action_cls | str | clap hand | action name of the segment |
| verb_cls | str | clap | verb name of the segment |
| noun_cls | str | hand | object name of the segment |
| toy_id | str | a30 | numeric ID of the toy in the segment |
| toy_name | str | suv | name of the toy in the segment |
| is_shared | bool | 0 | is the toy shared between the training and evaluation (validation/test) set? <br> `0` : not shared <br> `1` : shared <br> not shared corresponds to zero-shot classes|
| is_RGB | bool | 1 | is the view fixed (RGB) or monochrome? <br> `0` : monochrome <br> `1` : fixed (RGB)|
- - -
### head_actions.txt
This text file provides the 142 head action classes among a total of 1380 fine-grained action classes. The rest are all tail classes.


## Coarse Annotations
We have 202 coarse actions composed of a combination of 61 objects and 11 verbs.

The files/folders present under the `coarse-annotations` folder are:
- `actions.csv`
- `coarse_seq_views.txt`
- `tail_actions.txt`
- `coarse_labels` (folder)
- `coarse_splits` (folder)

### Folder structure
### coarse_labels
```
coarse_labels
├── assembly_nusar-2021_action_both_9011-a01_9011_user_id_2021-02-01_153724.txt
├── assembly_nusar-2021_action_both_9011-b06b_9011_user_id_2021-02-01_154253.txt
├── .
├── .
├── .
├── disassembly_nusar-2021_action_both_9011-a01_9011_user_id_2021-02-01_153724.txt
├── disassembly_nusar-2021_action_both_9011-b06b_9011_user_id_2021-02-01_154253.txt
├── .
├── .
├── .
├── disassembly_nusar-2021_action_both_9086-c14a_9086_user_id_2021-02-16_153910.txt
```
### coarse_splits
```
coarse_splits
├── train_coarse_assembly.txt
├── train_coarse_disassembly.txt
├── val_coarse_assembly.txt
├── val_coarse_disassembly.txt
├── test_coarse_assembly.txt
├── test_coarse_disassembly.txt
```

### File structure

### actions.csv
This csv provides the list of actions and its mapping to the corresponding verbs and nouns. The columns are:
| Column name | Type | example | Description | 
|-------------|------|---------|-------------|
| action_id | int | 21 | numeric action ID |
| verb_id | int | 0 | corresponding numeric verb ID |
| noun_id | int | 11 | corresponding numeric noun ID |
| action_cls | str | attach arm connector | action name |
| verb_cls | str | attach | verb name |
| noun_cls | str | arm connector | object name |

- - -

### coarse_splits/{`train/validation`}\_coarse\_{`assembly/disassembly`}.txt
This text provides the list of coarse action segments with their corresponding annotations that are split between training, validation and test. The columns are:
| Column Number | Column name | Type | example | Description | 
|---------------|-------------|------|---------|-------------|
|0 | sequence | str | assembly_nusar-2021_action_both_9033-a30_9033_user_id_2021-02-04_132852.txt | the video name in the format of {`assembly/disassembly`}_{sequence_name}.txt |
| 1 | is_shared | str | notshared | is the toy shared between the training and evaluation (validation/test) set? <br> not shared corresponds to zero-shot classes |
| 2 | toy_id | str | a30 | numeric ID of the toy in the segment |
| 3 | toy_name | str | suv | name of the toy in the segment |
- - -
### coarse_splits/test\_coarse\_{`assembly/disassembly`}.txt
This text provides the list of coarse action segments for each sequence. The columns are:
| Column Number | Column name | Type | example | Description | 
|---------------|-------------|------|---------|-------------|
|0 | sequence | str | assembly_nusar-2021_action_both_9033-a30_9033_user_id_2021-02-04_132852.txt | the video name in the format of {`assembly/disassembly`}_{sequence_name}.txt |
| 1 | is_shared | str | notshared | is the toy shared between the training and evaluation (validation/test) set? <br> not shared corresponds to zero-shot classes |

- - -
### coarse_labels/{`assembly/disassembly`}_{sequence}.txt
This text provides the list of coarse action segments with their corresponding annotations that are split between training, validation and test. The columns are:
| Column Number | Column name | Type | example | Description | 
|---------------|-------------|------|---------|-------------|
| 0 | start_frame | str | 000006833 | #frame **(@30fps)** when the coarse action starts |
| 1 | end_frame | str | 000007554 | #frame **(@30fps)** when the coarse action ends |
| 2 | action_cls | str | attempt to screw chassis | action name of the segment |
- - -
### coarse_seq_views.txt
A list of all the videos that exist for a sequence in the format of {`assembly/disassembly`}_{sequence_name}/{view_name}.mp4 <br> Most of the sequences have 12 views (8 fixed and 4 egocentric), hence most of the sequences have 12 videos associated to it.
- - -

### tail_actions.txt
This text file provides the 171 tail action classes among a total of 202 coarse action classes. The rest are all head classes.
# License
Assembly101 is licensed by us under the Creative Commons Attribution-NonCommercial 4.0 International License, found [here](https://creativecommons.org/licenses/by-nc/4.0/). The terms are :
- **Attribution** : You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- **NonCommercial** : You may not use the material for commercial purposes.

Xet Storage Details

Size:: 7.85 kB
Xet hash:: ee562e0ddb724cdbef4980d4744564456f3d727c8da53788691075b97384e0d2

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# Assembly101 annotations

	The annotations are divided into 2 granularities:
	- `fine-grained-annotations`: used for Action Recognition and Action Anticipation benchmarks
	- `coarse-annotations`: used for Temporal Action Segmentation benchmark

	```
	Raw videos are 60fps but the frame numbers in all the annotations are provided after extracting them from the raw video at 30fps.
	```

	If you use our dataset, kindly cite:
	```
	@article{sener2022assembly101,
	title = {Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities},
	author = {F. Sener and D. Chatterjee and D. Shelepov and K. He and D. Singhania and R. Wang and A. Yao},
	journal = {CVPR 2022},
	}
	```

	## Fine-Grained Annotations
	We have 1380 fine-grained actions composed of a combination of 90 objects and 24 verbs.

	The files present under the `fine-grained-annotations` folder are:
	- `actions.csv`
	- `train.csv`
	- `validation.csv`
	- `test.csv`
	- `head_actions.txt`

	### File structure

	### actions.csv
	This csv provides the list of actions and its mapping to the corresponding verbs and nouns. The columns are:
	\| Column name \| Type \| example \| Description \|
	\|-------------\|------\|---------\|-------------\|
	\| id \| int \| 12 \| row index \|
	\| action_id \| int \| 380 \| numeric action ID \|
	\| verb_id \| int \| 7 \| corresponding numeric verb ID \|
	\| noun_id \| int \| 22 \| corresponding numeric noun ID \|
	\| action_cls \| str \| remove excavator arm \| action name \|
	\| verb_cls \| str \| remove \| verb name \|
	\| noun_cls \| str \| excavator arm \| object name \|

	- - -

	### train.csv / validation.csv / test.csv
	This csv provides the list of fine-grained action segments with their corresponding annotations that are split between training, validation and test. The columns are:
	\| Column name \| Type \| example \| Description \|
	\|-------------\|------\|---------\|-------------\|
	\| id \| int \| 7 \| row (segment) index \|
	\| video \| str \| nusar-2021_action_both_9033-a30_9033_user_id_2021-02-04_131528/C10404_rgb.mp4 \| the video name in the format of {sequence_name}/{view_name}.mp4 \|
	\| start_frame \| int \| 84 \| #frame (@30fps) when the fine-grained action starts \|
	\| end_frame \| int \| 111 \| #frame (@30fps) when the fine-grained action ends \|
	\| action_id \| int \| 10 \| numeric action ID of the segment \|
	\| verb_id \| int \| 18 \| numeric verb ID of the segment \|
	\| noun_id \| int \| 27 \| numeric noun ID of the segment \|
	\| action_cls \| str \| clap hand \| action name of the segment \|
	\| verb_cls \| str \| clap \| verb name of the segment \|
	\| noun_cls \| str \| hand \| object name of the segment \|
	\| toy_id \| str \| a30 \| numeric ID of the toy in the segment \|
	\| toy_name \| str \| suv \| name of the toy in the segment \|
	\| is_shared \| bool \| 0 \| is the toy shared between the training and evaluation (validation/test) set? <br> `0` : not shared <br> `1` : shared <br> not shared corresponds to zero-shot classes\|
	\| is_RGB \| bool \| 1 \| is the view fixed (RGB) or monochrome? <br> `0` : monochrome <br> `1` : fixed (RGB)\|

	- - -

	### head_actions.txt
	This text file provides the 142 head action classes among a total of 1380 fine-grained action classes. The rest are all tail classes.


	## Coarse Annotations
	We have 202 coarse actions composed of a combination of 61 objects and 11 verbs.

	The files/folders present under the `coarse-annotations` folder are:
	- `actions.csv`
	- `coarse_seq_views.txt`
	- `tail_actions.txt`
	- `coarse_labels` (folder)
	- `coarse_splits` (folder)

	### Folder structure
	### coarse_labels
	```
	coarse_labels
	├── assembly_nusar-2021_action_both_9011-a01_9011_user_id_2021-02-01_153724.txt
	├── assembly_nusar-2021_action_both_9011-b06b_9011_user_id_2021-02-01_154253.txt
	├── .
	├── .
	├── .
	├── disassembly_nusar-2021_action_both_9011-a01_9011_user_id_2021-02-01_153724.txt
	├── disassembly_nusar-2021_action_both_9011-b06b_9011_user_id_2021-02-01_154253.txt
	├── .
	├── .
	├── .
	├── disassembly_nusar-2021_action_both_9086-c14a_9086_user_id_2021-02-16_153910.txt
	```

	### coarse_splits
	```
	coarse_splits
	├── train_coarse_assembly.txt
	├── train_coarse_disassembly.txt
	├── val_coarse_assembly.txt
	├── val_coarse_disassembly.txt
	├── test_coarse_assembly.txt
	├── test_coarse_disassembly.txt
	```

	### File structure

	### actions.csv
	This csv provides the list of actions and its mapping to the corresponding verbs and nouns. The columns are:
	\| Column name \| Type \| example \| Description \|
	\|-------------\|------\|---------\|-------------\|
	\| action_id \| int \| 21 \| numeric action ID \|
	\| verb_id \| int \| 0 \| corresponding numeric verb ID \|
	\| noun_id \| int \| 11 \| corresponding numeric noun ID \|
	\| action_cls \| str \| attach arm connector \| action name \|
	\| verb_cls \| str \| attach \| verb name \|
	\| noun_cls \| str \| arm connector \| object name \|

	- - -

	### coarse_splits/{`train/validation`}\_coarse\_{`assembly/disassembly`}.txt
	This text provides the list of coarse action segments with their corresponding annotations that are split between training, validation and test. The columns are:
	\| Column Number \| Column name \| Type \| example \| Description \|
	\|---------------\|-------------\|------\|---------\|-------------\|
	\|0 \| sequence \| str \| assembly_nusar-2021_action_both_9033-a30_9033_user_id_2021-02-04_132852.txt \| the video name in the format of {`assembly/disassembly`}_{sequence_name}.txt \|
	\| 1 \| is_shared \| str \| notshared \| is the toy shared between the training and evaluation (validation/test) set? <br> not shared corresponds to zero-shot classes \|
	\| 2 \| toy_id \| str \| a30 \| numeric ID of the toy in the segment \|
	\| 3 \| toy_name \| str \| suv \| name of the toy in the segment \|
	- - -

	### coarse_splits/test\_coarse\_{`assembly/disassembly`}.txt
	This text provides the list of coarse action segments for each sequence. The columns are:
	\| Column Number \| Column name \| Type \| example \| Description \|
	\|---------------\|-------------\|------\|---------\|-------------\|
	\|0 \| sequence \| str \| assembly_nusar-2021_action_both_9033-a30_9033_user_id_2021-02-04_132852.txt \| the video name in the format of {`assembly/disassembly`}_{sequence_name}.txt \|
	\| 1 \| is_shared \| str \| notshared \| is the toy shared between the training and evaluation (validation/test) set? <br> not shared corresponds to zero-shot classes \|

	- - -
	### coarse_labels/{`assembly/disassembly`}_{sequence}.txt
	This text provides the list of coarse action segments with their corresponding annotations that are split between training, validation and test. The columns are:
	\| Column Number \| Column name \| Type \| example \| Description \|
	\|---------------\|-------------\|------\|---------\|-------------\|
	\| 0 \| start_frame \| str \| 000006833 \| #frame (@30fps) when the coarse action starts \|
	\| 1 \| end_frame \| str \| 000007554 \| #frame (@30fps) when the coarse action ends \|
	\| 2 \| action_cls \| str \| attempt to screw chassis \| action name of the segment \|
	- - -

	### coarse_seq_views.txt
	A list of all the videos that exist for a sequence in the format of {`assembly/disassembly`}_{sequence_name}/{view_name}.mp4 <br> Most of the sequences have 12 views (8 fixed and 4 egocentric), hence most of the sequences have 12 videos associated to it.
	- - -

	### tail_actions.txt
	This text file provides the 171 tail action classes among a total of 202 coarse action classes. The rest are all head classes.


	# License
	Assembly101 is licensed by us under the Creative Commons Attribution-NonCommercial 4.0 International License, found [here](https://creativecommons.org/licenses/by-nc/4.0/). The terms are :
	- Attribution : You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
	- NonCommercial : You may not use the material for commercial purposes.