| ### TalkSet Generation | |
| You can check the 'train.txt' and 'test.txt' to generate TalkSet by your own. | |
| This `generate_TalkSet.py` code is just used for your reference, I did not check it recently. | |
| Input the LRS3, VoxCeleb2, 3 list files in `lists_in` | |
| Output TalkSet, train.txt, test.txt (Here the test set is the validation set actually) | |
| ### Usage: | |
| Set the following parser based on the location of your data: | |
| `out_path`: the output TalkSet location | |
| `Vox_audio`: Location of the Vox2, training set, audio location | |
| `Vox_video`: Location of the Vox2, training set, video location | |
| `lrs3_audio`: Location of the LRS3, audio location | |
| `lrs3_video`: Location of the LRS3, video location | |
| `task`: The part of the TalkSet you want to generate, eg: TAudio | |
| `num_cpu`: The num of the threads, higher will be faster, based on your PC performance, eg: 10 | |
| ``` | |
| python TalkSet/generate_TalkSet.py --task 'TAudio' | |
| python TalkSet/generate_TalkSet.py --task 'FAudio' | |
| python TalkSet/generate_TalkSet.py --task 'TFAudio' | |
| python TalkSet/generate_TalkSet.py --task 'TSilence' | |
| python TalkSet/generate_TalkSet.py --task 'FSilence' | |
| python TalkSet/generate_TalkSet.py --task 'Fusion' | |
| ``` | |
| ### Description: | |
| For `lists_out\*.txt` files: | |
| - The 1st row is the face clips data type, | |
| - TAudio: audio is active, lip is moving, audio and lip are sync | |
| - FAudio: audio is active, lip is moving, audio and lip are not sync (Speech from others) | |
| - TFAudio: one part is 'TAudio', the other part is 'FAudio' | |
| - TSilence: one part is 'TAudio', in the other part, audio is non-active, lip is not moving | |
| - FSilence: one part is 'silence'(audio is non-active, lip is not moving), in the other part, audio is active, lip is not moving (Speech from others) | |
| - The 2nd row is the path for the audio file (filename started from 'silence' is the data from LRS3, filename started from 'id.....' is the data from VoxCeleb2) | |
| - The 3rd row is the path for the video file | |
| - The 4th row is the length(seconds) of this data | |
| - The 5th row is the start of 'active' clip (in FSilence, it presents the 'silence' part) | |
| - The 6th row is the end of 'active' clip | |
| - The 7th row is the start of 'non-active' clip (in FSilence, it presents the 'speech from others' part) | |
| - The 8th row is the end of 'non-active' clip | |
| - The 9th row is the file ID | |
| The dataset generated will not be fixed each time because we randomly select FSlience data, and the change point is the random number. We believe the result will be similar. The whole time to generate the TalkSet will use about 3 to 6 hours in our experiments. |