File size: 1,525 Bytes
babffc8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | # Data Directory
This directory is intentionally empty in git. You do not have to store the raw
dataset here, but you can if you want.
The training/evaluation scripts read small prepared JSONL files from here:
- `asl_citizen_train.jsonl`
- `asl_citizen_val.jsonl`
- `how2sign_train.jsonl`
- `how2sign_val.jsonl`
Each JSONL row stores a path to a local video file and the target text. Paths
may be absolute or relative to the repository root, but no script hard-codes
local machine paths.
## Recommended Local Layout
If you want a simple layout, put downloaded datasets under `data/raw/`:
```text
data/
raw/
asl_citizen/
metadata.csv
videos/
example_001.mp4
example_002.mp4
```
Then run:
```bash
python scripts/prepare_asl_citizen.py \
--metadata data/raw/asl_citizen/metadata.csv \
--video_root data/raw/asl_citizen/videos \
--out_train data/asl_citizen_train.jsonl \
--out_val data/asl_citizen_val.jsonl
```
If your dataset is somewhere else, leave it there and pass those paths instead.
The preparation script does not copy videos. It only creates JSONL rows pointing
to the video files.
## What Metadata Needs
For ASL Citizen, the metadata file can be CSV, JSON, or JSONL. It needs:
- a video column: `video_path`, `path`, `file`, `filename`, or `video`
- a label column: `label`, `gloss`, `sign`, `sign_label`, or `text`
- optionally a split column: `split` or `subset`
If there is no split column, the script makes a train/validation split using
`--val_fraction`.
|