data/README.md

# Data Directory

This directory is intentionally empty in git. You do not have to store the raw
dataset here, but you can if you want.

The training/evaluation scripts read small prepared JSONL files from here:

- `asl_citizen_train.jsonl`
- `asl_citizen_val.jsonl`
- `how2sign_train.jsonl`
- `how2sign_val.jsonl`

Each JSONL row stores a path to a local video file and the target text. Paths
may be absolute or relative to the repository root, but no script hard-codes
local machine paths.

## Recommended Local Layout

If you want a simple layout, put downloaded datasets under `data/raw/`:

```text
data/
  raw/
    asl_citizen/
      metadata.csv
      videos/
        example_001.mp4
        example_002.mp4
```

Then run:

```bash
python scripts/prepare_asl_citizen.py \
  --metadata data/raw/asl_citizen/metadata.csv \
  --video_root data/raw/asl_citizen/videos \
  --out_train data/asl_citizen_train.jsonl \
  --out_val data/asl_citizen_val.jsonl
```

If your dataset is somewhere else, leave it there and pass those paths instead.
The preparation script does not copy videos. It only creates JSONL rows pointing
to the video files.

## What Metadata Needs

For ASL Citizen, the metadata file can be CSV, JSON, or JSONL. It needs:

- a video column: `video_path`, `path`, `file`, `filename`, or `video`
- a label column: `label`, `gloss`, `sign`, `sign_label`, or `text`
- optionally a split column: `split` or `subset`

If there is no split column, the script makes a train/validation split using
`--val_fraction`.