File size: 3,522 Bytes
a097504 15240f0 a097504 15240f0 a097504 15240f0 a097504 15240f0 a097504 15240f0 a097504 15240f0 a097504 15240f0 a097504 15240f0 a097504 15240f0 a097504 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 | # Data Provenance
This document records the datasets used or staged during FastFace phase 1. It is
intended to make the training data transparent without redistributing raw
third-party images from this repository.
## Policy
- Do not commit raw face images to GitHub.
- Do not upload third-party raw datasets unless the upstream license explicitly
allows redistribution.
- Keep dataset acquisition scripts, manifest builders, row counts, and source
notes in this repository.
- Keep large raw data under `data` on the GPU host.
- Use public datasets only for the allowed research/evaluation/deployment scope
after license review.
## Staged Sources
| Dataset | Status | Rows | Raw/Staging Path | Notes |
| --- | --- | ---: | --- | --- |
| FairFace | complete | 97,698 | `data/raw/fairface` | Official `margin025` archive and train/val labels. |
| UTKFace | complete | 23,705 | `data/raw/utkface` | Public Hugging Face mirror of cropped images. |
| IMDB-clean | complete | 285,946 | `data/raw/imdb_clean` and `third_party/imdb-clean` | Original IMDB-WIKI images plus MiVOLO IMDB annotations and face boxes. |
| Lagenda-HF | staged | 12,409 | `data/raw/lagenda_hf_uaebn` | Hugging Face mirror; requires `bbox_face` crop. Not promoted in phase 1. |
## Manifest Paths
```text
data/manifests/fairface.jsonl
data/manifests/utkface.jsonl
data/manifests/imdb_clean.jsonl
data/manifests/lagenda_hf_uaebn.jsonl
```
## Source Notes
### FairFace
- Source: FairFace repository and paper.
- Purpose: gender training, public validation, fairness-oriented source slice.
- Labels: exact gender; age range.
- Phase-1 issue: FairFace age labels are ranges, so age MAE is only directional.
### UTKFace
- Source: public UTKFace cropped image mirror.
- Purpose: exact-age and gender training/validation.
- Labels: gender and exact age parsed from filename.
- Phase-1 issue: license/redistribution must be reviewed before any broader
product use.
### IMDB-clean
- Sources:
- IMDB-clean repository staged under `third_party/imdb-clean`.
- MiVOLO IMDB annotations staged as
`data/raw/imdb_clean/mivolo_imdb_annotations.zip`.
- Original IMDB-WIKI tar files downloaded from ETH with segmented `aria2c`.
- Purpose: exact-age/gender data expansion and teacher/student training.
- Labels: exact age, exact gender, face boxes.
- Completed state:
- 460,723 raw images.
- 285,946 valid manifest rows.
- 183,887 train rows.
- 102,059 validation rows.
- No missing images, invalid bboxes, or duplicate sample ids after validation.
- Phase-1 issue: celebrity-heavy distribution can dominate mixed metrics.
### Lagenda-HF
- Source: `uaebn/lagenda` Hugging Face mirror.
- Purpose: staged exact-age/gender data; not promoted.
- Labels: exact age/gender and `bbox_face`.
- Phase-1 issue: raw images are full scenes; training without face cropping
failed badly on Lagenda validation.
## Rebuild Commands
Download/stage public datasets:
```sh
cd <repo-root>
bash scripts/download-public-datasets.sh
```
Build manifests:
```sh
cd <repo-root>
bash scripts/build-training-manifests.sh
```
Validate IMDB-clean:
```sh
cd <repo-root>
bash scripts/validate-imdb-manifest.sh
```
Report dataset state:
```sh
cd <repo-root>
bash scripts/report-dataset-status.sh
```
## References
- FairFace paper: https://arxiv.org/abs/1908.04913
- FairFace repository: https://github.com/dchen236/FairFace
- MiVOLO repository: https://github.com/WildChlamydia/MiVOLO
- IMDB-clean repository: https://github.com/yiminglin-ai/imdb-clean
|