_transfer_records/README.md

Transfer records — AguaTrack global (AR6) bucket

One manifest per year listing EVERY object for that year, with its exact byte size, so a download can be verified per file rather than by counting.

manifest_YYYY.tsv     kind <TAB> remote_path <TAB> size_bytes
  monthly   {REGION}/{YEAR}/{slice}/monthly/backtrack_{YEAR}-MM.nc
  ckpt      checkpoints/{REGION}/slice_{i}/restart_{YEAR}-MM-01.nc

Generated from hf sync --dry-run's own planner, NOT hf buckets ls -R — the recursive listing was observed silently truncating large prefixes, returning fewer rows with no error.

Verifying a transfer

./transfer_year.sh YEAR /path/to/dest        # download + verify
./transfer_year.sh YEAR /path/to/dest --verify-only

Delete from the bucket only on verdict: COMPLETE.

Two things that are easy to get wrong

Checkpoints have no year folder — they live at checkpoints/{REGION}/slice_{i}/restart_{YEAR}-MM-01.nc and must be selected by filename (--include '*/restart_YYYY-*.nc'). A loop syncing checkpoints/{YEAR} transfers nothing and reports success.

Per region, monthly and checkpoint counts are both slices × 12 and must be EQUAL. That invariant is what caught SEA's 2023 checkpoints coming back empty from a transient planner failure; without it the manifest would have under-reported and verification would have passed while 108 objects were never downloaded.

Per-year totals

year	monthly	ckpt	total	size
2018	3504	3504	7008	1.770 TB
2019	3504	3504	7008	1.773 TB
2020	3504	3504	7008	1.743 TB
2021	3504	3504	7008	1.715 TB
2022	3504	3504	7008	1.719 TB
2023	3504	3504	7008	1.702 TB
2024	3504	3504	7008	1.747 TB
2025	3504	3504	7008	1.730 TB
2026	292	292	584	0.092 TB

All four published years pass the invariant (monthly==ckpt per region). 2022 and older become eligible as the run closes each year; the run walks backward, so a year is only shippable once tracking has passed BELOW it.