_transfer_records/README.md

# Transfer records — AguaTrack global (AR6) bucket

One manifest per year listing EVERY object for that year, with its exact byte size,
so a download can be verified per file rather than by counting.

    manifest_YYYY.tsv     kind <TAB> remote_path <TAB> size_bytes
      monthly   {REGION}/{YEAR}/{slice}/monthly/backtrack_{YEAR}-MM.nc
      ckpt      checkpoints/{REGION}/slice_{i}/restart_{YEAR}-MM-01.nc

Generated from `hf sync --dry-run`'s own planner, NOT `hf buckets ls -R` — the
recursive listing was observed silently truncating large prefixes, returning fewer
rows with no error.

## Verifying a transfer

    ./transfer_year.sh YEAR /path/to/dest        # download + verify
    ./transfer_year.sh YEAR /path/to/dest --verify-only

Delete from the bucket only on `verdict: COMPLETE`.

## Two things that are easy to get wrong

Checkpoints have **no year folder** — they live at
`checkpoints/{REGION}/slice_{i}/restart_{YEAR}-MM-01.nc` and must be selected by
filename (`--include '*/restart_YYYY-*.nc'`). A loop syncing `checkpoints/{YEAR}`
transfers nothing and reports success.

Per region, monthly and checkpoint counts are both `slices × 12` and must be
EQUAL. That invariant is what caught SEA's 2023 checkpoints coming back empty from
a transient planner failure; without it the manifest would have under-reported and
verification would have passed while 108 objects were never downloaded.

## Per-year totals

| year | monthly | ckpt | total | size |
|---|---|---|---|---|
| 2018 | 3504 | 3504 | 7008 | 1.770 TB |
| 2019 | 3504 | 3504 | 7008 | 1.773 TB |
| 2020 | 3504 | 3504 | 7008 | 1.743 TB |
| 2021 | 3504 | 3504 | 7008 | 1.715 TB |
| 2022 | 3504 | 3504 | 7008 | 1.719 TB |
| 2023 | 3504 | 3504 | 7008 | 1.702 TB |
| 2024 | 3504 | 3504 | 7008 | 1.747 TB |
| 2025 | 3504 | 3504 | 7008 | 1.730 TB |
| 2026 |  292 |  292 |  584 | 0.092 TB |  (2026-01 only)

All four published years pass the invariant (monthly==ckpt per region).
2022 and older become eligible as the run closes each year; the run walks
backward, so a year is only shippable once tracking has passed BELOW it.