Buckets:
| license: cc-by-4.0 | |
| pretty_name: DepthDif GeoTIFF raster and aligned ARGO dataset | |
| tags: | |
| - oceanography | |
| - argo | |
| - glorys | |
| - ostia | |
| - sea-level | |
| - sea-surface-salinity | |
| - geotiff | |
| - zarr | |
| configs: | |
| - config_name: profile-index | |
| data_files: | |
| - split: profiles | |
| path: indices/profiles.parquet | |
| - split: variables | |
| path: indices/variables.parquet | |
| <p align="center"> | |
| <img src="assets/figures/banner_depthdif.png" width="85%" alt="Banner Image" /> | |
| </p> | |
| <p align="center"> | |
| <a href="https://depthdif.donike.net/"> | |
| <img src="https://img.shields.io/badge/Project-Website-0b2e4f?style=for-the-badge" alt="Open Documentation" /> | |
| </a> | |
| <a href="https://github.com/simon-donike/DepthDif/"> | |
| <img src="https://img.shields.io/badge/Project-GitHub-0b2e4f?style=for-the-badge" alt="Open GitHub" /> | |
| </a> | |
| </p> | |
| # DepthDif GeoTIFF Raster and Aligned ARGO Dataset | |
| This dataset package contains the model-ready Ocean variables (ARGO submarine data, sea surface height, sea surface temperature and | |
| salinity, as well as GLORYS reanalysis information for 50 depth levels. The ARGO data has been projected onto the GLORYS grid in order | |
| to build a ML-ready dataset. The intention is that users can create tensors easily for CV-inspired ML approaches to ocean-variable | |
| reconstruction. While traditionally this problem has been approached point-wise, this dataset enables easy training of CV models such | |
| as inpainitng-inspired approaches. | |
| ## Dataset Preview | |
| Rasterized Argo-profiles aligned with GLORYS product, examples: | |
| <p align="center"> | |
| <img src="assets/data/geotiff_dataset_random100_surface.png" width="85%" alt="Random surface-level training dataset patches" /> | |
| </p> | |
| ## Related Dataset | |
| [OceanTACO](https://huggingface.co/datasets/nilsleh/OceanTACO) is a concurrent | |
| development of a related global ocean dataset built from overlapping source | |
| products. We focus on L4 products and make some assumptions for the users in order to enable | |
| comparative benchmarks. If you want easy access to the actual underlying data, check out this dataset. | |
| ## Layout | |
| The `rasters/` directory is intentionally at the repository root. It contains | |
| the aligned uint8 GeoTIFF products used by the pixel-space dataloader. The | |
| compact `argo/argo_profiles_on_grid.zarr` store is the grid-indexed ARGO input | |
| used by that dataloader. | |
| The package intentionally contains two ARGO Zarr stores with different roles. | |
| `argo/argo_profiles_on_grid.zarr` is the compact grid-indexed store meant to be | |
| used together with the GeoTIFF raster dataset. `data/argo_glors_ostia_ssh.zarr` | |
| is the full enriched profile-level store and holds the complete ARGO | |
| collocation dataset, including the sampled GLORYS, OSTIA, sea-level, and | |
| sea-surface-salinity context. | |
| ## Raster Products | |
| All GeoTIFF rasters are exported on the GLORYS 0.1 degree global grid | |
| (`EPSG:4326`, 3600 x 1800 pixels, west-to-east longitudes from -180 to 180 and | |
| north-to-south latitudes from 90 to -90). The current package contains 1283 | |
| weekly target dates per raster product, from 2000-01-01 through 2024-07-26. | |
| Files are named `<variable>_YYYYMMDD.tif`. | |
| The GLORYS variables are depth-resolved 50-band GeoTIFFs: | |
| - `rasters/glorys/thetao/`: potential temperature, encoded as Kelvin. | |
| - `rasters/glorys/so/`: salinity, encoded as PSU. | |
| The surface products are single-band GeoTIFFs aggregated to the same weekly | |
| target dates with a centered 7-day mean window: | |
| - `rasters/ostia/analysed_sst/`: OSTIA analysed sea-surface temperature in Kelvin. | |
| - `rasters/sealevel/adt/`: absolute dynamic topography in meters. | |
| - `rasters/sss/sos/`: sea-surface salinity in PSU. | |
| - `rasters/sss/dos/`: sea-surface density in kg/m3. | |
| Raster pixels are stored as `uint8` with `255` reserved for nodata. Valid codes | |
| `0..254` are linearly decoded using the stretch ranges in `manifest.yaml`; | |
| per-file statistics, source filenames, compression, target dates, and the full | |
| depth axis are also recorded there. | |
| ## PyTorch Dataset and DataLoader | |
| The repository includes a standalone loader package in `depthdif_dataset/`. | |
| It is designed to work directly from a local Hugging Face dataset checkout | |
| without installing the full DepthDif training repository. | |
| Install the loader dependencies in your environment: | |
| ```bash | |
| pip install -r requirements-loader.txt | |
| ``` | |
| Minimal PyTorch usage from the dataset repository root: | |
| ```python | |
| from depthdif_dataset import ArgoGeoTIFFGriddedPatchDataset, build_dataloader | |
| dataset = ArgoGeoTIFFGriddedPatchDataset( | |
| geotiff_root_dir=".", | |
| split="all", | |
| tile_size=128, | |
| patch_stride=128, | |
| max_dates=1, | |
| metadata_cache_dir=None, | |
| ) | |
| loader = build_dataloader(dataset, batch_size=2, num_workers=0) | |
| batch = next(iter(loader)) | |
| print(batch["x"].shape, batch["eo"].shape, batch["land_mask"].shape) | |
| ``` | |
| The default sample contains normalized tensors for sparse ARGO temperature | |
| input (`x`), dense GLORYS temperature target (`y`), dense surface context | |
| (`eo`), validity masks, the ocean `land_mask`, target `date`, patch | |
| coordinates, and a small `info` dictionary. Set `include_salinity=True` to | |
| add `x_salinity`, `y_salinity`, and their masks. Use `eo_source="sss"` with | |
| `eo_var_name="sos"` to use sea-surface salinity as the surface context | |
| instead of OSTIA SST. | |
| A runnable smoke test is included: | |
| ```bash | |
| python examples/torch_dataloader.py --root . --date-start 20000101 --max-dates 1 --batch-size 2 | |
| ``` | |
| ARGO-support filtering for train/validation splits requires counting profile | |
| overlap per patch/date. The loader skips that scan for `split="all"` unless | |
| `count_argo_support=True` or `--require-argo` is set in the example script. | |
| ### Patch Grid, Overlap, and Land Filtering | |
| The loader builds square patches on the fixed 0.1 degree GLORYS grid. With the | |
| default `tile_size=128`, one sample covers a 12.8 x 12.8 degree region and has | |
| 50 depth bands. `patch_stride` controls how far the next patch starts: | |
| - `patch_stride=128`: non-overlapping global tiles. | |
| - `patch_stride=96`: 32-pixel overlap, or 3.2 degrees at 0.1 degree resolution. | |
| - `patch_stride=32`: 96-pixel overlap, or 9.6 degrees at 0.1 degree resolution. | |
| Smaller strides create more samples and make neighboring patches share more | |
| context, but they also increase row-index size and training time. If you use | |
| overlap for train/validation splits, prefer a temporal validation split such as | |
| `val_year=2018`; spatial random splits with overlapping patches can leak nearly | |
| identical context between train and validation. | |
| <p align="center"> | |
| <img src="assets/data/patch_grid/patch_grid_global_overview.webp" width="78%" alt="Global patch grid overview" /> | |
| </p> | |
| <p align="center"> | |
| <img src="assets/data/patch_grid/patch_overlap_regional_example.webp" width="72%" alt="Regional example of overlapping patch windows" /> | |
| </p> | |
| `max_land_fraction` filters out patches that are mostly land. The default | |
| `max_land_fraction=0.30` keeps patches with at least 70 percent ocean pixels. | |
| Increase it for coastal finetuning, or decrease it for open-ocean training. | |
| The returned `land_mask` tensor uses `1` for ocean/support pixels and `0` for | |
| land or unavailable support. | |
| <p align="center"> | |
| <img src="assets/data/patch_grid/land_fraction_filter_examples.webp" width="72%" alt="Examples of patch filtering by land fraction" /> | |
| </p> | |
| ### Common Loader Recipes | |
| Use a small date slice for quick inspection without building a large metadata | |
| cache: | |
| ```python | |
| dataset = ArgoGeoTIFFGriddedPatchDataset( | |
| geotiff_root_dir=".", | |
| split="all", | |
| date_start=20000101, | |
| max_dates=1, | |
| tile_size=128, | |
| patch_stride=128, | |
| metadata_cache_dir=None, | |
| ) | |
| ``` | |
| Use overlapping patches for training: | |
| ```python | |
| dataset = ArgoGeoTIFFGriddedPatchDataset( | |
| geotiff_root_dir=".", | |
| split="train", | |
| tile_size=128, | |
| patch_stride=32, | |
| val_year=2018, | |
| metadata_cache_dir="depthdif_cache", | |
| ) | |
| ``` | |
| Use salinity targets and ARGO salinity inputs: | |
| ```python | |
| dataset = ArgoGeoTIFFGriddedPatchDataset( | |
| geotiff_root_dir=".", | |
| split="all", | |
| include_salinity=True, | |
| output_fields=("temperature", "salinity"), | |
| metadata_cache_dir=None, | |
| max_dates=1, | |
| ) | |
| ``` | |
| Use sea-surface salinity as the surface context instead of OSTIA SST: | |
| ```python | |
| dataset = ArgoGeoTIFFGriddedPatchDataset( | |
| geotiff_root_dir=".", | |
| split="all", | |
| eo_source="sss", | |
| eo_var_name="sos", | |
| include_salinity=True, | |
| metadata_cache_dir=None, | |
| max_dates=1, | |
| ) | |
| ``` | |
| Use synthetic sparse observations sampled from the dense GLORYS target instead | |
| of real ARGO profiles: | |
| ```python | |
| dataset = ArgoGeoTIFFGriddedPatchDataset( | |
| geotiff_root_dir=".", | |
| split="all", | |
| synthetic_mode=True, | |
| synthetic_pixel_count=250, | |
| require_argo_for_all=False, | |
| metadata_cache_dir=None, | |
| max_dates=1, | |
| ) | |
| ``` | |
| Filter samples to patch/date rows that contain ARGO support. This does extra | |
| index work and is worth caching when used repeatedly: | |
| ```python | |
| dataset = ArgoGeoTIFFGriddedPatchDataset( | |
| geotiff_root_dir=".", | |
| split="all", | |
| count_argo_support=True, | |
| require_argo_for_all=True, | |
| metadata_cache_dir="depthdif_cache", | |
| date_start=20000101, | |
| max_dates=1, | |
| ) | |
| ``` | |
| The ARGO support map below shows why this filter changes the row distribution: | |
| many open-ocean patches have dense support, while other valid ocean patches may | |
| have no profiles for a given weekly target date. | |
| <p align="center"> | |
| <img src="assets/data/argo_valid_pixels_per_patch.webp" width="78%" alt="ARGO valid pixels per training patch" /> | |
| </p> | |
| ### Sample Dictionary | |
| The default dataset returns normalized tensors: | |
| - `x`: sparse ARGO temperature input, shape `(50, tile_size, tile_size)`. | |
| - `y`: dense GLORYS temperature target, shape `(50, tile_size, tile_size)`. | |
| - `eo`: dense surface context, shape `(1, tile_size, tile_size)`. | |
| - `x_valid_mask` and `y_valid_mask`: boolean temperature masks. | |
| - `x_valid_mask_1d`: depth-collapsed sparse-input support mask. | |
| - `land_mask`: ocean/support mask, shape `(1, tile_size, tile_size)`. | |
| - `date`: target date as `YYYYMMDD`. | |
| - `coords`: patch center latitude and longitude when `return_coords=True`. | |
| - `info`: patch/date metadata when `return_info=True`. | |
| When `include_salinity=True`, samples also include `x_salinity`, | |
| `y_salinity`, `x_salinity_valid_mask`, `y_salinity_valid_mask`, and | |
| `x_salinity_valid_mask_1d`. | |
| Temperatures are normalized from Celsius using the DepthDif training statistics; | |
| salinity is normalized from PSU. To recover physical units: | |
| ```python | |
| from depthdif_dataset import salinity_normalize, temperature_normalize | |
| temperature_c = temperature_normalize(mode="denorm", tensor=batch["y"]) | |
| salinity_psu = salinity_normalize(mode="denorm", tensor=batch["y_salinity"]) | |
| ``` | |
| ## ARGO Alignment Examples | |
| ARGO profiles are projected onto the fixed 50-level GLORYS depth axis before | |
| spatial rasterization. The examples below show the grid-indexed ARGO | |
| representation and profile-level alignment quality. | |
| <p align="center"> | |
| <img src="assets/data/argo_on_glorys_grid_3D.gif" width="70%" alt="Depth-aligned ARGO values on the GLORYS grid" /> | |
| </p> | |
| <p align="center"> | |
| <img src="assets/data/profile_comparison_good_alignment.png" width="72%" alt="Example of ARGO-to-GLORYS profile alignment" /> | |
| </p> | |
| The full enriched profile-level ARGO collocation dataset is available at: | |
| ```python | |
| import xarray as xr | |
| ds = xr.open_zarr("data/argo_glors_ostia_ssh.zarr", consolidated=None) | |
| ``` | |
| The lightweight Parquet indices are included for preview and filtering: | |
| ```python | |
| import pandas as pd | |
| profiles = pd.read_parquet("indices/profiles.parquet") | |
| variables = pd.read_parquet("indices/variables.parquet") | |
| ``` | |
| Coverage: | |
| - Raster target dates: 2000-01-01 to 2024-07-26 | |
| - Raster target date count per product: 1283 | |
| - Enriched ARGO profiles: 9485977 | |
| - Enriched ARGO profile dates: 2000-01-01 to 2024-07-31 | |
| - Compact grid-indexed ARGO profiles: 9451644 | |
| - GLORYS depth levels: 50 | |
| The package is released as CC BY 4.0. Upstream product licenses and citation | |
| requirements for EN4/ARGO, GLORYS, OSTIA, sea-level, and | |
| sea-surface-salinity products still apply; see `LICENSE` for the attribution | |
| notice. | |
Xet Storage Details
- Size:
- 12.1 kB
- Xet hash:
- e0782932db20ddb459775a27437348d3e25bb4210a725633d2d8af68f3cc889a
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.