Buckets:

dClimate
/

forecast-ic

Files

xet

dClimate/forecast-ic / README.md

PhiltheBuilder

about 1 month ago

preview code

download

raw

7.54 kB

# Weather model initial conditions

Analysis-ready atmospheric initial conditions on a 0.25° global grid, from two
sources, in one file format. One file per init time, containing the inputs of
the published data-driven weather models (GraphCast, GenCast, Pangu, Aurora,
AIFS, FourCastNet/SFNO): take the fields your model needs.

Public; readable without a Hugging Face account.

| source | from | available | levels |
|---|---|---|---|
| ERA5 (bucket root) | ARCO-ERA5 reanalysis | ~6 days after the init | 37 |
| IFS (`ifs/`) | ECMWF operational IFS analysis, open data | ~8 hours after | 14 |

Both use the same variable names, units, grid, orientation and timesteps, so
code written for one reads the other. Each `spec.json` names its `ic_source`;
an IFS one also lists what is absent under `unavailable_variables` and what
only it has under `extra_variables` (see [IFS source](#ifs-source)).

## Layout

```
2026-09-10T00Z/          ERA5
  ic.nc        NetCDF4, float32, zlib-compressed, ~1.5 GB
  spec.json    field list, grid definition, provenance
ifs/2026-09-10T00Z/      IFS
  ic.nc        ~0.44 GB
  spec.json
```

One directory per init time, named `YYYY-MM-DDTHHZ` (UTC, synoptic hours only:
00, 06, 12, 18). A directory is complete only when its `spec.json` is present;
without one it is mid-upload.

## Contents

280 fields per timestep.

**7 atmospheric variables** on all **37 pressure levels** of ERA5
(1 to 1000 hPa, ascending):
`geopotential`, `temperature`, `u_component_of_wind`, `v_component_of_wind`,
`specific_humidity`, `vertical_velocity`, `relative_humidity`
— dims `(time, level, latitude, longitude)`. Select the 13-level
GraphCast/Pangu set with `.sel(level=[50, 100, ..., 1000])`.

**21 surface variables**: `2m_temperature`, `10m_u_component_of_wind`,
`10m_v_component_of_wind`, `100m_u_component_of_wind`,
`100m_v_component_of_wind`, `mean_sea_level_pressure`, `surface_pressure`,
`2m_dewpoint_temperature`, `skin_temperature`, `total_column_water_vapour`,
`total_column_water`, `soil_temperature_level_1`, `soil_temperature_level_2`,
`volumetric_soil_water_layer_1`, `volumetric_soil_water_layer_2`,
`sea_surface_temperature`, `sea_ice_cover`, `total_precipitation`,
`total_precipitation_6hr`, `total_precipitation_12hr`,
`toa_incident_solar_radiation`
— dims `(time, latitude, longitude)`

**5 static fields**: `geopotential_at_surface`, `land_sea_mask`, `soil_type`,
`standard_deviation_of_orography`, `slope_of_sub_gridscale_orography`
— dims `(latitude, longitude)`, time dimension already dropped

**Three timesteps**: T-12h, T-6h and T0, 6 hours apart. `time[-1]` is the init
time.

### Derived fields

Not in ARCO; computed when the file is built.

- `total_precipitation_6hr` / `_12hr`: the sum of ERA5's hourly
  `total_precipitation` over the 6 or 12 hours **ending at** each step. Matches
  WeatherBench2's fields of the same name, which GraphCast and GenCast were
  trained on.
- `relative_humidity`: percent, reproducing ECMWF's published ERA5 `r`
  (saturation over water above 0 °C, over ice below −23 °C, blended between;
  below ground, the near-surface value). Not clipped: like ECMWF's own field it
  exceeds 100% in cold air and can dip slightly below 0 where ERA5's specific
  humidity does. WeatherBench2's `relative_humidity` uses a water-only formula
  and differs; a model trained on it should recompute from `temperature` and
  `specific_humidity`.

### Missing values

`sea_surface_temperature` and `sea_ice_cover` are **NaN over land**, as ERA5
publishes them. Every other field is finite everywhere.

## Grid

0.25° global, 721 × 1440.

- **Latitude descends: 90.0 → -90.0**
- **Longitude runs 0.0 → 359.75**, not -180 → 180

Both are easy to assume backwards, and reading them the wrong way round yields
output that looks plausible and is globally wrong.

Values are in SI units, as ERA5 publishes them: temperature in K, geopotential
in m²/s² (divide by 9.80665 for metres), pressure in Pa.

## Reading it

```python
from huggingface_hub import download_bucket_files
import xarray as xr
download_bucket_files(
    "dClimate/forecast-ic",
    files=[("2026-09-10T00Z/ic.nc", "ic.nc")],
    token=False,          # public; no account needed
)
ds = xr.open_dataset("ic.nc")
```

Inside a Hugging Face Job, mount it instead of downloading:

```python
Volume(type="bucket", source="dClimate/forecast-ic",
       mount_path="/inputs", read_only=True)
```

A check that your read is oriented correctly — for `2026-09-10T00Z`:

| quantity | expected |
|---|---|
| 500 hPa geopotential height, global mean | ≈ 5539 m |
| 2 m temperature, global mean | ≈ 280.4 K |
| land fraction (`land_sea_mask > 0.5`) | ≈ 33.9 % |

## Source and provenance

Built from [ARCO-ERA5](https://github.com/google-research/arco-era5)
(`ar/full_37-1h-0p25deg-chunk-1.zarr-v3`), Google's analysis-ready ERA5.

Each `spec.json` records the source store and its `last_updated` /
`valid_time_stop` / `valid_time_stop_era5t` attributes at build time. ARCO
publishes no content hash, so these attributes are how a file is pinned to an
upstream release.

Files are immutable once published. ERA5T values are revised by the provider for
several days after publication, so an init is built only once it is at least 5
days old, and is not rebuilt afterwards.

`spec.json` carries `provenance.is_era5t`. True means that init came from ERA5T
rather than the finalized archive — still fixed here, but worth knowing when
comparing against another ERA5 copy.

## IFS source

Under `ifs/`: ECMWF's operational HRES analysis from
[open data](https://www.ecmwf.int/en/forecasts/datasets/open-data), 00Z inits,
published about eight hours after the init. Each step is the step-0 analysis of
the run at that time.

- **14 pressure levels**: 10, 50, 100, 150, 200, 250, 300, 400, 500, 600, 700,
  850, 925, 1000 hPa. Every 13-level model's set is included.
- `relative_humidity` is ECMWF's own `r` from the same analysis.
- `total_precipitation_6hr` / `_12hr` are the forecast totals of the runs that
  started 6 and 12 hours before each step (the 18Z run's first six hours, for
  the 6h total at 00Z).
- **Not present**: `total_precipitation` (hourly; open data is 3-hourly),
  `toa_incident_solar_radiation`, `sea_surface_temperature`, `sea_ice_cover`,
  `soil_type`.
- **Only here** (`extra_variables`): `snow_depth` (m of water equivalent) and
  eleven fields from the wave model's analysis:
  `significant_height_of_combined_wind_waves_and_swell`,
  `significant_wave_height_{10_to_12,12_to_14,14_to_17,17_to_21,21_to_25,25_to_30}s`,
  `mean_wave_direction`, `mean_wave_period`, `coefficient_of_drag_with_waves`,
  `model_bathymetry`. The wave fields are NaN wherever the wave model has no
  open sea: land, lakes and sea ice, about 36% of the grid
  (`ocean_only_variables`). These are what ECMWF's AIFS Single 2.0 reads.
- From 2026-05-13T00Z, the first init after IFS cycle 50r1, which added the
  wave fields to open data. `spec_version` 1.1.0.

The operational analysis is not revised, so an init is final when published.
`spec.json`'s `provenance.runs` lists the exact open-data files it was built
from.

## Retention

Rolling 6 months (183 days) for each source. Older inits are removed.

---

ERA5 is produced by ECMWF and distributed via the Copernicus Climate Change
Service (C3S); neither ECMWF nor C3S is responsible for any use made of it here.
The IFS files are derived from ECMWF open data, © ECMWF, licensed under
[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).

Xet Storage Details

Size:: 7.54 kB
Xet hash:: a16a8b160f1ffed8c9168c9bb440a76d69ea484f62e0a89adc8d382bfd830a0f

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# Weather model initial conditions

	Analysis-ready atmospheric initial conditions on a 0.25° global grid, from two
	sources, in one file format. One file per init time, containing the inputs of
	the published data-driven weather models (GraphCast, GenCast, Pangu, Aurora,
	AIFS, FourCastNet/SFNO): take the fields your model needs.

	Public; readable without a Hugging Face account.

	\| source \| from \| available \| levels \|
	\|---\|---\|---\|---\|
	\| ERA5 (bucket root) \| ARCO-ERA5 reanalysis \| ~6 days after the init \| 37 \|
	\| IFS (`ifs/`) \| ECMWF operational IFS analysis, open data \| ~8 hours after \| 14 \|

	Both use the same variable names, units, grid, orientation and timesteps, so
	code written for one reads the other. Each `spec.json` names its `ic_source`;
	an IFS one also lists what is absent under `unavailable_variables` and what
	only it has under `extra_variables` (see [IFS source](#ifs-source)).

	## Layout

	```
	2026-09-10T00Z/ ERA5
	ic.nc NetCDF4, float32, zlib-compressed, ~1.5 GB
	spec.json field list, grid definition, provenance
	ifs/2026-09-10T00Z/ IFS
	ic.nc ~0.44 GB
	spec.json
	```

	One directory per init time, named `YYYY-MM-DDTHHZ` (UTC, synoptic hours only:
	00, 06, 12, 18). A directory is complete only when its `spec.json` is present;
	without one it is mid-upload.

	## Contents

	280 fields per timestep.

	7 atmospheric variables on all 37 pressure levels of ERA5
	(1 to 1000 hPa, ascending):
	`geopotential`, `temperature`, `u_component_of_wind`, `v_component_of_wind`,
	`specific_humidity`, `vertical_velocity`, `relative_humidity`
	— dims `(time, level, latitude, longitude)`. Select the 13-level
	GraphCast/Pangu set with `.sel(level=[50, 100, ..., 1000])`.

	21 surface variables: `2m_temperature`, `10m_u_component_of_wind`,
	`10m_v_component_of_wind`, `100m_u_component_of_wind`,
	`100m_v_component_of_wind`, `mean_sea_level_pressure`, `surface_pressure`,
	`2m_dewpoint_temperature`, `skin_temperature`, `total_column_water_vapour`,
	`total_column_water`, `soil_temperature_level_1`, `soil_temperature_level_2`,
	`volumetric_soil_water_layer_1`, `volumetric_soil_water_layer_2`,
	`sea_surface_temperature`, `sea_ice_cover`, `total_precipitation`,
	`total_precipitation_6hr`, `total_precipitation_12hr`,
	`toa_incident_solar_radiation`
	— dims `(time, latitude, longitude)`

	5 static fields: `geopotential_at_surface`, `land_sea_mask`, `soil_type`,
	`standard_deviation_of_orography`, `slope_of_sub_gridscale_orography`
	— dims `(latitude, longitude)`, time dimension already dropped

	Three timesteps: T-12h, T-6h and T0, 6 hours apart. `time[-1]` is the init
	time.

	### Derived fields

	Not in ARCO; computed when the file is built.

	- `total_precipitation_6hr` / `_12hr`: the sum of ERA5's hourly
	`total_precipitation` over the 6 or 12 hours ending at each step. Matches
	WeatherBench2's fields of the same name, which GraphCast and GenCast were
	trained on.
	- `relative_humidity`: percent, reproducing ECMWF's published ERA5 `r`
	(saturation over water above 0 °C, over ice below −23 °C, blended between;
	below ground, the near-surface value). Not clipped: like ECMWF's own field it
	exceeds 100% in cold air and can dip slightly below 0 where ERA5's specific
	humidity does. WeatherBench2's `relative_humidity` uses a water-only formula
	and differs; a model trained on it should recompute from `temperature` and
	`specific_humidity`.

	### Missing values

	`sea_surface_temperature` and `sea_ice_cover` are NaN over land, as ERA5
	publishes them. Every other field is finite everywhere.

	## Grid

	0.25° global, 721 × 1440.

	- Latitude descends: 90.0 → -90.0
	- Longitude runs 0.0 → 359.75, not -180 → 180

	Both are easy to assume backwards, and reading them the wrong way round yields
	output that looks plausible and is globally wrong.

	Values are in SI units, as ERA5 publishes them: temperature in K, geopotential
	in m²/s² (divide by 9.80665 for metres), pressure in Pa.

	## Reading it

	```python
	from huggingface_hub import download_bucket_files
	import xarray as xr

	download_bucket_files(
	"dClimate/forecast-ic",
	files=[("2026-09-10T00Z/ic.nc", "ic.nc")],
	token=False, # public; no account needed
	)
	ds = xr.open_dataset("ic.nc")
	```

	Inside a Hugging Face Job, mount it instead of downloading:

	```python
	Volume(type="bucket", source="dClimate/forecast-ic",
	mount_path="/inputs", read_only=True)
	```

	A check that your read is oriented correctly — for `2026-09-10T00Z`:

	\| quantity \| expected \|
	\|---\|---\|
	\| 500 hPa geopotential height, global mean \| ≈ 5539 m \|
	\| 2 m temperature, global mean \| ≈ 280.4 K \|
	\| land fraction (`land_sea_mask > 0.5`) \| ≈ 33.9 % \|

	## Source and provenance

	Built from [ARCO-ERA5](https://github.com/google-research/arco-era5)
	(`ar/full_37-1h-0p25deg-chunk-1.zarr-v3`), Google's analysis-ready ERA5.

	Each `spec.json` records the source store and its `last_updated` /
	`valid_time_stop` / `valid_time_stop_era5t` attributes at build time. ARCO
	publishes no content hash, so these attributes are how a file is pinned to an
	upstream release.

	Files are immutable once published. ERA5T values are revised by the provider for
	several days after publication, so an init is built only once it is at least 5
	days old, and is not rebuilt afterwards.

	`spec.json` carries `provenance.is_era5t`. True means that init came from ERA5T
	rather than the finalized archive — still fixed here, but worth knowing when
	comparing against another ERA5 copy.

	## IFS source

	Under `ifs/`: ECMWF's operational HRES analysis from
	[open data](https://www.ecmwf.int/en/forecasts/datasets/open-data), 00Z inits,
	published about eight hours after the init. Each step is the step-0 analysis of
	the run at that time.

	- 14 pressure levels: 10, 50, 100, 150, 200, 250, 300, 400, 500, 600, 700,
	850, 925, 1000 hPa. Every 13-level model's set is included.
	- `relative_humidity` is ECMWF's own `r` from the same analysis.
	- `total_precipitation_6hr` / `_12hr` are the forecast totals of the runs that
	started 6 and 12 hours before each step (the 18Z run's first six hours, for
	the 6h total at 00Z).
	- Not present: `total_precipitation` (hourly; open data is 3-hourly),
	`toa_incident_solar_radiation`, `sea_surface_temperature`, `sea_ice_cover`,
	`soil_type`.
	- Only here (`extra_variables`): `snow_depth` (m of water equivalent) and
	eleven fields from the wave model's analysis:
	`significant_height_of_combined_wind_waves_and_swell`,
	`significant_wave_height_{10_to_12,12_to_14,14_to_17,17_to_21,21_to_25,25_to_30}s`,
	`mean_wave_direction`, `mean_wave_period`, `coefficient_of_drag_with_waves`,
	`model_bathymetry`. The wave fields are NaN wherever the wave model has no
	open sea: land, lakes and sea ice, about 36% of the grid
	(`ocean_only_variables`). These are what ECMWF's AIFS Single 2.0 reads.
	- From 2026-05-13T00Z, the first init after IFS cycle 50r1, which added the
	wave fields to open data. `spec_version` 1.1.0.

	The operational analysis is not revised, so an init is final when published.
	`spec.json`'s `provenance.runs` lists the exact open-data files it was built
	from.

	## Retention

	Rolling 6 months (183 days) for each source. Older inits are removed.

	---

	ERA5 is produced by ECMWF and distributed via the Copernicus Climate Change
	Service (C3S); neither ECMWF nor C3S is responsible for any use made of it here.
	The IFS files are derived from ECMWF open data, © ECMWF, licensed under
	[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).