--- license: cc-by-4.0 pipeline_tag: graph-ml language: - en library_name: anemoi --- # AIFS ENS - v1.0 Here, we introduce the Artificial Intelligence Forecasting System (AIFS) ENS, an ensemble-based probablistic data driven forecast model developed by the [European Centre for Medium-Range Weather Forecasts (ECMWF)](https://www.ecmwf.int/). AIFS produces highly skilled forecasts for upper-air variables, surface weather parameters and tropical cyclone tracks. AIFS ENS is run four times daily alongside ECMWF’s physics-based NWP model and forecasts are available to the public under ECMWF’s [open data policy](https://www.ecmwf.int/en/forecasts/datasets/open-data). Note that due to the non-determinism of GPUs, users will be unable to exactly reproduce an official AIFS ENS forecast when running AIFS ENS themselves. For more details please refer to our [documentation](https://confluence.ecmwf.int/display/FCST/Implementation+of+AIFS+ENS+v1). ## Data Details ### Data parameters **Note** Regarding precipitation units, it's worth noting that AIFS model was trained on \\(m^{3}/m^{2}\\) and will therefore produce precip in that units. If one wants to retrieve precipitation from Open data, the units will be \\(mm\\). ## Model Details ### Description For full details of the AIFS ENS, please see [Lang et al. 2024](https://arxiv.org/abs/2412.15832). AIFS is based on a graph neural network (GNN) encoder and decoder, and a sliding window transformer processor, and is trained on ECMWF’s ERA5 re-analysis and ECMWF’s operational numerical weather prediction (NWP) analyses.
Encoder graph Decoder graph
It has a flexible and modular design and supports several levels of parallelism to enable training on high resolution input data. AIFS forecast skill is assessed by comparing its forecasts to NWP analyses and direct observational data. AIFS ENS was trained to minimise the Continuous Ranked Probability Score (CRPS) calculated over a small ensemble.
Training schematic
- **Developed by:** ECMWF - **Model type:** Encoder-processor-decoder model - **License:** These model weights are published under a Creative Commons Attribution 4.0 International (CC BY 4.0). To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/. The notebooks and other script files are published under an Apache 2.0 licence, to view a copy of this license, visit https://www.apache.org/licenses/LICENSE-2.0.txt. ### Resolution | | Component | Horizontal Resolution [kms] | Vertical Resolution [levels] | |---|:---:|:---:|:---:| | Atmosphere | AIFS ENS v1.0 | ~ 31 | 13 | ### Sources - **Repository:** [Anemoi](https://anemoi.readthedocs.io/en/latest/index.html) is an open-source framework for creating machine learning (ML) weather forecasting systems, co-developed by ECMWF and a range of national meteorological services across Europe. - **Paper:** https://arxiv.org/abs/2412.15832 ## Get started 🚨 **Note** `aifs-ens-1.0` requires `38GB` of GPU memory by default, it is possible to reduce this by setting `ANEMOI_INFERENCE_NUM_CHUNKS` to a multiple of 2. For example setting it to `16` reduces the memory requirements to `24GB` at the cost of throughput. Therefore issues may arise with lower end GPUs. To generate a new forecast using AIFS ENS, you can use [anemoi-inference](https://github.com/ecmwf/anemoi-inference). In the [example notebook](run_AIFS_ENS_v1.ipynb), a step-by-step workflow is specified to run AIFS ENS using the HuggingFace model: 1. **Install Required Packages and Imports** - `torch==2.5.0 anemoi-inference[huggingface]==0.6.0 anemoi-models==0.6.0 anemoi-graphs==0.6.0 anemoi-datasets==0.5.23` - `earthkit-regrid==0.4.0 'ecmwf-opendata>=0.3.19'` - `flash_attn` 2. **Retrieve Initial Conditions from ECMWF Open Data** - Select a date - Get the data from the [ECMWF Open Data API](https://www.ecmwf.int/en/forecasts/datasets/open-data) - Get input fields - Add the single levels fields and pressure levels fields - Convert geopotential height into geopotential - Create the initial state 3. **Load the Model and Run the Forecast** - Download the Model's Checkpoint from Hugging Face - Create a runner - Run the forecast using anemoi-inference 4. **Inspect the generated forecast** - Plot a field 🚨 **Note** we train AIFS using [`flash_attention`](https://github.com/Dao-AILab/flash-attention). The use of the 'Flash Attention' package also imposes certain requirements in terms of software and hardware, which can be found under [Installation and Features](https://github.com/Dao-AILab/flash-attention?tab=readme-ov-file#installation-and-features) on the GitHub repo. 🚨 **Note** the `aifs_ens_v1.0.ckpt` checkpoint just contains the model’s weights. It does not contain any information about the optimizer states, lr-scheduler states, etc. ## Citation If you use this model in your work, please cite it as follows: **BibTeX:** ``` @misc{lang2024aifscrps, title={AIFS-CRPS: Ensemble forecasting using a model trained with a loss function based on the Continuous Ranked Probability Score}, author={Simon Lang and Mihai Alexe and Mariana C. A. Clare and Christopher Roberts and Rilwan Adewoyin and Zied Ben Bouallègue and Matthew Chantry and Jesper Dramsch and Peter D. Dueben and Sara Hahner and Pedro Maciel and Ana Prieto-Nemesio and Cathal O'Brien and Florian Pinault and Jan Polster and Baudouin Raoult and Steffen Tietsche and Martin Leutbecher}, year={2024}, eprint={2412.15832}, archivePrefix={arXiv}, primaryClass={physics.ao-ph}, url={https://arxiv.org/abs/2412.15832}, } ``` **APA:** ``` Lang, S., Alexe, M., Clare, M. C., Roberts, C., Adewoyin, R., Bouallègue, Z. B., ... & Leutbecher, M. (2024). AIFS-CRPS: Ensemble forecasting using a model trained with a loss function based on the Continuous Ranked Probability Score. arXiv preprint arXiv:2412.15832. ``` ## More Information [Find the paper here](https://arxiv.org/pdf/2412.15832)