AIFS-ENS 1.0 Release

.gitattributes

@@ -0,0 +1,39 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+aifs_diagram.png filter=lfs diff=lfs merge=lfs -text
+decoder_graph.jpeg filter=lfs diff=lfs merge=lfs -text
+encoder_graph.jpeg filter=lfs diff=lfs merge=lfs -text
+assets/aifs-crps-schematic.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,149 @@
+---
+license: cc-by-4.0
+pipeline_tag: graph-ml
+language:
+- en
+library_name: anemoi
+---
+
+# AIFS ENS - v1.0
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+Here, we introduce the Artificial Intelligence Forecasting System (AIFS) ENS, 
+an ensemble-based probablistic data driven forecast model developed by the 
+[European Centre for Medium-Range Weather Forecasts (ECMWF)](https://www.ecmwf.int/).
+
+AIFS produces highly skilled forecasts for upper-air variables, surface weather parameters and
+tropical cyclone tracks. AIFS ENS is run four times daily alongside ECMWF’s physics-based NWP model and forecasts
+are available to the public under ECMWF’s [open data policy](https://www.ecmwf.int/en/forecasts/datasets/open-data). 
+Note that due to the non-determinism of GPUs, users will be unable to exactly reproduce an official AIFS ENS forecast 
+when running AIFS ENS themselves.
+
+For more details please refer to our [documentation](https://confluence.ecmwf.int/display/FCST/Implementation+of+AIFS+ENS+v1).
+
+## Data Details
+
+### Data parameters
+
+
+**Note**
+Regarding precipitation units, it's worth noting that AIFS model was trained on \\(m^{3}/m^{2}\\) and will therefore produce precip in that units.
+If one wants to retrieve precipitation from Open data, the units will be \\(mm\\).
+
+
+## Model Details
+
+### Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+For full details of the AIFS ENS, please see [Lang et al. 2024](https://arxiv.org/abs/2412.15832).
+
+AIFS is based on a graph neural network (GNN) encoder and decoder, and a sliding window transformer processor, 
+and is trained on ECMWF’s ERA5 re-analysis and ECMWF’s operational numerical weather prediction (NWP) analyses. 
+
+<div style="display: flex; justify-content: center;">
+  <img src="assets/encoder_graph.jpeg" alt="Encoder graph" style="width: 50%;"/>
+  <img src="assets/decoder_graph.jpeg" alt="Decoder graph" style="width: 50%;"/>
+</div>
+
+It has a flexible and modular design and supports several levels of parallelism to enable training on
+high resolution input data. AIFS forecast skill is assessed by comparing its forecasts to NWP analyses
+and direct observational data. 
+
+AIFS ENS was trained to minimise the Continuous Ranked Probability Score (CRPS) calculated over 
+a small ensemble.
+<div style="display: flex; justify-content: center;">
+  <img src="assets/aifs-crps-schematic.png" alt="Training schematic" style="width: 75%;"/>
+</div>
+
+- **Developed by:** ECMWF
+- **Model type:** Encoder-processor-decoder model
+- **License:** These model weights are published under a Creative Commons Attribution 4.0 International (CC BY 4.0).
+To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/. 
+The notebooks and other script files are published under an Apache 2.0 licence, to view a copy of this license, visit https://www.apache.org/licenses/LICENSE-2.0.txt.
+
+### Resolution
+
+| | Component | Horizontal Resolution [kms] | Vertical Resolution [levels] |
+|---|:---:|:---:|:---:|
+| Atmosphere | AIFS ENS v1.0 | ~ 31 |  13 |
+
+
+### Sources
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [Anemoi](https://anemoi.readthedocs.io/en/latest/index.html) is an open-source framework for
+  creating machine learning (ML) weather forecasting systems, co-developed by ECMWF and a range of national meteorological 
+  services across Europe.
+- **Paper:** https://arxiv.org/abs/2412.15832
+
+## Get started
+
+🚨  **Note** `aifs-ens-1.0` requires `38GB` of GPU memory by default, it is possible to reduce this by setting `ANEMOI_INFERENCE_NUM_CHUNKS` to a multiple of 2.
+For example setting it to `16` reduces the memory requirements to `24GB` at the cost of throughput. Therefore issues may arise with lower end GPUs.
+
+
+To generate a new forecast using AIFS ENS, you can use [anemoi-inference](https://github.com/ecmwf/anemoi-inference). In the [example notebook](run_AIFS_ENS_v1.ipynb), a 
+step-by-step workflow is specified to run AIFS ENS using the HuggingFace model:
+
+1. **Install Required Packages and Imports**
+   - `torch==2.5.0 anemoi-inference[huggingface]==0.6.0 anemoi-models==0.6.0 anemoi-graphs==0.6.0 anemoi-datasets==0.5.23`
+   - `earthkit-regrid==0.4.0 'ecmwf-opendata>=0.3.19'`
+   - `flash_attn`
+2. **Retrieve Initial Conditions from ECMWF Open Data**
+   - Select a date
+   - Get the data from the [ECMWF Open Data API](https://www.ecmwf.int/en/forecasts/datasets/open-data)
+   - Get input fields
+   - Add the single levels fields and pressure levels fields
+   - Convert geopotential height into geopotential
+   - Create the initial state
+3. **Load the Model and Run the Forecast** 
+   - Download the Model's Checkpoint from Hugging Face
+   - Create a runner
+   - Run the forecast using anemoi-inference
+4. **Inspect the generated forecast**
+   - Plot a field
+
+🚨  **Note** we train AIFS using [`flash_attention`](https://github.com/Dao-AILab/flash-attention).
+The use of the 'Flash Attention' package also imposes certain requirements in terms of software and hardware, which can be found under 
+[Installation and Features](https://github.com/Dao-AILab/flash-attention?tab=readme-ov-file#installation-and-features) on the GitHub repo.
+
+🚨 **Note** the `aifs_ens_v1.0.ckpt` checkpoint just contains the model’s weights. It does not contain any information about the optimizer states,
+lr-scheduler states, etc.
+
+## Citation
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+If you use this model in your work, please cite it as follows:
+
+**BibTeX:**
+
+```
+@misc{lang2024aifscrps,
+      title={AIFS-CRPS: Ensemble forecasting using a model trained with a loss function based on the Continuous Ranked Probability Score}, 
+      author={Simon Lang and Mihai Alexe and Mariana C. A. Clare and Christopher Roberts and Rilwan Adewoyin and Zied Ben Bouallègue and Matthew Chantry and Jesper Dramsch and Peter D. Dueben and Sara Hahner and Pedro Maciel and Ana Prieto-Nemesio and Cathal O'Brien and Florian Pinault and Jan Polster and Baudouin Raoult and Steffen Tietsche and Martin Leutbecher},
+      year={2024},
+      eprint={2412.15832},
+      archivePrefix={arXiv},
+      primaryClass={physics.ao-ph},
+      url={https://arxiv.org/abs/2412.15832}, 
+}
+
+```
+
+**APA:**
+
+```
+Lang, S., Alexe, M., Clare, M. C., Roberts, C., Adewoyin, R., Bouallègue, Z. B., ... & Leutbecher, M. (2024). AIFS-CRPS: Ensemble forecasting using a model trained with a loss function based on the Continuous Ranked Probability Score. arXiv preprint arXiv:2412.15832.
+
+```
+
+
+## More Information
+
+[Find the paper here](https://arxiv.org/pdf/2412.15832)
+

aifs-ens-crps-1.0.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f176e2e018f2c8f54cbb8927f7ee64a948bfdb6960a4867645304e1f6e3134a
+size 921591109