zhangrenchao commited on
Commit
22b01d4
·
verified ·
1 Parent(s): 5b3329b

Update English model card

Browse files
Files changed (1) hide show
  1. README.md +44 -60
README.md CHANGED
@@ -1,20 +1,18 @@
1
  ---
2
- frameworks: PyTorch
3
  language:
4
  - en
5
- license: apache-2.0
6
  tags:
7
  - OneScience
8
  - Earth Science
9
  - Weather Downscaling
10
  - Diffusion Model
11
- - ERA5
12
- - CorrDiff
13
- - arxiv:2309.15214
14
- tasks: []
15
  datasets:
16
- - OneScience-Group/ERA5
 
17
  ---
 
18
  <p align="center">
19
  <strong>
20
  <span style="font-size: 30px;">CorrDiff</span>
@@ -23,50 +21,48 @@ datasets:
23
 
24
  # Model Introduction
25
 
26
- CorrDiff (Residual Corrective Diffusion Model) is a two-stage generative model for kilometer-scale regional weather downscaling. It downscales approximately 25 km ERA5 fields to approximately 2 km regional weather fields over Taiwan and synthesizes radar reflectivity that is not present in the input.
27
-
28
- Paper: Residual Corrective Diffusion Modeling for Km-scale Atmospheric Downscaling
29
 
 
30
  https://arxiv.org/abs/2309.15214
31
 
32
  # Model Description
33
 
34
- CorrDiff was proposed by NVIDIA and its collaborators and was trained with coarse-resolution ERA5 reanalysis and high-resolution WRF regional model data from Taiwan's Central Weather Administration.
35
- The model is suitable for converting coarse-resolution global weather fields into high-resolution regional weather fields and producing probabilistic weather predictions.
36
 
37
  # Use Cases
38
 
39
  | Scenario | Description |
40
  | :---: | :--- |
41
- | Regional weather downscaling training | Train CorrDiff with time-aligned coarse-resolution ERA5 inputs and high-resolution CWA-WRF targets. |
42
- | Local quick validation | Use synthetic paired data to check data loading, two-stage training, ensemble inference, and result visualization. |
43
- | Hugging Face / OneCode execution | Download the standalone model package, install dependencies, and run the scripts directly. |
44
- | Multi-GPU training | Launch multi-process training with `torchrun` after adapting the real-data pipeline for distributed training. |
 
 
45
 
46
  # Usage Guide
47
 
48
- ## 1. OneCode Usage
49
 
50
  Experience intelligent one-click AI4S programming through the OneCode online environment:
51
 
52
  [Click to Experience Intelligent One-Click AI4S Programming](https://web-2069360198568017922-iaaj.ksai.scnet.cn:58043/home)
53
 
54
- ## 2. Manual Installation and Usage
55
-
56
- **Hardware Requirements**
57
-
58
- - A GPU or DCU is recommended.
59
- - CPU can be used for import and small-scale connectivity verification; full training and inference will be slow.
60
- - DCU users must install DTK in advance. DTK 25.04.2 or above, or the OneScience recommended version matching your cluster, is recommended.
61
-
62
- ### Download the Model Package
63
 
64
  ```bash
65
  hf download OneScience-Group/CorrDiff --local-dir ./CorrDiff
66
  cd CorrDiff
67
  ```
68
 
69
- ### Install the Runtime Environment
 
 
 
 
 
 
70
 
71
  **DCU Environment**
72
 
@@ -90,55 +86,45 @@ pip install onescience[earth-gpu] -i http://mirrors.onescience.ai:3141/pypi/simp
90
 
91
  ### Training Data Introduction
92
 
93
- This repository uses synthetic paired data by default to validate training, ensemble inference, and evaluation:
94
 
95
- ```bash
96
- python scripts/fake_data.py --output data/era5_corrdiff.npz
97
- ```
98
-
99
- The synthetic file contains `input` with shape `[N, 12, 36, 36]` and `target` with shape `[N, 4, 448, 448]`. It is intended for pipeline validation only and does not represent real weather predictions.
100
 
101
- ERA5 inputs can be downloaded from the OneScience community:
102
 
103
- ```bash
104
- hf download --repo-type dataset OneScience-Group/ERA5 --local-dir ./data/era5
 
105
  ```
106
 
107
- The paired CWA-WRF resource can be downloaded from NVIDIA NGC:
108
 
109
  ```bash
110
- ngc registry resource download-version "nvidia/modulus/modulus_datasets_cwa:v1"
111
  ```
112
 
113
- ERA5 alone is insufficient for supervised CorrDiff training. Real data must be time-aligned and preprocessed into the paired `input` and `target` interface expected by the scripts. Verify the data path in `conf/config.yaml` before training.
114
-
115
  ### Training
116
 
117
- Single GPU:
118
-
119
  ```bash
120
  python scripts/train.py
121
  ```
122
 
123
- Multi-GPU:
124
 
125
  ```bash
126
- torchrun --nproc_per_node=8 --nnodes=1 --rdzv_id=1000 --rdzv_backend=c10d --max_restarts=0 --master_addr="localhost" --master_port=29500 scripts/train.py
127
  ```
128
 
129
- Training outputs:
130
 
131
  ```text
132
- data/checkpoints/regression_model.pth
133
- data/checkpoints/diffusion_model.pth
134
- data/checkpoints/training_history.npz
135
  ```
136
 
137
- Training first fits the conditional-mean regression model and then freezes it while fitting the residual diffusion model. `training_history.npz` stores the regression and diffusion denoising losses.
138
-
139
  ### Training Weights
140
 
141
- The training command generates `regression_model.pth` and `diffusion_model.pth` from `data/era5_corrdiff.npz`. Weights produced with synthetic data validate the training and inference pipeline only and do not provide real weather forecasting skill. This repository does not present them as pretrained weights reproducing the paper.
142
 
143
  ### Inference
144
 
@@ -146,29 +132,25 @@ The training command generates `regression_model.pth` and `diffusion_model.pth`
146
  python scripts/inference.py
147
  ```
148
 
149
- Prediction output:
150
 
151
  ```text
152
  result/output/predictions.npz
153
  ```
154
 
155
- The file stores the ensemble predictions, ensemble mean, ensemble standard deviation, input, and target generated by the current trained weights.
156
-
157
  ### Evaluation and Visualization
158
 
159
  ```bash
160
  python scripts/result.py
161
  ```
162
 
163
- Evaluation outputs:
164
 
165
  ```text
166
- result/output/metrics.json
167
- result/output/prediction_comparison.png
168
  ```
169
 
170
- `metrics.json` stores MAE and CRPS for the four output variables. `prediction_comparison.png` compares the ensemble mean, ensemble standard deviation, and target. Metrics and figures generated from synthetic data are pipeline checks, not paper results.
171
-
172
  # Official OneScience Resources
173
 
174
  | Platform | OneScience Main Repository | Skills Repository |
@@ -178,4 +160,6 @@ result/output/prediction_comparison.png
178
 
179
  # Citation and License
180
 
181
- - This repository is a reproduction of the original CorrDiff paper.
 
 
 
1
  ---
2
+ license: apache-2.0
3
  language:
4
  - en
 
5
  tags:
6
  - OneScience
7
  - Earth Science
8
  - Weather Downscaling
9
  - Diffusion Model
10
+ frameworks: PyTorch
 
 
 
11
  datasets:
12
+ - ERA5
13
+ - CWA-WRF
14
  ---
15
+
16
  <p align="center">
17
  <strong>
18
  <span style="font-size: 30px;">CorrDiff</span>
 
21
 
22
  # Model Introduction
23
 
24
+ CorrDiff is a two-stage generative model for kilometer-scale atmospheric downscaling. It first predicts a high-resolution mean with conditional regression and then uses a residual diffusion model to generate local stochastic details and ensemble forecasts.
 
 
25
 
26
+ Paper: Residual Corrective Diffusion Modeling for Km-scale Atmospheric Downscaling
27
  https://arxiv.org/abs/2309.15214
28
 
29
  # Model Description
30
 
31
+ CorrDiff was proposed by an NVIDIA research team. The model is trained using ERA5 reanalysis data and high-resolution regional CWA-WRF simulation data, and performs weather-field downscaling through regression and residual diffusion. It is suitable for high-resolution weather-field generation, kilometer-scale downscaling, and ensemble uncertainty analysis.
 
32
 
33
  # Use Cases
34
 
35
  | Scenario | Description |
36
  | :---: | :--- |
37
+ | Weather downscaling | Downscale `36x36` coarse-resolution condition fields to `448x448`. |
38
+ | Ensemble forecasting | Generate multiple possible high-resolution results through diffusion sampling. |
39
+ | Radar reflectivity generation | Generate high-resolution maximum radar reflectivity that is absent from the ERA5 condition fields, supporting analysis of fine-scale precipitation-system structures. |
40
+ | Extreme-weather risk analysis | Use ensemble members and spatial uncertainty to characterize the possible evolution of local heavy precipitation and other high-impact weather. |
41
+ | Local engineering validation | Use a small amount of synthetic data to check training, inference, and evaluation workflows. |
42
+ | Multi-GPU training | Launch distributed training with `torchrun`. |
43
 
44
  # Usage Guide
45
 
46
+ ## 1.OneCode
47
 
48
  Experience intelligent one-click AI4S programming through the OneCode online environment:
49
 
50
  [Click to Experience Intelligent One-Click AI4S Programming](https://web-2069360198568017922-iaaj.ksai.scnet.cn:58043/home)
51
 
52
+ ## 2.Download and Installation
 
 
 
 
 
 
 
 
53
 
54
  ```bash
55
  hf download OneScience-Group/CorrDiff --local-dir ./CorrDiff
56
  cd CorrDiff
57
  ```
58
 
59
+ ### Environment Dependencies
60
+
61
+ **Hardware Requirements**
62
+
63
+ - A GPU or DCU is recommended.
64
+ - CPU can be used for connectivity validation with a small configuration; full training and inference are slower.
65
+ - DCU users must install DTK in advance. DTK 25.04.2 or above, or the OneScience-recommended version matching the current cluster, is recommended.
66
 
67
  **DCU Environment**
68
 
 
86
 
87
  ### Training Data Introduction
88
 
89
+ By default, two synthetic samples following the same protocol are used to validate the engineering workflow. The data are saved as `data/corrdiff.npz`, and the synthetic data do not represent real weather distributions.
90
 
91
+ The synthetic data retain the input and output specifications of the Taiwan downscaling experiment in the paper: 12-channel `36x36` condition fields and 4-channel `448x448` target fields.
 
 
 
 
92
 
93
+ Real data must be preprocessed and converted to the following NPZ training protocol. This protocol is consistent with the model input specification but is not equivalent to the original datasets' download format.
94
 
95
+ ```text
96
+ input: float32 [N,12,36,36]
97
+ target: float32 [N,4,448,448]
98
  ```
99
 
100
+ `fake_data.py` automatically writes the `protocol` and `data_source` protocol metadata. These fields must be retained when using real data.
101
 
102
  ```bash
103
+ python scripts/fake_data.py
104
  ```
105
 
 
 
106
  ### Training
107
 
 
 
108
  ```bash
109
  python scripts/train.py
110
  ```
111
 
112
+ Multi-GPU training can use:
113
 
114
  ```bash
115
+ torchrun --nproc_per_node=8 scripts/train.py
116
  ```
117
 
118
+ Training sequentially optimizes the conditional-regression and residual-diffusion stages and saves a checkpoint and overall training metrics. The default configuration is intended for quick workflow validation. Formal experiments should use the data scale, model configuration, and training schedule corresponding to the paper.
119
 
120
  ```text
121
+ result/checkpoints/corrdiff.pt
122
+ result/training/metrics.json
 
123
  ```
124
 
 
 
125
  ### Training Weights
126
 
127
+ This repository will provide CorrDiff training weights in the `weight/` folder. The weight files will be uploaded soon and are expected to be available in the near future.
128
 
129
  ### Inference
130
 
 
132
  python scripts/inference.py
133
  ```
134
 
135
+ Inference loads the training checkpoint, generates high-resolution ensemble forecasts according to the configuration, and saves the results to:
136
 
137
  ```text
138
  result/output/predictions.npz
139
  ```
140
 
 
 
141
  ### Evaluation and Visualization
142
 
143
  ```bash
144
  python scripts/result.py
145
  ```
146
 
147
+ Evaluation reports overall MAE, RMSE, ensemble CRPS, and ensemble spread, and generates an ensemble diagnostic figure. Synthetic-data results are only for validating the engineering workflow and do not represent the full performance reported in the paper.
148
 
149
  ```text
150
+ result/evaluation/metrics.json
151
+ result/evaluation/ensemble_diagnostics.png
152
  ```
153
 
 
 
154
  # Official OneScience Resources
155
 
156
  | Platform | OneScience Main Repository | Skills Repository |
 
160
 
161
  # Citation and License
162
 
163
+ This repository is a reproduction of the original CorrDiff paper.
164
+
165
+ The use of the code and data in this repository remains subject to the licenses and terms of use of their respective projects.