EQC Quality Assessment: "Assessing the impact of spatial scale and temporal trends on seasonal forecast quality"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: resolution_q06  |  Category: Seasonal_Forecasts
Section: Assessing the impact of spatial scale and temporal trends on seasonal forecast quality > Quality assessment question
---
* How does the quality of seasonal forecasts change when looking at different sized areas? What are the associated limitations?
* How do long term temporal trends impact the quality of seasonal forecasts?

The effectiveness of seasonal forecasts in predicting the development of key climate variables—such as temperature, precipitation and sea surface temperatures (SSTs)—varies considerably by variable, region, season, lead time and the state of large-scale climate drivers like ENSO (El Niño–Southern Oscillation). This notebook investigates how a potential trend in the data (e.g. as discussed in [Greuell et al. (2019) \[1\]](https://doi.org/10.5194/hess-23-371-2019)) along with selection of regions at various spatial scales (e.g. as discussed in [Prodhomme et al. (2021) \[2\]](https://doi.org/10.1007/s00382-021-05828-3), [Gubler et al. (2020) \[3\]](https://doi.org/10.1175/WAF-D-19-0106.1) [Quaglia et al. (2021) \[4\]](https://doi.org/10.1007/s00382-021-05895-6)) affect forecasting quality. For the purpose of this assessment, seasonal forecast quality is measured by the temporal correlation of the ensemble mean with ERA5 reanalysis.


EQC Quality Assessment: "Assessing the impact of spatial scale and temporal trends on seasonal forecast quality"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: resolution_q06  |  Category: Seasonal_Forecasts
Section: Assessing the impact of spatial scale and temporal trends on seasonal forecast quality > Quality assessment statement
---
These are the key outcomes of this assessment

* The forecast quality (ensemble mean correlation) of t2m varies depending on the selected spatial scale and the choice of the regions or locations.
  * There is a clear drop in forecast quality when moving from the global scale to continental scale (Europe), to a national scale (Germany), and finally to a city scale (Bonn).
  * This is not the case when comparing the global scale, to the Pacific and NINO region, where the forecast quality remains largely constant, or even better, the smaller the selected region.
  * The regional mean for NINO3.4 is better correlated to the reanalysis than the global mean.
  * The nearest grid cell to Bonn (an example city in Germany) has low correlation and most of the quality can be attributed to the underlying trend in the dataset.
  * In contrast, the nearest grid cell to Addis Ababa (an example city in Eastern Africa) has some correlation which is retained even after detrending.
* For regions and locations which have a visible trend in t2m, the trend inflates the correlation of t2m compared to reanalysis.
  * For Europe, most of the visible correlation can be attributed to the underlying trend.
  * This contrasts with the region of greater horn of Africa, where less of the visible correlation can be attributed to the underlying trend.
* Users are generally advised to avoid selecting small regions or single grid cells when using seasonal forecasts as they are only expected to be skillful in predicting large scale variations in monthly or seasonal deviations. Selecting larger regions based on a common climatology generally results in increased robustness and signal to noise.
```


EQC Quality Assessment: "Assessing the impact of spatial scale and temporal trends on seasonal forecast quality"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: resolution_q06  |  Category: Seasonal_Forecasts
Section: Assessing the impact of spatial scale and temporal trends on seasonal forecast quality > Methodology
---
This notebook provides an assessment of the quality of seasonal monthly forecast temperature anomalies derived from the monthly means (https://cds.climate.copernicus.eu/datasets/seasonal-monthly-single-levels) through comparison with the ERA5 reanalysis (https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels-monthly-means). The hindcast anomalies are calculated from the monthly means as the anomalies catalogue entry only provides the real time forecasts. Data were accessed from the CDS.

Using this data the anomaly was calculated for the two meter temperature (t2m), against the period 1993-2024, and was detrended assuming a linear trend. Note that this period contains both hindcast and forecast, which are not entirely equivalent with respect to initialization, and contain 25 and 51 ensemble members respectively. A dataset was then generated containing both the original anomaly and the detrended. The analysis was then carried out globally and for three regions chosen based on effects of trend on the correlation.

First, the global case was considered both including and excluding trend, plotting anomaly for the ensemble members, ensemble mean, and ERA5. The correlation was also calculated.

To further consider the correlation, maps were generated to more clearly view the spatial variations of correlation in anomaly (as can seen on the [C3S verification page](https://confluence.ecmwf.int/display/CKB/C3S+seasonal+forecasts+verification+plots)) between the seasonal forecast and ERA5. Based on these maps, a set of regions were selected for further study, based on the magnitude of the correlation.
Plots were then generated for nested domains. Three spatial scales were considered, a larger, continental scale, a smaller country scale and a much smaller city scale (a single grid box in the 1x1 degree grid). Data were selected for each domain based on a lat/lon box.

The resulting plots, displaying the ensemble members, ensemble mean, and ERA5, aim to visualise the variation in quality based on the selected regions of interest. This was also done for the detrended anomaly, and correlation was also calculated for each of the domains.

**[](section-1)**
 * Choose a selection of forecast systems and model versions, hindcast period (normally 1993-2016 to align with the C3S common hindcast period), forecast and leadtime months
     * Ensure that the ERA5 and Seasonal forecast data are regridded to the same grid by using the grid-keyword to the cdsapi.
 * Compute the anomaly of the forecast and reanalysis data
 * The data is detrended assuming a linear trend in the data from start year to end year
 * Both the original and detrended data is saved for the following analysis

**[](section-2)**
 * Compute the ensemble mean of the forecasted anomaly
 * Plot the correlation of forecast and reanalysis data, before and after performing the detrending

**[](section-3)**
 * Correlation is plotted for each grid point on a map
 * Based on this, regions are selected to investigate further; Addis Ababa, Bonn, NINO3.4

**[](section-4)**
 * Plots are generated for the regions chosen in the previous section

**Key limitations:**
 * This assessment primarily considers the correlation between the ensemble mean and ERA5, while the full ensemble spread is visualized in some of the plots, the maps of correlation only use the ensemble mean.
 * The method used to detrend the data is very simple, and thus may impact the quality of the detrended data.
 * The usage of ERA5 as ground truth, while convenient, may be considered a limitation as it is a reanalysis and not observation.
 * The region plots use data at scales that may consider only one or a few grid cells, this is generally not a large enough domain to expect skillful prediction of large scale variations in monthly or seasonal deviations.
 * The assessment in this notebook only makes use of one forecasting system, considering one lead time. While the code should work for other models, and lead times this has not been shown explicitly in this assessment. Therefore, the results outlined here are specific to this forecasting system, period and lead time, further study would need to be conducted to see if the conclusions are general.


EQC Quality Assessment: "Assessing the impact of spatial scale and temporal trends on seasonal forecast quality"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: resolution_q06  |  Category: Seasonal_Forecasts
Section: Assessing the impact of spatial scale and temporal trends on seasonal forecast quality > Analysis and results > 1. Choose data to use and setup code
---
This section contains the setup and data processing needed for performing the analysis.

Import external code libraries needed.

collapsable code cells - note that the code cell will be collapsed by the addition of a 'hide-input' tag when the Jupyter Book page is built

Define functions to be used throughout the notebook.

make the map global rather than have it zoom in to
the extents of any plotted data

This notebook uses the t2m monthly means from one forecasting system, in this case from ECMWF (system code 51), for a forecast produced in May for leadtime months June, July and August. This was requested for both forecast and hindcast periods, thus there is a difference in the number of ensemble members before and after 2016.

Seasonal forecast
Use dataarrays directly... for now
da_seasonal = xr.concat(dataarrays, "leadtime_month")

(section-2)=


EQC Quality Assessment: "Assessing the impact of spatial scale and temporal trends on seasonal forecast quality"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: resolution_q06  |  Category: Seasonal_Forecasts
Section: Assessing the impact of spatial scale and temporal trends on seasonal forecast quality > Analysis and results > 2. Comparing global correlation
---
Compute the ensemble mean of the seasonal forecast anomaly data.

leadtime here maps to index of leadtime_month: 0 is 2, here 0 is forecast for june from may., 2 is forecast for august from may (leadtime_month = 4)

Plot the original anomaly data for each ensemble member, the ensemble mean, and the reanalysis anomaly.
Please note that the ensemble member points are coloured by number to highlight the switch from the hindcast configuration (25 members) to the forecasts (51 members).


EQC Quality Assessment: "Assessing the impact of spatial scale and temporal trends on seasonal forecast quality"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: resolution_q06  |  Category: Seasonal_Forecasts
Section: Assessing the impact of spatial scale and temporal trends on seasonal forecast quality > Analysis and results > 3. Map of correlation for detrended data
---
The correlation of the detrended data is plotted compared to reanalysis data for each grid cell on a map.

The hatched area is where the correlation exceeds the critical value.

It can be seen in the above plots that there is a high correlation over the North American Great Lakes. This is likely due to the fact that they are large enough that the heat retention impacts correlation skill at one month lead time.


EQC Quality Assessment: "Assessing the impact of spatial scale and temporal trends on seasonal forecast quality"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: resolution_q06  |  Category: Seasonal_Forecasts
Section: Assessing the impact of spatial scale and temporal trends on seasonal forecast quality > Analysis and results > 3. Map of correlation for detrended data > Selecting regions
---
Based on the correlation maps, regions are selected to investigate further; Addis Ababa, Bonn, NINO3.4.

(section-4)=


EQC Quality Assessment: "Assessing the impact of spatial scale and temporal trends on seasonal forecast quality"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: resolution_q06  |  Category: Seasonal_Forecasts
Section: Assessing the impact of spatial scale and temporal trends on seasonal forecast quality > Analysis and results > 4. Plot and describe results > Discussion
---
Looking at the comparison of different regions at different spatial scales a few conclusions can be drawn.

For regions with a visible trend, that trend inflates the correlation for t2m compared against analysis. This can be seen for Europe, where comparing the detrended and trended plots, the correlation is greater for the data with trend in Europe. Hence, for Europe, most of the visible correlation can be attributed to the underlying trend. This can be compared with the region of greater horn of Africa. Where comparing the trended and detrended case indicates that less of the visible correlation can be attributed to the underlying trend.

For the selected regions, forecast quality depending on scale varies. Looking at the Europe, Germany, Bonn case, there is a drop in forecast quality as the scale decreases. This cannot be observed in the NINO3.4 where the quality remains more or less constant, despite the decrease in region. The Greater Horn of Africa, where the drop off in quality with scale can be seen somewhat in the case of Addis Ababa, but not on the same scale as can be seen in the Europe case.

Considering this, and the anomalies for the different regions at different scales, it can be seen that the region of NINO3.4 has a better correlation with the reanalysis than the global regional mean, while Bonn has a low correlation where most of the quality can be attributed to the underlying trend. Addis Ababa has some correlation, which is retained even after detrending. The high correlations for the NINO3.4 region are partially dependent on the start date and forecast length, which can be seen in the [SST indices plots on the C3S verification page](https://confluence.ecmwf.int/display/CKB/C3S+seasonal+forecasts+verification+plots)), though it should be noted that these plots use sea surface temperature and not t2m. Forecast quality, and seasonal forecast biases, generally vary with start date and lead time, as seen in some of the other assessments.


EQC Quality Assessment: "Assessing the impact of spatial scale and temporal trends on seasonal forecast quality"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: resolution_q06  |  Category: Seasonal_Forecasts
Section: Assessing the impact of spatial scale and temporal trends on seasonal forecast quality > ℹ If you want to know more > Key resources
---
* Seasonal forecast monthly statistics on single levels: [10.24381/cds.68dd14c3](https://doi.org/10.24381/cds.68dd14c3)
 * ERA5 monthly averaged data on single levels from 1940 to present:  [10.24381/cds.f17050d7](https://doi.org/10.24381/cds.f17050d7)


EQC Quality Assessment: "Assessing the impact of spatial scale and temporal trends on seasonal forecast quality"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: resolution_q06  |  Category: Seasonal_Forecasts
Section: Assessing the impact of spatial scale and temporal trends on seasonal forecast quality > ℹ If you want to know more > Code libraries used:
---
* [C3S EQC custom functions](https://github.com/bopen/c3s-eqc-automatic-quality-control/tree/main/c3s_eqc_automatic_quality_control), `c3s_eqc_automatic_quality_control`,  prepared by [B-Open](https://www.bopen.eu/)
 * xarray
 * numpy
 * matplotlib
 * cartopy
 * linear_model from sklearn


EQC Quality Assessment: "Assessing the impact of spatial scale and temporal trends on seasonal forecast quality"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: resolution_q06  |  Category: Seasonal_Forecasts
Section: Assessing the impact of spatial scale and temporal trends on seasonal forecast quality > ℹ If you want to know more > References
---
[1] Greuell, W., Franssen, W. H. P., and Hutjes, R. W. A.: Seasonal streamflow forecasts for Europe – Part 2: Sources of skill, Hydrol. Earth Syst. Sci., 23, 371–391, https://doi.org/10.5194/hess-23-371-2019, 2019.

[2] Prodhomme, C., Materia, S., Ardilouze, C. et al. Seasonal prediction of European summer heatwaves. Clim Dyn 58, 2149–2166 (2022). https://doi.org/10.1007/s00382-021-05828-3

[3] Gubler, S., and Coauthors, 2020: Assessment of ECMWF SEAS5 Seasonal Forecast Performance over South America. Wea. Forecasting, 35, 561–584, https://doi.org/10.1175/WAF-D-19-0106.1

[4] Calì Quaglia, F., Terzago, S. & von Hardenberg, J. Temperature and precipitation seasonal forecasts over the Mediterranean region: added value compared to simple forecasting methods. Clim Dyn 58, 2167–2191 (2022). https://doi.org/10.1007/s00382-021-05895-6


EQC Quality Assessment: "Assessing possible outcomes of seasonal temperature forecast"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: uncertainty-quality-flags_q01  |  Category: Seasonal_Forecasts
Section: Assessing possible outcomes of seasonal temperature forecast > Quality assessment questions
---
* **How can I assess the forecast temperatures for two months time?**
* **Do the different forecast systems provide a consistent outlook?**

Seasonal forecast in C3S provides a prediction of the near future. For two months prediction time, emphasis should be put on a probabilistic approach and understanding the uncertainty in the prediction [[1]](https://doi.org/10.1098/rsif.2013.1162). To this end, here we assess important characteristics of seasonal forecasts, how they can be compared, interpreted and used.


EQC Quality Assessment: "Assessing possible outcomes of seasonal temperature forecast"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: uncertainty-quality-flags_q01  |  Category: Seasonal_Forecasts
Section: Assessing possible outcomes of seasonal temperature forecast > Quality assessment statement
---
These are the key outcomes of this assessment

* This notebook shows one way to visualise the range of outcomes covered by the ensemble members in seasonal forecast systems from C3S and compare it with the forecast system climatology as a reference.
* The resulting plots show the uncertainty in the forecast (range of outcomes for each forecast system and between forecast systems), how the seasonal forecast differs from the model climatology, and how likely it is that the coming months will be in the lower, middle or upper terciles of the climatology.
* For the specific example provided, forecasts issued in June 2023 and valid for 2m air temperature for a region over Southern Norway in July 2023 shows that:
    * All forecast systems provide a consistent outlook that the month will end up in the upper tercile of the climatology (probabilities between 45 and 61% for the different systems).
    * The climatology of the different forecast systems are quite different, hence they need to be bias adjusted before their face-value  can be combined in a multi-model forecast. This emphasises the need for always comparing the forecasts with the hindcast climatology.
* If the forecast systems yield inconsistent outlooks, a multi system summary (see below) can be employed to determine whether there's a broad consensus with only a few models deviating, or if there's a complete disagreement in forecasted probabilities. In the former scenario, the divergent model systems might be seen as outliers highlighting the possibility of different outcome than the majority. While in the latter, the reliability of the forecast is reduced.
* Seasonal forecasts are inherently probabilistic and must be interpreted considering the chaotic nature of the system and the forecast systems' capabilities and limitations. In general, multi-model system forecasts often provide better estimates of the uncertainty than applying a single system. Due to the long lead times involved and associated uncertainty, forecast skill can only be expected for large-scale features in space and time and using e.g. probabilities for terciles can be useful in decision-making.
```
![tercile_distribution_t2m_without_label.png](attachment:94c8182f-0d8a-4b57-bad1-b41d5cf7f1ba.png)


EQC Quality Assessment: "Assessing possible outcomes of seasonal temperature forecast"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: uncertainty-quality-flags_q01  |  Category: Seasonal_Forecasts
Section: Assessing possible outcomes of seasonal temperature forecast > Methodology
---
This code in this notebook enables an assessment of the range of possible outcomes for two months prediction time of seasonal forecast systems, in the case of near surface temperature. For reference, the forecasts are presented together with the expected (normal) range of temperatures for the month and region of interest. The normal range is based on so-called hindcasts of past weather from the same forecasting system.

This quality assessment does not assess the historical quality of the seasonal forecast, as the forecasts (e.g. hindcasts) are not compared with historical outcomes (e.g. observations or re-analyses like ERA5). This will be done in another notebook, and is also done [here](https://confluence.ecmwf.int/display/CKB/C3S+seasonal+forecasts+verification+plots), in verification plots for the C3S graphical products.

**[](section-1)**
   - Choose a selection of forecast systems and versions, forecast start time (year and month), hindcast period (normally 1993-2016), region (latitude-longitude box).
   - Define a weighting function for the gridded data, and if needed conversions for the variables used.

**[](section-2)**
   - Retrieve data for monthly means of all ensemble members for the selected parameters above, from the data catalogue ”Seasonal forecast monthly statistics on single levels”, for both forecasts and hindcasts
   - Compute the spatial mean of 2 m temperature.

**[](section-3)**
   - Showing the normal range of each forecast system (hindcast distribution) together with the forecast outcomes. Included in the plots are probabilities of near surface temperature forecast within the lower, middle and upper terciles of the normal range.

**[](section-4)**
 - The forecast probabilities are combined and visualised at the end in a multi-model plot.


EQC Quality Assessment: "Assessing possible outcomes of seasonal temperature forecast"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: uncertainty-quality-flags_q01  |  Category: Seasonal_Forecasts
Section: Assessing possible outcomes of seasonal temperature forecast > Analysis and results > 1. Choose data to use and setup code
---
In this section, the customisable options of the notebook are set. These variables consists of:

- Model forecast system
    - forecast centre
    - system version
- Time
    - forecast year
    - first year of the model hindcast
    - last year of the model hindcast
    - forecast month
- Region of interest
    - name of the region (only used in figure-captions)
    - latitudes given as slice(maxlat, minlat) in degrees (north is positive values)
    - longitudes given as slice(minlon, maxlon) in degrees (east is positive values)
- Weather parameters
    - name of variable in grib-file
    - name of variable in CADS-api
    - name of the three tercile categories (lower, middle and upper tercile)
- Download parameters
    - chunk size of the data
    - number of concurrent request for parallel download
- Combination of originating centre and model system (Operational forecast model systems per March 2024):
    - centre = "ecmwf", system = "51"
    - centre = "ukmo", system = "602"
    - centre = "meteo_france", system = "8"
    - centre = "dwd", system = "21"
    - centre = "cmcc", system = "35"
    - centre = "ncep", system = "2"
    - centre = "jma", system = "3"
    - centre = "eccc", system = "2"
    - centre = "eccc", system = "3"

To see the different alternatives see the API-request at the bottom of [this page](https://cds.climate.copernicus.eu/datasets/seasonal-monthly-single-levels?tab=form)

Note: if you want to change parameter you might also want to add conversion functions in this section. For example for precipitation you might want to convert from precipitation rate to total precipitation amount

The data is downloaded to a 1 degree lat/lon grid from 89.5 N to 89.5 S and 179.5 W to 179.5 E which is the grid for the latest model versions of all models except for JMA. The JMA data, which is on a 1.25 x 1.25 grid, is therefore interpolated. This is as a convencience to ensure that the lat-lon slices return the same grids for the different systems, and that we can use similar requests to cds. Otherwise the interpolation would not be needed since we are doing an area average. Also, the 'weights' flag will be set to 'True' in the next section, which means the area average is carried out taking into account the variation of the cell areas with cosine of latitude in the regular lat-lon grid.

Download parameters
Construct the request to be sent to CADS
Define the weighting function
Convert from K to C if variable is 2m air temperature

(section-2)=


EQC Quality Assessment: "Assessing possible outcomes of seasonal temperature forecast"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: uncertainty-quality-flags_q01  |  Category: Seasonal_Forecasts
Section: Assessing possible outcomes of seasonal temperature forecast > Analysis and results > 2. Seasonal forecast and hindcast data retrieval and area average
---
In this section the seasonal data are downloaded and transformed into regionalised means. The time is transformed into two variables ‘time/indexing_time’ and ‘forecast month’. For burst ensemble models ‘time’ is the time when all ensembles are initialised. For lagged ensemble models (JMA, NCEP and UK MetOffice) the ‘indexing_time’ must be used instead of ‘time’ in order to get the same variable, since ensemble members have different initialisation time (See more on this [confluence-page](https://confluence.ecmwf.int/display/CKB/Seasonal+forecasts+and+the+Copernicus+Climate+Change+Service#SeasonalforecastsandtheCopernicusClimateChangeService-%22Burst%22vs.%22lagged%22mode)).  The ‘forecast month’ is the lead time month from month 1 to 6.

Store the downloaded dataset in the dictionary

```text
100%|██████████| 24/24 [00:00<00:00, 34.70it/s]
100%|██████████| 1/1 [00:00<00:00, 47.55it/s]
100%|██████████| 24/24 [00:00<00:00, 48.03it/s]
100%|██████████| 1/1 [00:00<00:00, 54.61it/s]
100%|██████████| 24/24 [00:00<00:00, 63.89it/s]
100%|██████████| 1/1 [00:00<00:00, 43.95it/s]
100%|██████████| 24/24 [00:00<00:00, 39.22it/s]
100%|██████████| 1/1 [00:00<00:00, 51.80it/s]
100%|██████████| 24/24 [00:00<00:00, 50.26it/s]
100%|██████████| 1/1 [00:00<00:00, 32.12it/s]
100%|██████████| 24/24 [00:00<00:00, 33.74it/s]
100%|██████████| 1/1 [00:00<00:00, 45.89it/s]
100%|██████████| 24/24 [00:01<00:00, 20.87it/s]
100%|██████████| 1/1 [00:00<00:00,  3.85it/s]
100%|██████████| 24/24 [00:00<00:00, 33.76it/s]
100%|██████████| 1/1 [00:00<00:00, 43.51it/s]
100%|██████████| 24/24 [00:00<00:00, 47.80it/s]
100%|██████████| 1/1 [00:00<00:00, 46.99it/s]
```

(section-3)=


EQC Quality Assessment: "Assessing possible outcomes of seasonal temperature forecast"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: uncertainty-quality-flags_q01  |  Category: Seasonal_Forecasts
Section: Assessing possible outcomes of seasonal temperature forecast > Analysis and results > 3. Make histogram plots for each forecast system
---
This section extracts data for the selected region from the monthly data on single levels and compute the spatial mean of the chosen variable. Then density plots are created for each lead time month. The resulting plots show the uncertainty in the forecast (range of outcomes), how the seasonal forecast differs from the model climatology, and how likely it is that the coming months will be in the lower, middle or upper terciles of the climatology.

Set up for for multi-ensemble histogram plot
Histogram plot for each model system
Add background color and text
Text and labels
Text below each plot

(section-4)=


EQC Quality Assessment: "Assessing possible outcomes of seasonal temperature forecast"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: uncertainty-quality-flags_q01  |  Category: Seasonal_Forecasts
Section: Assessing possible outcomes of seasonal temperature forecast > Analysis and results > 4. Summarise the probabilities for each system
---
This section summarises the forecasted probabilites of the temperature being colder, near average or warmer than their respective hindcast climatologies for all systems.
The figure produced reinforces the findings from the figures produced in the last section, that it is more likely to be warmer than usual, compared to the model hindcast climatology than it is to be average or colder compared to the hindcast.

Plotting the multi-ensemble histogram

Figure 1 shows the uncertainty, the range of outcomes covered by the ensemble members in the different seasonal forecast systems separately, while Figure 2 shows the probabilities from all the individual forecast systems for the lower, middle, and upper tercile together. While all systems show the same tendency towards that the upper tercile has the highest probability, there is still a substantial chance for the lower and middle tercile to happen as well. In fact, the probability of not ending up in the upper tercile is still around 50%.

To answer to the questions stated initially; The generated plots assess the forecasted temperatures with a two months lead time and indicate that the most likely tercile is the upper tercile, but that it is almost equally likely that we will not end up in the upper tercile. In particular, Figure 2 shows that the different seasonal forecast systems provide a consistent outlook. In general, multi-model system forecasts often provide better forecasts and estimates of the uncertainty than applying a single model system (e.g. Doblas-Reyes et al., 2009 \[[2](https://doi.org/10.1002/qj.464)\]).

There are multiple ways to visualize seasonal forecasts and their associated uncertainty as probabilistic forecasts. C3S provides more examples of this as  in the graphical products: https://climate.copernicus.eu/charts/packages/c3s_seasonal/


EQC Quality Assessment: "Assessing possible outcomes of seasonal temperature forecast"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: uncertainty-quality-flags_q01  |  Category: Seasonal_Forecasts
Section: Assessing possible outcomes of seasonal temperature forecast > ℹ If you want to know more > Key resources
---
- Seasonal forecast single level anomalies: DOI: [10.24381/cds.7e37c951](https://doi.org/10.24381/cds.7e37c951)
 - Seasonal forecast monthly statistics single level: DOI: [10.24381/cds.68dd14c3](https://doi.org/10.24381/cds.68dd14c3)


EQC Quality Assessment: "Assessing possible outcomes of seasonal temperature forecast"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: uncertainty-quality-flags_q01  |  Category: Seasonal_Forecasts
Section: Assessing possible outcomes of seasonal temperature forecast > ℹ If you want to know more > Code libraries used
---
- xarray
 - numpy
 - express and figure_factory from plotly
 - [C3S EQC custom functions](https://github.com/bopen/c3s-eqc-automatic-quality-control/tree/main/c3s_eqc_automatic_quality_control), `c3s_eqc_automatic_quality_control`, prepared by [B-Open](https://www.bopen.eu/)


EQC Quality Assessment: "Assessing possible outcomes of seasonal temperature forecast"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: uncertainty-quality-flags_q01  |  Category: Seasonal_Forecasts
Section: Assessing possible outcomes of seasonal temperature forecast > ℹ If you want to know more > References
---
[[1]](https://doi.org/10.1098/rsif.2013.1162) - Weisheimer A. and Palmer T. N.. 2014 On the reliability of seasonal climate forecasts *J. R. Soc.* Interface. **11:** 20131162.
https://doi.org/10.1098/rsif.2013.1162

[[2]](https://doi.org/10.1002/qj.464) - Doblas-Reyes, F.J., Weisheimer, A., Déqué, M., Keenlyside, N., McVean, M., Murphy, J.M., Rogel, P., Smith, D. and Palmer, T.N. (2009), Addressing model uncertainty in seasonal and annual dynamical ensemble forecasts. Q.J.R. Meteorol. Soc., 135: 1538-1559.
https://doi.org/10.1002/qj.464


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts
---
Production date: 07-05-2025

Produced by: Sandro Calmanti, Alessandro Dell'Aquila (ENEA)


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Quality assessment questions
---
* **Can I improve the skill of seasonal forecasts by using a multi-model ensemble of climate predictions?**
* **Are there robust strategies for creating multi-model ensembles?**

C3S provides a multi-model ensemble of seasonal climate predictions produced by nine forecast centres in Europe, North America, Japan and Australia. The development of a European multi-model ensemble for seasonal forecasting has been supported throughout several research programs ( [DEMETER](https://cordis.europa.eu/article/id/81812-demeter-improves-european-weather-forecasts) / [ENSEMBLES](https://ensembles-eu.metoffice.gov.uk/docs/Ensembles_final_report_Nov09.pdf) / [EUROSIP](https://www.ecmwf.int/en/forecasts/documentation-and-support/long-range/seasonal-forecast-documentation/eurosip-user-guide/eurosip-operational-history)) which have progressively contributed to the improvement of the underpinning climate models (e.g. [[1]](https://doi.org/10.1029/2009GL040896), [[2]](https://doi.org/10.1029/2007GL030740), [[3]](https://doi.org/10.1007/s00382-011-1061-x) ). The multi-model ensemble available through C3S is therefore the outcome of a long lasting endeavour, enriched with similar predictions derived from other non-european forecasting centres.

In principle, it is possible to use the seasonal predictions from different centers as a single multi-model ensemble to increase the reliability and the overall value of the forecast.

However, handling multi-model ensembles to create climate information on a regular basis, is a complex and resource intensive task. This article summarises the significant experiences in working with multi-model ensembles with the objective of highlighting the potential added value, the challenges and limitations in the use of multi-model ensembles


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Quality assessment statement
---
These are the key outcomes of this assessment

* The multi-model ensemble (MME) approach consistently outperforms individual models in terms of skill metrics such as temporal correlation and probabilistic accuracy, particularly when the combined models are developed independently
* The use of multi-model ensembles allows for a better assessment of forecast uncertainty and has been shown to support more informed decision-making, especially in hydrological and water resource applications.
* The MME approach shows the greatest value in tropical regions, where model errors dominate over initial condition uncertainties, enabling effective bias compensation.
* Sectoral applications can benefit from filtering or weighting the most skillful models for a specific region or variable of interest, such as in national wheat yield forecasting or malaria risk.
* Bias adjustment and model recalibration are necessary post-processing steps that significantly enhance the reliability of multi-model forecasts, especially when outputs are used as input to impact models.
```

attachment:cd427a52-cbfe-49fb-9865-21858a87ac2e.png
---

---
The C3S multimodel June 2026 forecast of the of the NINO3.4 index highlighted the likelihood of a large El Nino event developing through the latter part of the year. 75% of members of the grand ensemble exceed 2.5°C amplitude in the Nino3.4 index at the end of the forecast period (November)

Plume charts of NINO predictions are available on the [C3S multi-model SST indices page](https://climate.copernicus.eu/charts/packages/c3s_seasonal/products/c3s_seasonal_plume_mm). Correlation heatmaps for the SST indices from individual models are available [here](https://confluence.ecmwf.int/display/CKB/C3S+seasonal+forecasts+verification+plots)
```


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Methodology
---
This notebook provides information on the use of seasonal forecast data available under different C3S catalogues, such as monthly statistics on [single levels](https://cds.climate.copernicus.eu/datasets/seasonal-monthly-single-levels?tab=overview) and and on [pressure levels](https://cds.climate.copernicus.eu/datasets/seasonal-monthly-pressure-levels?tab=overview), subdaily data on [single levels](https://cds.climate.copernicus.eu/datasets/seasonal-original-single-levels?tab=overview) and on [pressure levels](https://cds.climate.copernicus.eu/datasets/seasonal-original-pressure-levels?tab=overview), monthly anomalies on [single levels](https://cds.climate.copernicus.eu/datasets/seasonal-postprocessed-single-levels?tab=overview) and on [pressure levels](https://cds.climate.copernicus.eu/datasets/seasonal-postprocessed-pressure-levels?tab=overview).

The notebook focuses on the review of selected scientific literature on the state-of-the art knowledge and practice in the use of multi-model ensembles.

The discussion is organized in the following sections:

**[](multimodel:section-1)**

**[](multimodel:section-2)**
 * Health
 * Agriculture
 * Water resource management
 * Energy
 * Disaster preparedness
 * High-impact events

**[](multimodel:section-3)**
 * Bias adjustment
 * Combination

**[](multimodel:section-4)**

**[](multimodel:section-5)**


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Analysis and results > 1. Why the multi-model ensemble (MME)
---
There are two main sources of error which affect the performance of seasonal climate predictions in terms of reliability, accuracy and ultimately their usability and value for sectoral applications:
 * the initial condition uncertainties associated to the knowledge of the current state of the climate system, which is the starting point of a forecast [[4]](https://doi.org/10.1002/qj.49712252905);
 * the errors introduced by climate models, which describe the climate system by means of a finite number - albeit an incredibly large one - of interacting elements and by a set of empirical parametrizations and approximations of physical processes [[5]](https://doi.org/10.1002/qj.49712757202).

The first source of error is, to a large extent, unavoidable because it is associated with the limits of the global observing system and to the techniques adopted to extract information from observations. This limit is generally addressed by producing the largest possible ensembles of forecasts with similar initial conditions. The objective of producing large ensembles of forecasts is not necessarily, and only, the elimination or reduction of the forecast uncertainty. Instead, large ensembles aim at providing more robust statistics and produce the largest range of possible outcomes associated with uncertainty of the state of the climate system.

The second source of errors depends on the overall modelling approach, on the parameterizations adopted for some physical processes in the climate system, and on the specific solutions adopted for implementation of each climate modelling system. A fundamental goal of climate modelers is to reduce this kind of error as much as possible, in order to avoid the systematic misrepresentation of key features of the climate system (the so-called model bias), and the associated impact on essential climate variables such as precipitation intensity, temperature, or wind patterns.

The MME approach addresses the second source of error by leveraging the underlying assumption that seasonal predictions produced by different forecasting centers can be considered as the members of a single statistical ensemble. A further assumption is that the systematic errors produced by different climate models can partially compensate each other if the technical choices made for their development are sufficiently independent.

(multimodel:section-2)=


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Analysis and results > 2. Use cases and user needs
---
The MME approach to seasonal forecasting has already been tested in a number of applications to demonstrate the advantages over using a single modelling system.

Climate information is designed to meet specific user needs and a one-size-fits-all solution, rather than a systematic collection of existing applications, would be of limited use, if at all possible.

Therefore, a few examples of sectoral applications of the multi-model ensemble approach are provided with the purpose of illustrating real use cases and describe the potential added value of the MME approach.


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Analysis and results > 2. Use cases and user needs > Health
---
An early assessment of the added value of the multi-model approach has been conducted by focusing on malaria early warnings for southern Africa, using the models participating in the DEMETER project which still constitute the core of the C3S multi-model [[6]](https://doi.org/10.1038/nature04503) [[7]](https://doi.org/10.3402/tellusa.v57i3.14668). To generate the MME the difference in means and ratio of variances between each model and a reference dataset is computed and applied to the predictions of the target year.
In the DEMETER multi system, the fact that the sub-grid parameterizations in the component models have, to a large extent, been developed independently, is explicitly mentioned as a varied source of model uncertainty associated with the numerical approximation to the underlying partial differential equations that govern climate models.


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Analysis and results > 2. Use cases and user needs > Agriculture
---
At the global scale, two alternative approaches (average method and mosaic method) have been assessed for the forecasting of year-to-year variations in the global yield of key crop commodities such as maize, rice, wheat and soybean.  The ROC score analysis of within-season national yield predictions for maize, rice, wheat, and soybean based on mosaic method indicate skillful within-season predictions, especially for maize and wheat [[8]](https://doi.org/10.1016/j.cliser.2018.06.003).

The average method considers a simple average, with equal weights, over multiple forecasting systems for each location and cropping season. Within the mosaic method, the single best-performing forecasting system is selected for each location and cropping season, based on the corresponding grid-based skill score for yield variability. This study uses only one of the models available in C3S (NCEP) along with other SF available from other forecasting centres (APEC Climate Center, Korea; Meteorological Service of Canada; NASA; Pusan National University, Korea). For this application the mosaic method outperforms the average method and, by definition, it also outperforms the performance of individual forecasting systems. However, it may not be suitable for applications requiring large-scale consistency in forecasts, such as hydrological modeling (see below), where maintaining water mass conservation is essential for accurate streamflow predictions. In this case, the simple averaging of multi-model members has been adopted as a more relevant approach [[9]](https://doi.org/10.5194/egusphere-2023-569) .

On a more regional scale, a tailored multi-model ensemble has been developed to forecast national wheat yield in Argentina [[10]](https://doi.org/10.1088/1748-9326/ad627c).

attachment:08031f31-1537-434a-a742-1d056f529742.png
---
height: 400px
---
Example of mean absolute error between forecasted and reanalysis climate indicators for different forecasting systems and month of initialization. Column-wise for each month of initialization, the mean absolute error is expressed in units of standard deviation (σ), given that all features underwent standardization during preprocessing. Reproduced from [[Zachow et al. (2024]](https://doi.org/10.1088/1748-9326/ad627c) under [[CC BY 4.0]](https://creativecommons.org/licenses/by/4.0/).
```

This study considers all but one of the forecasting systems available at C3S and those available through the North-American-Multi-Model Ensemble (NMME). In this case, the multi-model ensemble is built by filtering out the group of three best-performing models for the region and for crop modelling application of interest. Interestingly, out of the initial set of 10 forecasting systems, the final best-performing subset of three systems is a mix of elements coming from both the C3S and of the NMME ensembles.


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Analysis and results > 2. Use cases and user needs > Water resource management
---
C3S has supported the development of an End-to-end Demonstrator for Improved Decision Making in the water sector in Europe (EdgE, [[11]](https://doi.org/10.1175/JHM-D-18-0040.1) ) based on the implementation of a multi-model prediction of streamflow at the European scale. In this case, four different climate modelling systems have been considered, in combination with four different approaches for the computation of the streamflow associated with the seasonal forecasts. This study emphasises how the multi-model approach allows for a better assessment of the uncertainty in the forecast and therefore a better framing of the value of the prediction for decision making.


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Analysis and results > 2. Use cases and user needs > Energy
---
The multi-model seasonal forecast approach has been tested for applications in the energy sector, both for energy production from renewable energy [[12]](https://doi.org/10.1007/s00382-019-04654-y) and to forecast energy demand [[13]](https://doi.org/10.1007/s00382-017-3766-y) at the national level.
It has been demonstrated that:
 * MME predictions indicate consistently higher performance than individual models in terms of different skill metrics such as temporal correlation coefficient (TCC) [[14]](https://doi.org/10.1002/2014WR015426) and fair ranked probability skill score (FRPSS) [[12]](https://doi.org/10.1007/s00382-019-04654-y);
 * the performance of a multi-model ensemble increases when using climate models which are as independent as possible from each other [[13]](https://doi.org/10.1007/s00382-017-3766-y).


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Analysis and results > 2. Use cases and user needs > Disaster preparedness
---
The use of multi-model seasonal forecasts has also emerged as a promising approach to enhance disaster preparedness and management efforts.

For example, advanced flood preparedness in Perù has leveraged multi-model seasonal climate predictions [[15]](https://doi.org/10.5194/nhess-21-2215-2021) by adopting a mix of best-performing climate model selection and model combination, starting from the North-America Multi-Model Ensemble NMME. The correlation between spring (FMA) precipitation and streamflow (the key impact indicator for this use case) is around 0.76 for individual models whereas it increases to 0.84 when averaging the outcome of the best-performing models from NMME.

A multi-model ensemble composed of 7 out of the 13 systems included in the NMME has been calibrated to enhance climate services in Ethiopia [[16]](https://doi.org/10.1016/j.cliser.2021.100272), whose National Meteorological Agency is particularly focused on issuing drought early warnings and supporting disaster preparedness and management. In this case, the multi-model ensemble is built by first correcting the bias in each model (i.e. calibration) and then creating a super-ensemble where each model is assigned the same weight independently of the predictive skill.

attachment:4c2f6447-b2e3-47d7-9e36-7492a59958f4.png
---
height: 400px
---
JJAS (Kiremt rainy season) precipitation anomalies in Ethiopia for 2020 (a) forecasted (b) observed. Unit of precipitation anomaly is mm/season. Reproduced from [Acharya et al. (2021)](https://doi.org/10.1016/j.cliser.2021.100272) under [CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/).
```

The analysis of the added value of the multi-model prediction compared to individual models is not explicitly reported. However the study emphasises the main source of predictability in the region is ENSO, whose prediction is known to benefit significantly from the multi-model approach [[17]](https://doi.org/10.1175/2008MWR2431.1).


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Analysis and results > 2. Use cases and user needs > High-impact events
---
The prediction of tropical storm frequency also benefits from the adoption of the multi-model approach as demonstrated using seven models participating in the DEMETER project [[18]](https://doi.org/10.1256/qj.05.65). By adding all ensemble forecasts after calibration, this analysis demonstrates that over specific regions, combining several models leads to better forecasts than the best individual model.

The analysis conducted during the ENSEMBLES project has demonstrated that the skill in forecasting different indices for temperature and rainfall extremes improves with a multi-model approach, compared to any individual model [[19]](https://doi.org/10.1016/j.wace.2015.06.005). This study makes limited, if any, reference to the added value for specific sectoral applications.

Also, the most significant improvements are mainly detected over the tropical ocean in the ENSO region. However, the study provides a key support for the development of applications that rely on the extracting information from the teleconnection between ENSO and local climate variables [[17]](https://doi.org/10.1175/2008MWR2431.1).

(multimodel:section-3)=


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Analysis and results > 3. Technical aspects in using the MME approach
---
The MME approach is designed to address multiple sources of climate model uncertainty, including not only differences in the numerical approximations to the governing physical equations, but also variations arising from ensemble size, sampling of initial condition uncertainty, data assimilation techniques, and other aspects of the modeling systems. By combining outputs from multiple models, the MME approach aims to provide more robust and reliable projections than single models alone.

Two main groups of post-processing approaches are adopted to implement the MME approach:

* the bias adjustment and recalibration methods applied to the output of individual forecasting systems [[20]](https://doi.org/10.1007/s00382-019-04640-4);
 * the combination of predictions issued with different forecasting systems [[21]](https://doi.org/10.1126/science.285.5433.1548) [[22]](https://doi.org/10.3402/tellusa.v57i3.14657).


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Analysis and results > 3. Technical aspects in using the MME approach > Bias adjustment
---
Bias adjustment and recalibration methods have been tested systematically on the datasets available on the Copernicus Data Store [[21]](https://doi.org/10.1126/science.285.5433.1548).

On the other hand, the multi-model approach based on combining predictions from different systems assumes that climate models are sufficiently independent to improve or at least partially compensate for the respective errors.

It has been demonstrated, using both toy-models and actual climate model simulations, that multi-model ensembles can outperform a ‘best-model’ approach because multi-model combinations reduce the average ensemble mean error at the cost of widening the spread of the overall ensemble [[23]](https://doi.org/10.1002/qj.210).

A systematic, effective  approach for the creation of multimodel ensemble, tested on C3S includes, recalibration and the equally weighting of all members of the multi-model ensemble [[24]](https://doi.org/10.1007/s00382-020-05314-2).


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Analysis and results > 3. Technical aspects in using the MME approach > Combination
---
In general, the multi-model approach is expected to bring much of its added value in the tropics, where the combination of models in a single grand-ensemble can offset errors affecting the predictable components of the climate system [[25]](https://doi.org/10.3402/tellusa.v57i3.14658).

In extratropical regions, e.g. over Europe, the multi-model does not always perform better than the best single models, therefore it is advisable to test the single models for each application [[26]](https://doi.org/10.1007/s00382-018-4404-z). The figure below shows an example of how different models can be more skillful over different areas.

Seasonal forecasting studies employ different approaches to generate multi‐model ensembles. As examples:

- simple equal‐weight pooling appears in four studies [[25]](https://doi.org/10.3402/tellusa.v57i3.14658) [[24]](https://doi.org/10.1007/s00382-020-05314-2);

- regression‐based optimal weighting [[26]](https://doi.org/10.1007/s00382-018-4404-z) [[25]](https://doi.org/10.3402/tellusa.v57i3.14658);

- bayesian methods [[27]](https://doi.org/10.1175/MWR2818.1);

- performance-based objective weighting approaches [[28]](https://doi.org/10.1175/BAMS-84-12-1783).


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Analysis and results > 3. Technical aspects in using the MME approach > Complementarity of bias adjustment and combination
---
While bias correction and model combination are sometimes discussed separately, they are in fact complementary components of the multi-model ensemble framework rather than competing alternatives. Bias correction improves the performance of individual models, ensuring that each contributes more reliable information to the ensemble. Model combination, in turn, integrates the corrected forecasts from multiple systems to capture a broader range of plausible outcomes and reduce the influence of any single model’s systematic deficiencies.

This complementarity between bias correction and model combination is also supported by comparison of the predictive skill of individual seasonal forecasting systems, which varies substantially across regions, variables, and seasons, with different models achieving the highest anomaly correlation in different areas [[26]](https://doi.org/10.1007/s00382-018-4404-z). Such results highlight that no single forecasting system consistently outperforms all others. Consequently, combining forecasts from multiple models provides an opportunity to exploit the strengths of each system while mitigating their individual weaknesses. Within this framework, bias correction enhances the reliability of the forecasts produced by each model, whereas model combination synthesizes the corrected predictions to generate a more robust and skillful ensemble forecast. Together, these approaches may contribute to improving both forecast calibration and predictive performance.


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Analysis and results > 3. Technical aspects in using the MME approach > The Signal to Noise Paradox
---
The systematic analysis of climate forecasts has revealed the existence of the so-called Signal-to-Noise Paradox (SNP). This paradox refers to the unexpected situation in which seasonal forecast models can predict the observed climate anomalies more accurately than they can predict their own internal variability.

For example, in winter NAO forecasts, the ensemble mean can anticipate whether the season will be dominated by a positive or negative NAO phase (i.e. stronger or weaker westerly winds over the North Atlantic) even though the individual ensemble members within the same model vary so much from each other that, taken separately, they appear to have very little predictive power. The mismatch between good prediction of reality despite weak internal consistency defines the Signal-to-Noise Paradox. [[29]](https://doi.org/10.1038/s41612-018-0038-4)

This is paradoxical because, in principle, if the model is realistic, the predictability within the model’s ensemble should be similar to that of the real world. The existance of a Signal-to-Noise Paradox suggests that the models are underestimating amplitude of their own predictable signal or overestimating internal noise, pointing to systematic deficiencies in the forecasting systems, which are still unclear. [[30]](https://doi.org/10.1175/BAMS-D-24-0019.1).

Nonetheless, it has been shown that, in a multi-model ensemble, the signal-to-noise ratio (SNR) can offer valuable insights into forecast reliability. In particular, years characterised by a high SNR tend to exhibit, on average, larger observed deviations from the mean than years with a low SNR, for both near-surface temperature (T2m) and precipitation. This suggests that forecast systems might be more reliable in predicting large anomalies (e.g., extremes) when there is greater coherence among ensemble members [[31]](https://doi.org/10.5194/wcd-4-823-2023).

(multimodel:section-4)=


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Analysis and results > 4. The MME approach at work for the C3S multi-model
---
On a monthly basis, C3S issues [seasonal forecast charts](https://climate.copernicus.eu/seasonal-forecasts) based on the multi-model data available [here](https://cds.climate.copernicus.eu/datasets/seasonal-original-single-levels?tab=overview).

In this case, the seasonal outlook is not designed for specific applications. The multi-model ensemble (MME) forecast is calculated as a weighted average of the ensemble means from nine component models: ECMWF, Met Office, Météo-France, CMCC, DWD, NCEP, JMA, BOM and ECCC.

The weighting ensures that each component contributes equally to the overall variance of the multi-model during the common hindcast period.
For each component model, ensemble mean anomalies are calculated relative to its own model climate.

When computing the multi-model mean, the weight assigned to each component is determined by dividing the square root of the average variance across all systems by the square root of the variance of the respective component.

A slightly different approach is adopted for the [North American Multi-Model Ensemble (NMME)](https://www.cpc.ncep.noaa.gov/products/NMME/). In this case, to calculate anomalies the forecast bias is removed and is calculated separately for each model using all ensemble members for that particular model. The grand ensemble
mean, and other diagnostics such as tercile probabilities, are defined as by assuming
assuming that each ensemble member of each model is equally probable .

attachment:ad6f1cd6-07a6-445f-8542-6287f198b380.jpg
---
height: 600px
---
Comparison of the July-August-Septmeber 2025 probabilities of above/below/neutral sea surface temperature anomalies  forecasted in June 2025, according to the C3S MME (left) and to the NMME (right) methodologies. Note the differences in the respective probabilities as well as the similarities in terms of the overall patterns of global anomalies.
```

(multimodel:section-5)=


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > Analysis and results > 5. Challenges and Limitations
---
The use of a multi-model approach is case specific and there is no evidence of a single standard approach to be adopted as an all-purpose solution.

For example, bias adjustment is essential when preparing the (usually daily based) input for downstream impact models, such as crop models or energy production models. In such circumstances, the added value of a multi-model approach may be off-set by the computational demand of the processing chain. On the other hand, recalibration approaches improve the overall reliability of seasonal indicators by building on the temporal correspondence between the ensemble mean predictions and the corresponding observations [[21]](https://doi.org/10.1126/science.285.5433.1548).


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > ℹ If you want to know more > Key resources
---
Explore the [C3S Seasonal Forecast Products](https://climate.copernicus.eu/seasonal-forecasts)

Explore the [North American Multi-Model Ensemble products](https://nmme.earth.miami.edu/forecasts/nmme_prob_forecast_all_var.html)

[PyCPT](https://iri-pycpt.github.io/PyCPT2-Seasonal-Forecast-User-Guide/intro.html) - a tool calibrate and verify multi-model seasonal forecasts of precipitation based on the NOAA North American Multi-Model Ensemble (NMME) and European Copernicus Climate Change Service (C3S) databases.


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > ℹ If you want to know more > References
---
[[1]](https://doi.org/10.1029/2009GL040896) Hagedorn, R., Doblas-Reyes, F. J., & Palmer, T. N. (2006). DEMETER and the application of seasonal forecasts. Predictability of weather and climate, 674-692.

[[2]](https://doi.org/10.1029/2007GL030740) Vitart, F., Huddleston, M. R., Déqué, M., Peake, D., Palmer, T. N., Stockdale, T. N., ... & Weisheimer, A. (2007). Dynamically‐based seasonal forecasts of Atlantic tropical storm activity issued in June by EUROSIP. Geophysical Research Letters, 34(16).

[[3]](https://doi.org/10.1007/s00382-011-1061-x) Rajeevan, M., Unnikrishnan, C.K. & Preethi, B. Evaluation of the ENSEMBLES multi-model seasonal forecasts of Indian summer monsoon variability. Clim Dyn 38, 2257–2274 (2012).

[[4]](https://doi.org/10.1002/qj.49712252905) Molteni, F., Buizza, R., Palmer, T. N., & Petroliagis, T. (1996). The ECMWF ensemble prediction system: Methodology and validation. Quarterly journal of the royal meteorological society, 122(529), 73-119.

[[5]](https://doi.org/10.1002/qj.49712757202) Palmer, T. N. (2001). A nonlinear dynamical perspective on model error: A proposal for non‐local stochastic‐dynamic parametrization in weather and climate prediction models. Quarterly Journal of the Royal Meteorological Society, 127(572), 279-304.

[[6]](https://doi.org/10.1038/nature04503) Thomson, M., Doblas-Reyes, F., Mason, S. et al. Malaria early warnings based on seasonal climate forecasts from multi-model ensembles. Nature 439, 576–579

[[7]](https://doi.org/10.3402/tellusa.v57i3.14668) Morse, A. P., Doblas-Reyes, F. J., Hoshen, M. B., Hagedorn, R., & Palmer, T. N. (2005). A forecast quality assessment of an end-to-end probabilistic multi-model seasonal forecast system using a malaria model. Tellus A: Dynamic Meteorology and Oceanography, 57(3), 464–475. https://doi.org/10.3402/tellusa.v57i3.14668

[[8]](https://doi.org/10.1016/j.cliser.2018.06.003) Iizumi, T., Shin, Y., Kim, W., Kim, M., & Choi, J. (2018). Global crop yield forecasting using seasonal climate information from a multi-model ensemble. Climate Services, 11, 13-23.

[[9]](https://doi.org/10.5194/egusphere-2023-569) Thébault, C., Perrin, C., Andréassian, V., Thirel, G., Legrand, S., & Delaigue, O. (2023). Multi-model approach in a variable spatial framework for streamflow simulation. EGUsphere, 2023, 1-34.

[[10]](https://doi.org/10.1088/1748-9326/ad627c) , Maximilian Zachow, Harald Kunstmann, Daniel Julio Miralles and Senthold Asseng (2024) Multi-model ensembles for regional and national wheat yield forecasts in Argentina  Environ. Res. Lett. 19 084037

[[11]](https://doi.org/10.1175/JHM-D-18-0040.1)Wanders, N., S. Thober, R. Kumar, M. Pan, J. Sheffield, L. Samaniego, and E. F. Wood, 2019: Development and Evaluation of a Pan-European Multimodel Seasonal Hydrological Forecasting System. J. Hydrometeor., 20, 99–115, https://doi.org/10.1175/JHM-D-18-0040.1 .


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > ℹ If you want to know more > References
---
-European Multimodel Seasonal Hydrological Forecasting System. J. Hydrometeor., 20, 99–115, https://doi.org/10.1175/JHM-D-18-0040.1 .

[[12]](https://doi.org/10.1007/s00382-019-04654-y) Lee, D. Y., Doblas-Reyes, F. J., Torralba, V., & Gonzalez-Reviriego, N. (2019). Multi-model seasonal forecasts for the wind energy sector. Climate Dynamics, 53, 2715-2729.

[[13]](https://doi.org/10.1007/s00382-017-3766-y) Alessandri, A., Felice, M.D., Catalano, F. et al. Grand European and Asian-Pacific multi-model seasonal forecasts: maximization of skill and of potential economical value to end-users . Clim Dyn 50, 2719–2738 (2018).

[[14]](https://doi.org/10.1002/2014WR015426) Mendoza, P. A., Rajagopalan, B., Clark, M. P., Cortés, G., & McPhee, J. (2014). A robust multimodel framework for ensemble seasonal hydroclimatic forecasts. Water Resources Research, 50(7), 6030-6052.

[[15]](https://doi.org/10.5194/nhess-21-2215-2021) Keating, C., Lee, D., Bazo, J., and Block, P.: Leveraging multi-model season-ahead streamflow forecasts to trigger advanced flood preparedness in Peru, Nat. Hazards Earth Syst. Sci., 21, 2215–2231,

[[16]](https://doi.org/10.1016/j.cliser.2021.100272) N. Acharya, M.A. Ehsan, A. Admasu, A. Teshome, K.J.C. Hall (2021) On the next generation (NextGen) seasonal prediction system to enhance climate services over Ethiopia Clim. Serv., 24 (2021), 10.1016/j.cliser.2021.100272

[[17]](https://doi.org/10.1175/2008MWR2431.1) Tippett, M. K., & Barnston, A. G. (2008). Skill of multimodel ENSO probability forecasts. Monthly Weather Review, 136(10), 3933-3946.

[[18]](https://doi.org/10.1256/qj.05.65) Vitart, F. (2006), Seasonal forecasting of tropical storm frequency using a multi-model ensemble. Q.J.R. Meteorol. Soc., 132: 647-666. https://doi.org/10.1256/qj.05.65

[[19]](https://doi.org/10.1016/j.wace.2015.06.005) Acacia S. Pepler, Leandro B. Díaz, Chloé Prodhomme, Francisco J. Doblas-Reyes, Arun Kumar, The ability of a multi-model seasonal forecasting ensemble to forecast the frequency of warm, cold and wet extremes (2015) , Weather and Climate Extremes, https://doi.org/10.1016/j.wace.2015.06.005 .

[[20]](https://doi.org/10.1007/s00382-019-04640-4) Manzanas, R., Gutiérrez, J.M., Bhend, J. et al. Bias adjustment and ensemble recalibration methods for seasonal forecasting: a comprehensive intercomparison using the C3S dataset. Clim Dyn 53, 1287–1305 (2019). https://doi.org/10.1007/s00382-019-04640-4

[[21]](https://doi.org/10.1126/science.285.5433.1548) Krishnamurti, T. N., Kishtawal, C. M., LaRow, T. E., Bachiochi, D. R., Zhang, Z., Williford, C. E., ... & Surendran, S. (1999). Improved weather and seasonal climate forecasts from multimodel superensemble. Science, 285(5433), 1548-1550.

[[22]](https://doi.org/10.3402/tellusa.v57i3.14657) Hagedorn, R., Doblas-Reyes, F. J., & Palmer, T. N. (2005). The rationale behind the success of multi-model ensembles in seasonal forecasting — I. Basic concept. Tellus A: Dynamic Meteorology and Oceanography, 57(3), 219–233. https://doi.org/10.3402/tellusa.v57i3.14657


EQC Quality Assessment: "Benefit and challenges of a multi-model approach to seasonal forecasts"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: validation_q05  |  Category: Seasonal_Forecasts
Section: Benefit and challenges of a multi-model approach to seasonal forecasts > ℹ If you want to know more > References
---
sembles in seasonal forecasting — I. Basic concept. Tellus A: Dynamic Meteorology and Oceanography, 57(3), 219–233. https://doi.org/10.3402/tellusa.v57i3.14657

[[23]](https://doi.org/10.1002/qj.210) Weigel, A. P., Liniger, M. A., & Appenzeller, C. (2008). Can multi‐model combination really enhance the prediction skill of probabilistic ensemble forecasts?. Quarterly Journal of the Royal Meteorological Society: A journal of the atmospheric sciences, applied meteorology and physical oceanography, 134(630), 241-260.

[[24]](https://doi.org/10.1007/s00382-020-05314-2) Hemri, S., Bhend, J., Liniger, M.A. et al. How to create an operational multi-model of seasonal forecasts?. Clim Dyn 55, 1141–1157 (2020). https://doi.org/10.1007/s00382-020-05314-2

[[25]](https://doi.org/10.3402/tellusa.v57i3.14658) Doblas-Reyes, F. J., Hagedorn, R., & Palmer, T. N. (2005). The rationale behind the success of multi-model ensembles in seasonal forecasting – II. Calibration and combination. Tellus A: Dynamic Meteorology and Oceanography, 57(3), 234–252.

[[26]](https://doi.org/10.1007/s00382-018-4404-z)  Mishra N, Prodhomme C, Guemas V (2018) Multi-model skill assessment of seasonal temperature and precipitation forecasts over Europe. Clim Dyn.

[[27]](https://doi.org/10.1175/MWR2818.1) Robertson, A. W., Lall, U., Zebiak, S. E., & Goddard, L. (2004). Improved combination of multiple atmospheric GCM ensembles for seasonal prediction. Monthly Weather Review, 132(12), 2732-2744.

[[28]](https://doi.org/10.1175/BAMS-84-12-1783) Barnston, A. G., Mason, S. J., Goddard, L., DeWitt, D. G., & Zebiak, S. E. (2003). Multimodel ensembling in seasonal climate forecasting at IRI. Bulletin of the American Meteorological Society, 84(12), 1783-1796.

[[29]](https://doi.org/10.1038/s41612-018-0038-4) Scaife, A. A., & Smith, D. (2018). A signal-to-noise paradox in climate science. npj Climate and Atmospheric Science, 1(1), 28.

[[30]](https://doi.org/10.1175/BAMS-D-24-0019.1) Weisheimer, A., Baker, L. H., Bröcker, J., Garfinkel, C. I., Hardiman, S. C., Hodson, D. L., ... & Sutton, R. T. (2024). The signal-to-noise paradox in climate forecasts: revisiting our understanding and identifying future priorities. Bulletin of the American Meteorological Society, 105(3), E651-E659.

[[31]](https://doi.org/10.5194/wcd-4-823-2023) Acosta Navarro, J. C., & Toreti, A. (2023). Exploiting the signal-to-noise ratio in multi-system predictions of boreal summer precipitation and temperature. Weather and Climate Dynamics, 4(3), 823-831.