EQC Quality Assessment: "Satellite Sea Surface Temperature Consistency in representing Ocean Warming Trends"
Dataset: satellite-sea-surface-temperature [CDS]
Aspect: consistency_q02  |  Category: Satellite_ECVs
Section: Satellite Sea Surface Temperature Consistency in representing Ocean Warming Trends > ℹ If you want to know more > References
---
[[1]](https://doi.org/10.1007/s00382-014-2147-z) Kang, S.M., Seager, R., Frierson, D.M.W. et al. Croll revisited: Why is the northern hemisphere warmer than the southern hemisphere?. Clim Dyn 44, 1457–1472 (2015). https://doi.org/10.1007/s00382-014-2147-z

[[2]](https://doi.org/10.1175/JCLI-D-20-0793.1) Yang, C., Leonelli, F. E., Marullo, S., Artale, V., Beggs, H., Nardelli, B. B., ... & Pisano, A. (2021). Sea surface temperature intercomparison in the framework of the Copernicus Climate Change Service (C3S). Journal of Climate, 34(13), 5257-5283

[[3]](https://doi.org/10.1016/j.rse.2018.12.015) Fiedler, E. K., McLaren, A., Banzon, V., Brasnett, B., Ishizaki, S., Kennedy, J., ... & Donlon, C. (2019). Intercomparison of long-term sea surface temperature analyses using the GHRSST Multi-Product Ensemble (GMPE) system. Remote sensing of environment, 222, 18-33

[[4]](https://doi.org/10.1038/s41597-019-0236-x) Merchant, C. J., Embury, O., Bulgin, C. E., Block, T., Corlett, G. K., Fiedler, E., ... & Donlon, C. (2019). Satellite-based time-series of sea-surface temperature since 1981 for climate applications. Scientific data, 6(1), 223. https://doi.org/10.1038/s41597-019-0236-x

[[5]](https://doi.org/10.3390/rs12010132) Pisano, A., Marullo, S., Artale, V., Falcini, F., Yang, C., Leonelli, F. E., Santoleri, R. and Buongiorno Nardelli, B.: New Evidence of Mediterranean Climate Change and Variability from Sea Surface Temperature Observations, Remote Sens., 12(1), 132, doi:10.3390/rs12010132, 2020

[[6]](https://doi.org/10.1080/1755876X.2018.1489208) Mulet, S., Buongiorno Nardelli, B., Good, S., Pisano, A., Greiner, E., Monier, M., Autret, E., Axell, L., Boberg, F., Ciliberti, S., Drévillon, M., Droghei, R., Embury, O., Gourrion, J., Høyer, J., Juza, M., Kennedy, J., Lemieux-Dudon, B., Peneva, E., Reid, R., Simoncelli, S., Storto, A., Tinker, J., Von Schuckmann, K., Wakelin, S. L., 2018. Ocean temperature and salinity. In: Copernicus Marine Service Ocean State Report, Issue 2, Journal of Operational Oceanography, 11:sup1, s5–s13, DOI: 10.1080/1755876X.2018.1489208

[[7]](http://hdl.handle.net/10013/epic.45156.d001) Pachauri, R. K., Allen, M. R., Barros, V. R., Broome, J., Cramer, W., Christ, R., Church, J. A., Clarke, L., Dahe, Q., Dasgupta, P., Dubash, N. K., et al.: Climate Change 2014: Synthesis Report. Contribution of Working Groups I, II and III to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change, edited by: Pachauri, R. and Meyer, L., Geneva, Switzerland, IPCC, ISBN 978-92-9169-143-2, 2014


EQC Quality Assessment: "Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes"
Dataset: seasonal-monthly-ocean [CDS]
Aspect: extremes-detection_q07  |  Category: Seasonal_Forecasts
Section: Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes > Quality assessment question
---
* **What is the forecast skill of the seasonal prediction system for detecting extreme upper ocean heat content events in the NDJ season, from October initializations?**

This notebook evaluates the skill of the Météo-France System-9 seasonal forecasting system in predicting upper ocean heat content extremes. We chose this system as an example since it had high data coverage of 300-m integrated potential temperature. The analysis uses:
- **Seasonal Forecast Data**: Monthly depth-averaged potential temperature for the upper 300 m, from Météo-France System-9 forecasts. [[doi:10.24381/cds.2f9be611]](https://doi.org/10.24381/cds.2f9be611)
- **Reanalysis Data**: Ocean heat content from the ORAS5 global ocean and sea-ice reanalysis dataset. [[doi:10.24381/cds.67e8eeb7]](https://doi.org/10.24381/cds.67e8eeb7)

We consider the skill in predicting upper ocean heat content extremes in November, December, and January between 1993 and 2023 for (re-)forecasts with a nominal start date October 1st, using metrics such as the Symmetric Extremal Dependence Index (SEDI) and the Brier Skill Score (BSS). The metrics considered follow  Jacox et. al. 2022, [[1]](https://doi.org/10.1038/s41586-022-04573-9), however in this work we look at upper ocean heat content and not sea surface temperature. Since ocean heat content (OHC) anomalies may persist for several months, this variable is an important component of seasonal predictability, containing relevant information even at a monthly time-scale (McAdam et. al. 2022, [[2]](https://doi.org/10.1007/S00382-021-06101-3)).


EQC Quality Assessment: "Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes"
Dataset: seasonal-monthly-ocean [CDS]
Aspect: extremes-detection_q07  |  Category: Seasonal_Forecasts
Section: Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes > Quality assessment statement
---
These are the key outcomes of this assessment

- The Météo-France System-9 forecast model demonstrates skill in detecting the risk of NDJ (November-January) upper ocean heatwaves. The model successfully captures 33% of observed heatwaves in November, with hit rates remaining consistent through December (30%) and January (27%).
- The forecast is useful for identifying broad, large-scale patterns of risk, particularly during major El Niño events. However, the Brier Skill Score (BSS) highlights that its probabilistic reliability varies by region. Using a multi-model forecast ensemble would likely increase the spread of the forecasts and provide more reliable results outside the ENSO region.
- The system has potential as a tool for anticipating and building resilience to marine heatwaves, especially in ENSO-sensitive regions. This analysis could be extended to other C3S seasonal systems for which the required data are available.

attachment:0fa94a15-3175-4725-863a-fc42e5bcb6c5.png
---
height: 600px
---
Figure 1: Spatial Forecast Skill for November-December-January (NDJ) Upper Ocean Heatwaves (1993–2023).
Skill of Météo-France System-9 forecasts (with nominal start date October 1st) verified against ORAS5 reanalysis. Heatwaves are defined as upper 300m ocean heat content (OHC) anomalies exceeding the 90th percentile within the 1993-2023 period. Rows show: (A) Symmetric Extremal Dependence Index (SEDI), (B) Brier Skill Score (BSS); and (C) Accuracy. In all maps, red areas indicate positive skill (forecast is better than the reference), while blue areas indicate negative skill. Masked areas are shown in grey. For the Accuracy maps (C) the areas with an accuracy lower than what would be expected by random chance (82%) are also shown with a grey color.

```

## 📋 Methodology

The following steps were applied to evaluate the skill of the seasonal forecasting system in predicting heatwave extremes:

[](seasonal_ohc:section-1)

This section includes:
- imports, settings, parameter definitions, and reusable functions required to initialize the analysis. Here the extent of the Nino3.4 area is given (5°N–5°S, 170°–120°W, also delineated in most map plots, e.g. [](fig-corr-map)). This area is selected to represent the tropical Pacific and ENSO, providing a key benchmark for seasonal predictability. The data is also loaded in this section.

- **Météo-France System-9**: The seasonal forecast data includes monthly depth-averaged potential temperature for the upper 300m. We use the hindcasts with a nominal start date of October 1st, between 1993 and 2023. For consistency only the hindcast period is used, since the model system has a different number of ensemble members in the forecast period compared to the hindcast period. *depth_average_potential_temperature_of_upper_300m* ($\bar{\Theta}$) was converted to OHC using the similar constants as used by ORAS5 (pers. comm.):
$\text{OHC} =\rho \, c_p \, \bar{ \Theta} \, dz $, where  $ \rho = 1026 \, \text{kg/m}^3$, $ c_p = 3991 \, \text{J/kg/K} $, and  $ dz = 300 \, \text{m} $.
- **ORAS5**: Reanalysis OHC for the upper 300m is used as the observational reference. Data is regridded to a common 1° spatial resolution using conservative normalization in xESMF.

- The seasonal forecast and reanalysis data were processed for the period from **1993 to 2023**; and we consider forecasts **with a nominal start date October 1st, valid for November, December, and January**. For instance, the last season considers forecasts with a nominal start date October 1st 2023 for November and December 2023, and January 2024. By choosing an October initialization, we avoid compounding the task of extreme event detection with the lower predictability associated with other initialization windows, such as the boreal spring period in the Eastern Tropical Pacific (Jacox et. al. 2022, [[1]](https://doi.org/10.1038/s41586-022-04573-9)).

- **Anomalies**: Monthly anomalies are calculated by subtracting the calendar month climatology.


EQC Quality Assessment: "Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes"
Dataset: seasonal-monthly-ocean [CDS]
Aspect: extremes-detection_q07  |  Category: Seasonal_Forecasts
Section: Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes > Quality assessment statement
---
. 2022, [[1]](https://doi.org/10.1038/s41586-022-04573-9)).

- **Anomalies**: Monthly anomalies are calculated by subtracting the calendar month climatology.

- We also apply a common land mask to the interpolated products, also omitting areas where at any time the seasonal forecasts has temperatures below 0°C (we could have used a lower threshold than 0°C, but use this threshold as it at least will mask out all ice covered areas).

- The seasonal forecast system (System 9) is initialized by nudging the ocean and sea-ice models towards the Mercator Ocean International GLORYS12V1 reanalysis, except for wihtin 225 km of coastlines or in the presence of sea ice (Specq et. al. 2024, [[3]](https://doi.org/10.5281/zenodo.15518106)). In contrast, ORAS5 is used as the reference reanalysis in this study. This makes the reference data somewhat independent, however, the data assimilated in ORAS5 and GLORYS12 likely overlap to a great extent. Differences in means will not affect the conclusions as anomalies are considered. Further, the limits for a heatwave is calculated separately for the seasonal forecast data and the reference data, allowing them to have different distributions.

[](seasonal_ohc:section-2)

- We compare the data in map plots and in area aggregated time-series plots. Initial inspection showed strong temporal trends in OHC during the time period. The linear trends were removed from the raw data, so that the trends would not inflate the skill scores. Further, the strong temporal trends would skew the diagnosis of heatwaves to primarily occur in the more recent years.

[](seasonal_ohc:section-3)

The OHC anomalies based on detrended data were used to assess heatwaves. We show that the reference data and seasonal forecasting data have different data distributions. To align our diagnosed heatwaves, the defined percentiles of a heatwave are calculated separately depending on both the data source and the calendar month. To avoid unrealistic jumps, and to align our analysis with Jacox et. al. 2022, [[1]](https://doi.org/10.1038/s41586-022-04573-9) we use a three-month centered rolling window to set the monthly heatwave thresholds. For instance, January thresholds are derived from pooled December, January, and February anomalies. Heatwaves are defined as anomalies exceeding the 90th percentile.

In the subsequent sections we evaluate the forecast over different dimensions:

### Pooling and grouping strategies

| Analysis Goal	| Grouping Strategy	| Question It Answers |
|---------------|-------------------|---------------------|
 | Skill Maps | Pool time and ensemble for each grid point.	| "What is the forecast skill at this specific location over the entire 30-year period?"|
| Annual Time Series |	Pooled horizontally and over the ensemble for each year. |	"What was the overall global forecast accuracy in the year 1997?"|
| Summary Tables |	Pool time, space, and ensemble for each lead-time (month). |	"What was the overall global forecast skill for all Novembers combined in the record?" |

[](seasonal_ohc:section-4)

We evaluate the forecasting system using the metrics outlined below.

### SEDI -  the Symmetric Extremal Dependence Index

SEDI quantifies the forecast skill for rare binary events (e.g., heatwaves). It is the Symmetric Extremal Dependence Index (Ferro & Stephenson 2011 [[4]](https://doi.org/10.1175/WAF-D-10-05030.1)). SEDI has a maximum value of one, and a minimum value of -1. Scores above zero indicate a forecast better than random chance.

SEDI is built on the output of a contingency table. A contingency table provides:  sample size, n; base rate, p; hit rate, H; and false-alarm rate, F. The ensemble members are pooled and each adds to the counts in the contingency table before the SEDI score is calculated.

|                     | Event observed | Non-event observed |       |
 |     ---             | ---            | ---                | ---   |
 |Event forecasted     | a = Hpn        | b = F(1-p)n        | a + b |
 |Non-event forecasted | c = (1-H)pn    | d = (1-F)(1-p)n    | c + d |
 |Sum                  | a + c = pn     | b + d = (1-p)n     | n     |

Hit rate (H) =  $ \frac{a}{a+c} $, False alarm rate (F) =  $ \frac{b}{b+d} $


EQC Quality Assessment: "Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes"
Dataset: seasonal-monthly-ocean [CDS]
Aspect: extremes-detection_q07  |  Category: Seasonal_Forecasts
Section: Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes > Quality assessment statement
---
+ d = (1-p)n     | n     |

Hit rate (H) =  $ \frac{a}{a+c} $, False alarm rate (F) =  $ \frac{b}{b+d} $

SEDI =  $ \frac{logF - logH  -log(1-F) + log(1-H)}{logF + logH  +log(1-F) + log(1-H)} $

### The Brier Skill Score (BSS)

The Brier Skill Score (BSS) compares the forecast skill against a reference forecast (e.g., climatology), which for heatwaves defined as values above 90% of those observed, would be that there always was about a 10% chance of a heatwave.

It is calculated as:
  $
  \text{BSS} = 1 - \frac{\text{Brier Score (BrS)}}{\text{BrS}_{\text{ref}}}
  $

It is based on the Brier Score, which measures the mean squared error between forecast probabilities and observed binary events.

The BSS's upper limit is 1, meanwhile it has no lower limit. Scores above zero indicate a forecast better than the climatology, which is around a 10% chance.

### Accuracy

Accuracy measures the proportion of correct predictions (both true positives and true negatives) out of the total number of forecasts made. The formula is:

$ \text{Accuracy} = \frac{\text{True Positives (TP)} + \text{True Negatives (TN)}}{\text{Total Predictions (N)}} $

For rare events like heatwaves (which occur about 10 % of the time in this study), a simple accuracy score can be misleading. A forecast that always predicts "no heatwave" would be 90 % accurate but have zero skill. Therefore, we compare the forecast's accuracy to a random chance benchmark. For an event with a 10% probability (p=0.1), the accuracy of a random forecast is calculated as:

$ \text{Random Accuracy}  =  p^2+(1−p)^2 = 0.1^2+(0.9)^2 = 0.82 $

This means the forecasting system must achieve an accuracy greater than 0.82 (or 82%) to be considered more skillful than random chance.

[](seasonal_ohc:section-5)
The contingency tables for all November, December, and January's are presented and discussed.

[](seasonal_ohc:section-6)
Finally, we display the annual metrics as time-series, also showing the components of the contingency table, allowing a discussion of the components of the metrics.

[](seasonal_ohc:section-7)

## 📈 Analysis and results
(seasonal_ohc:section-1)=
### 1. Data Preparation
This section includes imports, settings, parameter definitions, and reusable functions required to initialize the analysis.

#### Imports and Settings

### Imports and Settings

Select realizations for ensemble
Whether to detrend anomalies or not
Analysis time period
Use all realizations for ensemble
Various parameters
Physical constants for ocean heat content calculations as used in ORAS5
Conversion factor for heat content (J/m^2)
delta_level * cp * rho0 * (Seasonal_data - Tf)

#### Definition of Parameters

We have chosen Meteo-France System-9, and focus on the hindcast period, which are available from 1993 until 2024, to have a consistent ensemble throughout the analysis.

We focus on the forecasts initiated October 1st every year, and consider one, two, and three months leadtimes - i.e. the forecasted values for November, December, and January. In the initial processing also October and February values are used, as the heatwave thresholds are centered a three-month month moving window.

#### Helper Functions for Data Processing and Plotting


EQC Quality Assessment: "Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes"
Dataset: seasonal-monthly-ocean [CDS]
Aspect: extremes-detection_q07  |  Category: Seasonal_Forecasts
Section: Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes > Quality assessment statement
---
and January. In the initial processing also October and February values are used, as the heatwave thresholds are centered a three-month month moving window.

#### Helper Functions for Data Processing and Plotting

map plot helper
From https://github.com/COSIMA/ocean-regrid/blob/master/nemo_grid.py
These are the top righ-hand corner of t cells.
Extend south so that Southern most cells can have bottom corners.
Repeat first longitude so that Western most cells have left corners.
Corners of t points. Index 0 is bottom left and then
anti-clockwise.
Check if 'leadtime' is a dim, 'time' is a coord,
and 'time' is indexed by 'leadtime'.
Set 'bnds' variables as coordinates
Clean up 'realization' coordinate and ensure it's a dimension
--- Standard Monthly Climatology ---
Calculate mean for each month (Jan, Feb, etc.)
Calculate Anomalies
--- Month-by-Month Detrending ---
1. Group by month and fit trend using .map()
Extract the coefficients DataArray.
Reconstruct the full trend line (mx + c)
Isolate the slope component.
Subtract the slope
print(f"Number of 'True' points in sesonal static_sea_ice_proxy_mask: {static_not_sea_ice_proxy_mask_s.sum().compute().item()}")
static_not_sea_ice_proxy_mask_s.plot()
Step 1: Compute anomalies
apply the mask here
--------------------------
Step 2: Reindex to (year, month), assing year to fc_start
Step 1: Compute anomalies
--------------------------
Create the 'season_year'
the month coord will be [1, 2, 3, 4, 10, 11, 12]

#### Data Loading
We download the reanalysis and seasonal forecast datasets, regridding the reanalysis data to align with the seasonal forecast grid, and computing the detrended anomalies.
These transformations are performed using the *c3s_eqc_automatic_quality_control* *download.download_and_transform* function and custom preprocessing steps. xesmf conservative regridding was used to transform the ORAS5 ORCA025 grid to a global 1x1 latlon grid. Sensitivity tests were conducted to ensure that the regridding process (comparing area-conservative interpolation used for ORAS5 vs. a version of the two step distance-weighted interpolation that System 9 underwent) produced similar changes to the mean and variance of the original data (not shown).
 - The data is combined and reindexed so that year and calendar month are the dataset coordinates.
 - The long-term climatology and possibly the monthly trend is subtracted.
 - We apply a common land mask and also mask out areas where seasonal forecast data have values corresponding to a monthly mean potential temperature below 0°C.
 - The seasonal forecast vertically integrated potential temperature is converted to OHC, following the convention used in ORAS5.
 - The variable names are changed to reflect their current state.

requests_reanalysis = {leadtime: [] for leadtime in leadtimes}
post processing: conversion and applying the common mask
(the seas fc has an additional mask now, removing cold areas,
computed before converted to anomalies )
convert the seasonal values to potential temperature expressed as OHC
reaname the variables
add inn the common mask again, also using an ice-proxy from the seasonal fc data

```text
Building a flat list of unique reanalysis requests...
Found 155 unique months of reanalysis to download.
```

(seasonal_ohc:section-2)=
### 2. OHC Time Series and Trend Analysis

Compares the model's OHC and heatwave representation against the reanalysis. The time series of globally averaged OHC anomalies clearly shows a strong positive trend over the 1993-2023 period ([](ffig-anom-ts)). This warming trend is present in both the ORAS5 reanalysis and the Météo-France System 9 seasonal forecasts, though it is slightly stornger in Météo-France System 9. Because this predictable, long-term signal could artificially inflate skill scores, the linear trend is removed from both datasets for all subsequent heatwave and skill analyses. The detrended time series shows the interannual variability, such as the strong El Niño events, more clearly.

Time series plotting
Sort the data
Perform the mean operations (using ensemble mean for forecast)
--- Compute Monthly Correlations ---
We take the ensemble mean of the forecast first, then correlate with reanalysis per month
This produces a correlation value for each month [11, 12, 1]
Format the correlation strings for the title
--- Plotting ---
Convert to DataFrames for seaborn
Panel 0: Global
Panel 1: Nino 3.4
--- Execute plotting ---


EQC Quality Assessment: "Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes"
Dataset: seasonal-monthly-ocean [CDS]
Aspect: extremes-detection_q07  |  Category: Seasonal_Forecasts
Section: Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes > Quality assessment statement
---
, 12, 1]
Format the correlation strings for the title
--- Plotting ---
Convert to DataFrames for seaborn
Panel 0: Global
Panel 1: Nino 3.4
--- Execute plotting ---

(ffig-anom-ts)=
#### Figure 2
Annual time series of reanalysis and forecast anomalies according to calendar month. The forecast fields have a nominal start date October 1st. The single dots with a stronger color shows the reanalysis data, while the more lightly colored plot clouds show the seasonal forecasts (one dot per realization). The upper two rows show the anomalies in the original time-series for the global and Niño3.4 area, while the lower panels show the detrended data. The subplot-titles provide the Pearson correlation coefficients of the area averaged anomalies. Note that the y-axis range varies.

---

(seasonal_ohc:section-3)=
### 3. Model Performance vs. Reanalysis

This section directly compares the model's OHC and heatwave representation against the reanalysis, after having detrended both.

#### Correlation of OHC Anomalies

--- 1. Compute monthly correlations ---
Compute Pearson correlation between reanalysis and forecast ensemble mean per calendar month
--- 2. Visualization (3-panel plot) ---
Extract correlation map for the specific lead month
Plot correlation
Map configuration
Add shared horizontal colorbar at the bottom
Clean up memory

```text
/data/common/miniforge3/envs/wp3/lib/python3.12/site-packages/dask/array/numpy_compat.py:58: RuntimeWarning: invalid value encountered in divide
  x = np.divide(x1, x2, out)
```

(fig-corr-map)=
##### Figure 3
Map plot showing the temporal anomaly correlation coefficients (Pearson's) between the forecast ensemble mean and the reanalysis 1993-2023 NDJ OHC every valid grid box.

---

#### Comparison of distributions
We inspect the OHC anomalies in the two data sources to see if they share a similar distribution. We look at November, December, and January simultaneously and pool the seasonal forecast ensemble members. Identical distributions and tails would allow for the use of the same physical magnitudes for a heatwave threshold.

The differences in the probability density functions, the longer tails in the reanalysis versus the more blunt tails in the forecasts justify the use of relative heatwave thresholds depending on the specific data source.

1. Data Selection (NDJ) and Masking
Selecting late autumn/winter months and applying the common ocean mask
2. Spatial Subsampling
Compute early to bring data into memory for weighted statistics and quantiles
3. Area Weights Calculation
Using cosine of latitude to account for grid cell area convergence at poles
Broadcast weights to match the 2D/3D shape of the data for flattening
4. Compute Weighted Statistics
Calculate weighted quantiles for the Q-Q plot (focusing on the extreme upper tail)
5. Prepare Data for PDF/Kurtosis (flattening and dropping NaNs)
Calculate Fisher Kurtosis (Normal distribution = 0)
Positive = Leptokurtic (fat tails), Negative = Platykurtic (thin tails)
Kolmogorov-Smirnov test (unweighted) to check if distributions are identical
Calculate the number of ensemble members
Calculate points per realization for the forecast
Printing Results Table
7. Visualization
Left: Area-Weighted Probability Density Function (KDE)
Right: Q-Q Plot for the Upper Tail Comparison
Execution call

```text
Starting statistical analysis (subsampling step: 5)...
Computing weighted statistics...

--- Statistics Summary ---
           Metric Forecast (S9) Reanalysis (ORAS5)
    StdDev (J/m²)      6.40e+08           6.71e+08
         Kurtosis          4.18               6.50
     Total Points       3687357             118947
Points per Member        118947             118947
K-S Test p-value: 1.73e-47
```

(fig-pdfs)=
##### Figure 4
Distribution analysis of pooled OHC anomalies for the NDJ season. The left plot shows the area-weighted probability density functions (PDFs) for the forecast (blue) and reanalysis (red), where the broader "shoulders" of the blue curve indicate a blunter distribution. The right plot shows a quantile-quantile comparison of the upper tail ($P_{85}$ to $P_{99}$) of the two data sources. The black line falling below the diagonal indicate higher intensity values (longer tails) in the reanalysis data compared to the forecast.

---


EQC Quality Assessment: "Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes"
Dataset: seasonal-monthly-ocean [CDS]
Aspect: extremes-detection_q07  |  Category: Seasonal_Forecasts
Section: Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes > Quality assessment statement
---
tail ($P_{85}$ to $P_{99}$) of the two data sources. The black line falling below the diagonal indicate higher intensity values (longer tails) in the reanalysis data compared to the forecast.

---

#### Comparison of Heatwave Representation
**Heatwaves** are defined as monthly anomalies exceeding a 90th percentile threshold. The thresholds are calculated from a three-month centered moving window, and separately for each data source. For reanalysis data, the threshold is calculated across all years at each grid point. For the seasonal forecast, the threshold is calculated across all years and all ensemble members at each grid point.

--- 1. Data Preparation ---
--- 2. Threshold Calculation (Centered Windows) ---
Reanalysis thresholds (ORAS5 reference)
Seasonal Forecast thresholds (System 9)
Compute p90 values
--- 3. Unified Scaling and Plotting (2x3 Grid) ---
Unified color limits for all 6 panels to ensure fair comparison
Row 0: Reanalysis Reference (ORAS5)
Row 1: Forecast Thresholds (System 9)
Shared colorbar for all subplots
--- 4. Mask Generation ---
--- Call the function ---

```text
Computing p90 thresholds...
```

```text
/data/common/miniforge3/envs/wp3/lib/python3.12/site-packages/numpy/lib/_nanfunctions_impl.py:1617: RuntimeWarning: All-NaN slice encountered
  return fnb._ureduce(a,
```

```text
Thresholds computed.
Generating heatwave masks...
```

##### Figure 5
Map plot showing the definition of heatwave (or the grid cell threshold to be exceeded) in the OHC anomalies for the seasonal forecasts (left) and the reanalysis (right).

---

#### Diagnostic Plots.

We choose 1997 as our year to inspect the spatial patterns of the anomalies.

--- 1. Selection & Computation ---
Bring small spatial slices into memory
--- 2. Figure Setup ---
Consistent scale for rows 1 & 2
Separate scale for the difference row
--- 3. Plotting Grid ---
Row 1: Forecast (Météo-France System 9)
Row 2: Reanalysis (ORAS5)
Row 3: Difference (Bias)
Styling and horizontal colorbars
Positioning colorbars at the bottom
Run the plot

##### Figure 6
Map plots showing ensemble mean OHC anomalies from the seasonal forecast with a nomianal start date 1 October 1997 (upper row), the corresponding reanalysis fields (centre row), and their difference (forecast minus reanalysis, bottom row).

---

(seasonal_ohc:section-4)=
### 4. Metrics: SEDI, BSS, Accuracy

This section presents the primary skill scores (SEDI, BSS, Accuracy).

#### Symmetric Extremal Dependence Index (SEDI)

SEDI (Ferro & Stephenson 2011, [[4]](https://doi.org/10.1175/WAF-D-10-05030.1)) is calculated using the hit rate (H) and false alarm rate (F), as can be found in a contingency table:

$
  \text{SEDI} = \frac{\log(F) - \log(H) - \log(1-F) + \log(1-H)}{\log(F) + \log(H) + \log(1-F) + \log(1-H)}
  $

To avoid numerical issues, logarithms are computed using a lower bound of $10^{-6}$.

#### The Brier Skill Score (BSS)

The Brier Skill Score (BSS) here compares the forecast's Brier score against the Brier score of using the climatology of the reference data set as a forecast. The climatologically based heatwave forecast is near a 10% chance, but is here calculated explicitly per grid point. The BSS’s upper limit is 1, meanwhile it has no lower limit. Scores above zero indicate a forecast better than climatology.

#### Accuracy Assessment
Proportion of correct predictions relative to total events. Forecast accuracy was calculated as:
  $
  \text{Accuracy} = \frac{\text{True Positives} + \text{True Negatives}}{N}
  $
  where $N$ is the total number of events. Accuracy above a random threshold (e.g., 0.82 for a base rate of 0.1) indicates skill better than chance.

#### Pooling Strategy for Skill Metrics in Map Form

To calculate robust skill scores, data is pooled in different ways depending on the metric. The BSS pools the ensemble members at once, while this is done in a later step for SEDI and Accuracy:


EQC Quality Assessment: "Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes"
Dataset: seasonal-monthly-ocean [CDS]
Aspect: extremes-detection_q07  |  Category: Seasonal_Forecasts
Section: Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes > Quality assessment statement
---
Metrics in Map Form

To calculate robust skill scores, data is pooled in different ways depending on the metric. The BSS pools the ensemble members at once, while this is done in a later step for SEDI and Accuracy:

* **For SEDI and Accuracy**: These metrics are calculated from a contingency table (i.e., hits, misses, false alarms, correct rejections). For these every **ensemble member** in every **year** is treated as an independent event. This means we are **pooling across the time dimension (1993-2023) and the ensemble dimension**. This results in a single, robust skill score map that represents the overall performance for that calendar month across the entire period.

* **For the Brier Skill Score (BSS)**: This metric evaluates the performance of a probabilistic forecast compared to a reference forecast (here climatology) for each calendar month and grid cell. The calculation is done in steps:
    1.  First, for each year, the ensemble members are pooled to create a single **forecast probability** for that year (e.g., if 5 of 25 members predict a heatwave, the probability is 0.2).
    2.  The Brier Score is then calculated for each year by comparing that probability to the single observed outcome (`1` or `0`).
    3.  Finally, these annual Brier Scores are averaged across all years to get the final score. The BSS then compares this to the reference score from climatology.

#### Findings
A key finding of this analysis is a divergence between the metrics in several regions ([](fig-global-maps)). The SEDI and Accuracy scores ( > 0.82) indicate that the forecast is better than a reference climatology, while BSS is weakly negative in many regions. A negative BSS indicates that the forecast is less skillful than the long-term climatological average, i.e. assuming a ~10% chance of a heatwave everywhere, every year.

This suggests that while the model has the ability to correctly identify many heatwave events (demonstrated in the positive SEDI), it likely forecasts with too much confidence or has a high "false alarm rate," where it predicts heatwaves that don't occur. The BSS penalizes this lack of reliability, while SEDI, which focuses only on the co-occurrence of extreme events, is less sensitive to it.

The highest skill (SEDI > 0.5) is consistently found in the tropical Pacific Ocean. This region's variability is dominated by the El Niño-Southern Oscillation (ENSO), the primary source of seasonal predictability in the climate system. The model successfully captures the OHC anomalies associated with strong ENSO events.
The skill is lower in the mid-to-high latitudes. These regions are governed by more chaotic and faster-moving atmospheric and oceanic processes, making them more difficult to predict months in advance.

Function to compute skill score maps
--- Step 1: Brier Skill Score  ---
float32 conversion
Calculate probabilities
Calculate Brier Scores
Calculate final BSS
--- Execute the BSS calculation ---
--- Free memory ---
--- Step 2: SEDI and Accuracy ---
lazy and uses int8
Sum over 'year' and 'realization' to create the maps
--- Execute the SEDI/Accuracy calculation ---
--- Calculate final metrics---
--- Combine ---
--- Execute the skill calculation ---

The main plotting cell with mean values in titles
dictionaries
Define the labels you want for each row
-------------------------

(fig-global-maps)=
##### Figure 7
Skill of Météo-France System-9 forecasts with a nominal start date October 1st for November, December, and January, using the ORAS5 reanalysis as the reference dataset. Heatwaves are defined as upper 300m ocean heat content (OHC) anomalies exceeding the 90th percentile within the 1993-2023 period. Rows show: (A) Symmetric Extremal Dependence Index (SEDI), (B) Brier Skill Score (BSS); and (C) Accuracy. In all maps, red areas indicate positive skill (forecast is better than the reference), while blue areas indicate negative skill. Masked areas are shown in grey. For the Accuracy maps (C) the areas with an accuracy lower than what would be expected by random chance (82%) are also shown with a grey color.

---

(seasonal_ohc:section-5)=
### 5. Contingency Table Analysis
We examine the spatially-pooled contingency tables for each month to better understand the forecast behavior.

Create boolean fields.
Define dimensions to sum over
Execute all calculations in a single Dask operation
print("Computing contingency table")
Format to 2 decimal places with a '%' sign

```text
--- Contingency Tables in Percent (%) ---

November:
```

```text
December:
```

```text
January:
```


EQC Quality Assessment: "Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes"
Dataset: seasonal-monthly-ocean [CDS]
Aspect: extremes-detection_q07  |  Category: Seasonal_Forecasts
Section: Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes > Quality assessment statement
---
print("Computing contingency table")
Format to 2 decimal places with a '%' sign

```text
--- Contingency Tables in Percent (%) ---

November:
```

```text
December:
```

```text
January:
```

For all Novembers (one month leadtime, or 1-2 months, to be exact), when the model predicts a marine heatwave, that prediction is correct 32% of the time (Precision). Conversely, the False Discovery Rate is 68%, meaning roughly two-thirds of forecasted heatwaves did not materialize in observations. The predictive skill degrades slightly with lead time; by January (three month leadtime), the probability of a forecasted heatwave actually occurring drops to 26%.

Regarding the capture of observed events (Hit Rate), the model successfully anticipates 33% of the heatwaves that occurred in November. This capability decreases marginally to 30% in December and 27% in January. Overall, the system demonstrates an ability to capture approximately one-third of heatwave events at a one-month lead time, though users must be mindful of the high false alarm ratio.

(seasonal_ohc:section-6)=
### 6. Annual Variability and ENSO Influence
We now investigate the year-to-year skill for each month and its potential connection to ENSO.

---BSS in float ---
--- free memory ---
-------------------
SEDI & Rates from the int8 arrays
The int-based computations
--- free mem ---
-------------------
--- STEP 3: Combine results -------
--- combine and format ---

#### Annual metrics - global

--- Plotting function ---
--- Panel A: RONI NDJ index (narrow bar chart) ---
--- Panel B: MHW Occurrence ---
--- Panel C: Skill Scores ---
--- Panel D: Contingency Rates ---
Shared x-axis limits
2. Run global calculation and plot

(fig-global-ts-in)=
##### Figure 8
Annual time series of globally-averaged forecast performance for the NDJ season. The upper panel depticts occurrence of El Niño according to the NCEP NOAA Relative Oceanic Niño Index (RONI), defines as the three month running average of the relative Niño 3.4 index. The second row depicts global marine heatwave (MHW) occurrence each year, i.e. the percentage of gridcells in heatwave conditions. The third plot shows the SEDI and the BSS in NDJ averaged over the valid ocean region. The contingency table components, the False or True Positives and Negatives each year for a selected month, November, are shown in the lower plot. The figure is similar to Extended Data Fig. 5 in Jacox et. al. 2022, [[1]](https://doi.org/10.1038/s41586-022-04573-9).

---

An analysis of global year-to-year variability in SEDI and BSS ([](fig-global-ts-in)) reveals a divergence in skill metrics. The SEDI scores (orange lines) remain positive and notably peak during major El Niño events (e.g., 1997-98, 2015-16). This indicates that the forecast system is highly effective at signal detection; it successfully identifies high-risk periods during these major climate drivers. Conversely, the BSS (purple lines), which measures probabilistic skill, often deteriorates during these same events. This might indicate that the single-model system is overconfident. The distribution analysis ([](fig-pdfs)) shows that the reference reanalysis exhibits a sharper tail than the forecast ensemble. While the model often captures the correct warming signal, this lower extreme-tail variability may be linked to the observed overconfidence in its probabilistic assignments.

While the model correctly predicts the risk of a heatwave (high SEDI), the probability it assigns is less accurate. This is a common limitation of single-model systems, where the ensemble spread is insufficient to represent true uncertainty. Transitioning to a multi-model ensemble would likely improve reliability for decision-making by increasing that spread.

We focus on a key area: the Niño 3.4 region (5°N-5°S, 170-120°W). The Niño 3.4 region was chosen as it represents the tropical pacific and the ENSO.

#### Annual metrics - Niño 3.4 region

1. Select the data for the Niño 3.4 region
4. Clean up large intermediate masks
del nino34_mask_forecast, nino34_mask_reanalysis
gc.collect();

```text
--- Running Case Study: Niño 3.4 Region ---
```

(fig-nino-ts)=
##### Figure 9
As in [](fig-global-ts-in), but for the Niño 3.4 region.

---


EQC Quality Assessment: "Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes"
Dataset: seasonal-monthly-ocean [CDS]
Aspect: extremes-detection_q07  |  Category: Seasonal_Forecasts
Section: Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes > Quality assessment statement
---
--- Running Case Study: Niño 3.4 Region ---
```

(fig-nino-ts)=
##### Figure 9
As in [](fig-global-ts-in), but for the Niño 3.4 region.

---

In contrast, the skill within the Niño 3.4 region ([](fig-nino-ts)) is much higher and more reliable. Although the time-series exhibits higher variability, which is expected a dynamic and smaller region, the BSS is consistently positive, approaching 1.0 during the major 1997-98 and 2015-16 events. The high SEDI scores further confirm the model's ability to capture these extremes. This confirms that the probabilistic forecasts are both skillful and reliable regarding marine heatwaves during ENSO events, which are the primary driver of seasonal predictability.

(seasonal_ohc:section-7)=
### Summary of Results

This study compares the Météo-France System-9 seasonal forecasts against the ORAS5 ocean reanalysis for forecasts initialized around October 1st for mean monthly values in November, December, and January. The analysis focused on the forecast skill for extreme upper OHC events (anomalies >90th percentile) after removing long-term trends from both datasets. By looking at anomalies, detrending the data, and applying lead-time (or calendar month) dependent percentiles we do our best to match the seasonal data with the reference data. We assess skill 1–3 months after the forecasts nominal start date using metrics such as the Symmetric Extremal Dependence Index (SEDI) and the Brier Skill Score (BSS). The metrics considered follow  Jacox et al. (2022) [[1]](https://doi.org/10.1038/s41586-022-04573-9), however, in this work we look at OHC and not SST. It should be noted that this evaluation and use-case considers a single ocean reanalysis as a reference dataset. Using an ensemble of observational datasets could provide a more robust assessment.

The forecast system demonstrates clear, positive skill in detecting events. The Symmetric Extremal Dependence Index (SEDI) is positive globally and spikes during major El Niño events (e.g., 1997-98, 2015-16). This is corroborated by the contingency rate analysis ([](fig-global-ts-in) lower panel), which shows the True Positive (Hit) Rate increasing dramatically in these years, confirming the model successfully captures the El Niño signal, and the corresponding effect on the MHW index derived from the ocean heat content.

The system provides a valuable tool for anticipation of NDJ upper ocean heatwaves, with its utility and reliability being highest in ENSO-dominated regions. However, outside this region the Brier Skill Score (BSS) is often negative, indicating that the forecast probabilities are less reliable than the static climatological reference forecast (always a 10% chance). This might indicate that the model has too little ensemble spread in these regions. This makes the forecast a good "heads-up" warning for high-risk periods, but for higher reliability outside the main ENSO regions a multi-model system might be useful.

## ℹ If you want to know more

### Key resources

* Seasonal forecast monthly averages of ocean variables DOI: [10.24381/cds.2f9be611](https://doi.org/10.24381/cds.2f9be611)
- ORAS5 global ocean reanalysis monthly data from 1958 to present DOI:  [10.24381/cds.67e8eeb7](https://doi.org/10.24381/cds.67e8eeb7)

Code libraries used:
* [C3S EQC custom functions](https://github.com/bopen/c3s-eqc-automatic-quality-control/tree/main/c3s_eqc_automatic_quality_control), `c3s_eqc_automatic_quality_control`, prepared and advised on by [B-Open](https://www.bopen.eu/)

* xesmf

* [ocean-regrid](https://github.com/COSIMA/ocean-regrid/tree/master) `make_corners` to provide cell edge coordinates for the ORCA025 tripolar grid

* xskillscore

* dask

* xarray

* pandas

* matplotlib

* seaborn

* scipy

* numpy

* cartopy

### References

[[1]](https://doi.org/10.1038/s41586-022-04573-9) Jacox, M. G., Alexander, M. A., Amaya, D., Becker, E., Bograd, S. J., Brodie, S., Hazen, E. L., Pozo Buil, M., & Tommasi, D. (2022). Global seasonal forecasts of marine heatwaves. Nature, 604(7906), 486–490.


EQC Quality Assessment: "Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes"
Dataset: seasonal-monthly-ocean [CDS]
Aspect: extremes-detection_q07  |  Category: Seasonal_Forecasts
Section: Seasonal Forecast Prediction of Upper Ocean Heat Content Extremes > Quality assessment statement
---
S., Hazen, E. L., Pozo Buil, M., & Tommasi, D. (2022). Global seasonal forecasts of marine heatwaves. Nature, 604(7906), 486–490.

[[2]](https://doi.org/10.1007/S00382-021-06101-3) McAdam, R., Masina, S., Balmaseda, M., Gualdi, S., Senan, R., & Mayer, M. (2022). Seasonal forecast skill of upper-ocean heat content in coupled high-resolution systems. Climate Dynamics, 58(11–12), 3335–3350.

[[3]](https://doi.org/10.5281/zenodo.15518106) Specq, D., Dorel, L., Beuvier, J., Ardilouze, C., & Batté, L. (2024). Documentation of the Météo-France seasonal forecasting system 9. Zenodo.

[[4]](https://doi.org/10.1175/WAF-D-10-05030.1) Ferro, C. A. T., & Stephenson, D. B. (2011). Extremal Dependence Indices: Improved Verification Measures for Deterministic Forecasts of Rare Binary Events. Weather and Forecasting, 26(5), 699–713.


EQC Quality Assessment: "Seasonal forecasts bias assessment for impact models"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q02  |  Category: Seasonal_Forecasts
Section: Seasonal forecasts bias assessment for impact models > Quality assessment questions
---
* **Do I have to correct systematic errors in temperature and precipitation before using seasonal forecasts as an input for my crop model?**
* **Can I assume that climate models always produce the same systematic error for seasonal forecasts over a certain area?**

Seasonal forecasts can be used as an input to crop models for agricultural risk assessment [[1]](https://doi.org/10.1038/s41598-018-20628-2) [[2]](https://doi.org/10.1016/j.agrformet.2018.11.029) [[3]](https://doi.org/10.1016/j.cliser.2022.100324). However, the presence of systematic errors can undermine the performance of crop models in a number of ways, particularly when critical thresholds in the plant growth process are considered. Similar problems can arise in other sectoral applications, such as the management of renewable energy facilities [[4]](https://doi.org/10.1016/j.renene.2019.04.135), or the alerting of national health services for the occurrence of summer heat waves [[5]](https://doi.org/10.1007/s00382-021-05828-3). In addition to temperature and precipitation, the most commonly used climate variables for risk assessment in all sectoral applications, the notebook also considers wind speed and dew point temperature. While these variables are commonly used in agricultural applications to calculate potential evapotranspiration, the results of the assessment are also valuable for several other sectors.


EQC Quality Assessment: "Seasonal forecasts bias assessment for impact models"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q02  |  Category: Seasonal_Forecasts
Section: Seasonal forecasts bias assessment for impact models > Quality assessment statement
---
These are the key outcomes of this assessment

* Seasonal forecasts should be bias-corrected before being used as input to climate impact models. Depending on the impact model, it might be advisable to bias-correct the resulting impact indicator [[6]](https://doi.org/10.1175/JHM-D-19-0042.1)
* Model drifts and bias tend to be a complex function of both the start time of the forecast and of its valid time, with a slightly stronger dependence on the valid time for the case of temperature and precipitation. Therefore, the most appropriate approach to produce unbiased forecasts for use in impact models must be selected on a case-by-case basis.[[7]](https://doi.org/10.1007/s00382-019-04640-4) [[8]](https://doi.org/10.1007/s00382-017-3962-9)
* For the implementation of bias correction procedures, it is recommended that reference climatologies are computed for time scales no longer than one month, since the typical model bias changes on a monthly basis. Instead, the correction of model biases on a seasonal (i.e. three months) timescale may lead to a loss of quality.
* Similarly, the calculation of multi-model statistics requires a recalibration of the model output in order to align all models around the same climatological distribution. [[9]](https://doi.org/10.1007/s00382-020-05314-2)
```

attachment:8568344f-a005-4668-b14a-05d8ff7f3b96.png
---
height: 500px
---
The figure shows a sample  of the ECMWF monthly temperature bias for Western North America as a function of the starting month and of the valid month of the forecast. More details on how to interpret the results are available below in the notebook.
```


EQC Quality Assessment: "Seasonal forecasts bias assessment for impact models"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q02  |  Category: Seasonal_Forecasts
Section: Seasonal forecasts bias assessment for impact models > Methodology
---
This notebook provides a comprehensive assessment of the systematic errors of the forecast systems by comparing the model predictions with the ERA5 reanalysis. The analysis is carried out for eight regions selected from those used in the IPCC-AR6 (see figure below), which capture the spatial scale of the main systematic errors already analysed in previous studies [[10]](https://doi.org/10.1007/s00382-019-04640-4)

The analysis and results are organised in the following steps, which are detailed in the sections below:

**[](section-1):**
 * Import required packages
 * Define parameters describing regions, variables, and data
 * Plot the chosen regions
 * Define required functions
 * Show a sample global map of temperature biases

**[](section-2):**
 * Retrieve monthly ERA5 data and compute the regional mean climatology, i.e. 12 values for each area
 * Compute area averages

**[](section-3):**
 * Retrieve monthly seasonal forecast data from all available centres and for all starting dates, compute the monthly mean climatology (i.e. 6 values, since each forecast is valid for 6 months) and subtract the corresponding ERA5 climatology.
 * Compute area averages
 * Plot a sample map

**[](section-4)**

**[](section-5):**
 * Plots are shown in a comprehensive set of panels that provide an overview of the multimodel performance
 * The results are summarised and discussed

Note that crop models and other sectoral applications tend to use daily values rather than the monthly statistics considered in this notebook, and they focus on smaller areas compared to the regions considered below. However, the analysis of monthly data over the IPCC-AR6 regions provides a useful overview of the overall characteristics of the dataset.

Note that for ECCC, only one of the two systems has been analysed (system 3). This is different from the approach taken in the C3S charts, and CanSIPSv2.1, where both systems are combined (system 2 and system 3).


EQC Quality Assessment: "Seasonal forecasts bias assessment for impact models"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q02  |  Category: Seasonal_Forecasts
Section: Seasonal forecasts bias assessment for impact models > Analysis and results > 1. Choose the data to use and setup the code > Import required packages
---
We use the `regionmask package` to extract climatologies for a few selected regiones that have already e been adopted in several climate studies, including the analysis presented in the latest IPCC report.

A description of the regions adopted for climate assesements in the scientific literature is available [here](https://regionmask.readthedocs.io/en/stable/defined_scientific.html).


EQC Quality Assessment: "Seasonal forecasts bias assessment for impact models"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q02  |  Category: Seasonal_Forecasts
Section: Seasonal forecasts bias assessment for impact models > Analysis and results > 1. Choose the data to use and setup the code > Define parameters describing regions, variables, and data
---
In this notebook the analysis of the model bias is done for 5 variables from all 8 originating centres. The SREX regions selected for the analysis are:

- NEAF: Northern-East Africa
- ENA: Eastern North America
- MED: Southern Europe/Mediterranean
- NEB: North-eastern Brazil
- SAS: Southern Asia
- SEA: South-Eastern Asia
- WNA: Western North America
- NWS: Western Coast of South America

A `common_request` is defined in order to cover the entire global domain and the entire hindcast period from 1993 to 2016. Specific data requests are then defined for the reanalysis and for the seasonal forecasts, so that all available months and lead times are retrieved for the analysis.

Define centres with missing variables
Plotting settings
Define data requests


EQC Quality Assessment: "Seasonal forecasts bias assessment for impact models"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q02  |  Category: Seasonal_Forecasts
Section: Seasonal forecasts bias assessment for impact models > Analysis and results > 1. Choose the data to use and setup the code > Plot the chosen regions
---
For the bias analysis we use 8 out of the 45 regions defined in the `regionmask` package in order to assess the systematic errors of the modelling systems under different climatic regimes (tropical, extratropical, continental, maritime) with a number of cases maneageable within the notebook.


EQC Quality Assessment: "Seasonal forecasts bias assessment for impact models"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q02  |  Category: Seasonal_Forecasts
Section: Seasonal forecasts bias assessment for impact models > Analysis and results > 1. Choose the data to use and setup the code > Define the required functions
---
- The `regionalised_spatial_weighted_mean` function extracts the regional means over the selected domains. It uses spatial weighting to account for the latitudinal dependence of the grid size in the lon-lat grids used for the reanalysis and for the forecast models. The bias is then calculated later in the notebook by simply subtracting the regional means.

- The `postprocess_dataarray` function performs two important postprocessings operations required to align the arrays containing the reanalsysis and the forecast data.

- The time dimensions are renamed to allow the broadcasting of the dimension `valid_time`, which is used to compute the bias, independently of the corresponding `lead_times` in the forecast.

- Furthermore, the cumulative variables have to be rescaled in order to make ERA5 data and seasonal forecast data comparable on a monthly timescale. A [conversion table](https://confluence.ecmwf.int/pages/viewpage.action?pageId=197702790)  for accumulated variables is available in the [Copernicus Knowledge Base](https://confluence.ecmwf.int/pages/viewpage.action?pageId=55116796) repository.

- The function `get_seasonal_maps` is used to derive a sample bias map to illustrate how the analysis is performed.

- The `monthly_mean` function will be used to derive the monthly values at the appropriate `monthly_mean` before calculating the bias.

Convert units

(section-2)=


EQC Quality Assessment: "Seasonal forecasts bias assessment for impact models"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q02  |  Category: Seasonal_Forecasts
Section: Seasonal forecasts bias assessment for impact models > Analysis and results > 2. ERA5 data retrieval and area average
---
The `regionalised_spatial_weighted_mean` is applied for all regions and all variables selected for download. All time series of the regionalised means are collected in the same xarray `ds_reanalysis`.

Get the reanalysis data

(section-3)=


EQC Quality Assessment: "Seasonal forecasts bias assessment for impact models"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q02  |  Category: Seasonal_Forecasts
Section: Seasonal forecasts bias assessment for impact models > Analysis and results > 3. Seasonal hindcast data retrieval and area average
---
The `regionalised_spatial_weighted_mean` is applied for all centres, regions and all variables selected for download. Missing variables are handled, so the download and transform process is not interrupted. All time series of the regionalised means are collected in the same xarray `ds_seasonal`.

Get the seasonal forecast data


EQC Quality Assessment: "Seasonal forecasts bias assessment for impact models"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q02  |  Category: Seasonal_Forecasts
Section: Seasonal forecasts bias assessment for impact models > Analysis and results > 3. Seasonal hindcast data retrieval and area average > Show a sample global map of temperature biases
---
To reduce computation, this notebook simplifies the bias analysis by focusing on mean values of surface variables across the chosen regions. The goal is to offer a broad overview of the bias evolution over lead times. Thus, it is crucial that the selected regions exhibit sufficient homogeneity to minimize the compensation effects from systematic errors of varying sign.

This section of the notebook offers a simple tool for exploring sample bias maps and confirming that bias sign remain relatively constant within selected regions. Additionally, it serves as an insight on the behavior of individual models summarized in the comprehensive tables provided later in the notebook.

Set parameters
Get reanalysis map
Get seasonal map
Compute bias

(section-4)=


EQC Quality Assessment: "Seasonal forecasts bias assessment for impact models"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q02  |  Category: Seasonal_Forecasts
Section: Seasonal forecasts bias assessment for impact models > Analysis and results > 4. Compute the hindcast bias
---
The the seasonal hindcast data are arranged in a four-dimensional array `ds_seasonal[i,j,n,k]`, where:
> * `i` is the reference start month of the forecast (from January to December)
> * `j` is the valid time of the forecast (from January to December)
> * `n` is the originating centre of the forecast (currently a maximum of 8)
> * `k` is the region (currently 8)

Correspondingly, ERA5 data are arranged in a three-dimensional array `ds_seasonal[j,n,k]`, where:
> * `j` is the valid time of the forecast (from January to December)
> * `n` is the originating centre of the forecast (currently a maximum of 8)
> * `k` is the region (currently 8)

With such a shaping and setup of the datasets `ds_seasonal` and `ds_reanalysis` the computation of the bias is straightforward.

Avoid ticks interpolation

(section-5)=


EQC Quality Assessment: "Seasonal forecasts bias assessment for impact models"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q02  |  Category: Seasonal_Forecasts
Section: Seasonal forecasts bias assessment for impact models > Analysis and results > 5. Plot and describe results > How to read the results
---
The following tables summarise the monthly bias for chosen C3S seasonal systems (labelled by producing centre) and selected areas.

In each panel, the horizontal axis represents the validity period of the forecast, while the vertical axis shows the corresponding start dates. Thus, each horizontal strip of coloured tiles shows the evolution of the mean temperature bias over the six-month period of validity of each forecast.

Reading each panel from bottom to top, the strips of coloured tiles are progressively shifted by one month, corresponding to the same shift in the starting date of the forecast. The horizontal axis is periodic: forecasts starting after August have their last months on the left side of the panel.


EQC Quality Assessment: "Seasonal forecasts bias assessment for impact models"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q02  |  Category: Seasonal_Forecasts
Section: Seasonal forecasts bias assessment for impact models > Analysis and results > 5. Plot and describe results > Discussion
---
The tables below summarise the monthly bias for all originating centres (columns) and for all selected areas (rows). Note that only one system (system 3) is shown from the ECCC contributions.

**Temperature** - All seasonal forecast models produce a significant temperature bias over most of the regions selected for analysis.  Two centres, DWD and METEOFRANCE, produce a forecast with a predominant warm bias over most regions and in all seasons.  The temperature bias is a function of both the start time and the valid time of the forecast, with a slightly stronger dependence on the valid time. Therefore, the bias does not increase with the lead time of the forecast. Instead, it tends to be a characteristic aspect of each model over the specific region and time of year. The seasonal cycle of the bias is particularly evident over western North America.

**Precipitation** - As in the case of temperature, the bias in precipitation tends to be more dependent on the validity period of the forecasts. The regions with the largest systematic bias are northeastern Brazil and southern Asia, where all models tend to overestimate the amplitude of the seasonal cycle.

**Wind speed** - Unlike temperature and precipitation, the bias of wind speed does not show a significant seasonal cycle over most of the selected regions. The only exceptions are northeastern Brazil and southern Asia, where some models (e.g. ECCC and NCEP) show a positive bias during the months of June to September.

**Dew point temperature** - As in the case of wind speed, the dew point temperature bias tends to have a weaker seasonal cycle than temperature and precipitation. However, some regions (southern Asia and western North America) show a slightly stronger seasonal dependence of the bias. NOTE: Dew point temperature is not available for ECCC.

Create a colorbar for each row


EQC Quality Assessment: "Seasonal forecasts bias assessment for impact models"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q02  |  Category: Seasonal_Forecasts
Section: Seasonal forecasts bias assessment for impact models > ℹ If you want to know more > Key resources
---
Some key resources and further reading were linked throughout this assessment.

The CDS catalogue entries for the data used were:
* Seasonal forecast monthly statistics on single levels: https://cds.climate.copernicus.eu/datasets/seasonal-monthly-single-levels?tab=overview
* ERA5 monthly averaged data on single levels from 1940 to present: https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels-monthly-means?tab=overview

Code libraries used:
* [C3S EQC custom functions](https://github.com/bopen/c3s-eqc-automatic-quality-control/tree/main/c3s_eqc_automatic_quality_control), `c3s_eqc_automatic_quality_control`,  prepared by [B-Open](https://www.bopen.eu/)


EQC Quality Assessment: "Seasonal forecasts bias assessment for impact models"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q02  |  Category: Seasonal_Forecasts
Section: Seasonal forecasts bias assessment for impact models > ℹ If you want to know more > References
---
[[1]](https://doi.org/10.1038/s41598-018-20628-2) Rodriguez, D., De Voil, P., Hudson, D., Brown, J. N., Hayman, P., Marrou, H., & Meinke, H. (2018). Predicting optimum crop designs using crop models and seasonal climate forecasts. Scientific reports, 8(1), 2231.

[[2]](https://doi.org/10.1016/j.agrformet.2018.11.029) Jha, P. K., Athanasiadis, P., Gualdi, S., Trabucco, A., Mereu, V., Shelia, V., & Hoogenboom, G. (2019). Using daily data from seasonal forecasts in dynamic crop models for yield prediction: A case study for rice in Nepal’s Terai. Agricultural and forest meteorology, 265, 349-358.

[[3]](https://doi.org/10.1016/j.cliser.2022.100324) Dainelli, R., Calmanti, S., Pasqui, M., Rocchi, L., Di Giuseppe, E., Monotti, C., ... & Toscano, P. (2022). Moving climate seasonal forecasts information from useful to usable for early within-season predictions of durum wheat yield. Climate Services, 28, 100324.

[[4]](https://doi.org/10.1016/j.renene.2019.04.135) Lledó, L., Torralba, V., Soret, A., Ramon, J., & Doblas-Reyes, F. J. (2019). Seasonal forecasts of wind power generation. Renewable Energy, 143, 91-100.

[[5]](https://doi.org/10.1007/s00382-021-05828-3)  Prodhomme, C., Materia, S., Ardilouze, C., White, R. H., Batté, L., Guemas, V., ... & García-Serrano, J. (2021). Seasonal prediction of European summer heatwaves. Climate Dynamics, 1-18.

[[6]](https://doi.org/10.1175/JHM-D-19-0042.1) Li, W., Chen, J. I. E., Li, L. U., Chen, H. U. A., Liu, B., Xu, C. Y., & Li, X. (2019). Evaluation and bias correction of S2S precipitation for hydrological extremes. Journal of Hydrometeorology, 20(9), 1887-1906.

[[7]](https://doi.org/10.1007/s00382-019-04640-4) Manzanas, R. (2020). Assessment of model drifts in seasonal forecasting: Sensitivity to ensemble size and implications for bias correction. Journal of Advances in Modeling Earth Systems, 12(3), e2019MS001751.

[[8]](https://doi.org/10.1007/s00382-017-3962-9) Hermanson, L., Ren, H. L., Vellinga, M., Dunstone, N. D., Hyder, P., Ineson, S., ... & Williams, K. D. (2018). Different types of drifts in two seasonal forecast systems and their dependence on ENSO. Climate dynamics, 51, 1411-1426.

[[9]](https://doi.org/10.1007/s00382-020-05314-2) Hemri, S., Bhend, J., Liniger, M. A., Manzanas, R., Siegert, S., Stephenson, D. B., ... & Doblas-Reyes, F. J. (2020). How to create an operational multi-model of seasonal forecasts?. Climate Dynamics, 55, 1141-1157.

[[10]](https://doi.org/10.1007/s00382-019-04640-4) Manzanas, R., Gutiérrez, J. M., Bhend, J., Hemri, S., Doblas-Reyes, F. J., Torralba, V., ... & Brookshaw, A. (2019). Bias adjustment and ensemble recalibration methods for seasonal forecasting: a comprehensive intercomparison using the C3S dataset. Climate Dynamics, 53, 1287-1305.


EQC Quality Assessment: "Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q04  |  Category: Seasonal_Forecasts
Section: Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies
---
Production date: 30/05/2024

Produced by: Sandro Calmanti (ENEA), Alessandro Dell'Aquila (ENEA)


EQC Quality Assessment: "Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q04  |  Category: Seasonal_Forecasts
Section: Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies > Quality assessment questions
---
* **What are the hit-rates for terciles categories in seasonal forecasts for anomalies above/below normal?**
* **Can I use information from seasonal forecasts of monthly anomalies to make decisions?**

Seasonal forecasts are not deterministic: they are not expected to provide quantitative information on the value of climate parameters with an attached error bar.

Instead, **seasonal forecasts are probabilistic**: they provide information on how climate parameters are expected to change over the coming months, compared to climatology, in terms of their main statistical aspects, given the current state of the climate system.

A key issue regarding the usability of seasonal forecasts is that end-users do not always have a way to incorporate a probability distribution function (PDF) related to a climate variability in their decision workflow. Many practical decisions, for which seasonal forecast might be beneficial require some form of binary choice [[1]](https://doi.org/10.1016/j.cliser.2024.100496), for example:

- Do I have to plan for an anticipated harvest?
 - Do I have to choose the long-cycle or short-cycle maize variety for the coming season?
 - Will the coming season allow a reduced number of treatments for my crops?
 - Do I have to purchase in advance, and at a lower price, an extra stock of dispatchable energy for air conditioning?

This notebook focuses on the **hit-rate** for terciles categories in seasonal forecasts, a simple yet fundamental building block for measuring the quality, and potential value, of seasonal forecasts.

The hit-rate (also mentioned sometimes as _hit-score_) addresses the key question: how often did the forecasted category with the highest probability actually occur? As such, the hit-rate for terciles categories describes a basic attribute of probabilistic forecasts, i.e. that the outcome of a given season is expected to differ from the normal if the forecast differs from the normal. This characteristic of probabilistic forecasts is called **resolution**.

A complementary aspect in describing the quality of seasonal forecasts is **discrimination**, which is the answer to the opposite question: do the forecasts differ when the outcome differ? As an example if, on average, a forecasting system issues always the same forecast, regardless of the expected outcome, there can be some _hits_ in the forecast but, at the same time, the forecast would not be able to discriminate the occurrence of an event from a non-event. Discrimination is best measured by **Relative Operating Characteristic (ROC)** scores.

As long as there is some resolution or discrimination, there is a potential to extract some valuable information from the forecasts.

Indeed, the WMO [Guidance on Verification of Operational Seasonal Climate Forecasts](https://library.wmo.int/idurl/4/56227) highlights how the use of generalised methodologies for the assessment of seasonal forecasts sometimes represent limited attraction for new users. Therefore, the [Guidance](https://library.wmo.int/idurl/4/56227) recommends a range of methodologies of increasing complexity, including very simple ones like the hit-rate, which are particularly useful for introducing the potential value of seasonal forecasts to non-specialists.


EQC Quality Assessment: "Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q04  |  Category: Seasonal_Forecasts
Section: Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies > Quality assessment statement
---
These are the key outcomes of this assessment

* The hit-rate for terciles cetegories of seasonal forecasts in predicting anomalies at a monthly time-scale depends significantly on the variable and on the region of interest.
* The hit-rates for terciles of different variables do not depend significantly on the forecasting system.
* The larger hit-rates for terciles of monthly anomalies (above 70%) are for 2m temperatures in regions surrounding the Pacific, under the direct influence of El Niño [[2]](https://doi.org/10.1038/s41612-023-00519-8).
* The usability and value of seasonal forecasts must be assessed on a case-by-case basis, by explicitly accounting for the impact on the decision at stake [[3]](https://doi.org/10.1002/wcc.523).
```

attachment:a684544d-7232-4092-82a5-7ec01184a99a.png
---
height: 500px
---
The figure shows a sample  of the hit-rate for terciles of ECMWF monthly anomalies for 2m temperature over South East Asia as a function of the starting month and of the valid month of the forecast. More details on how to interpret the results are available below in the notebook.
```


EQC Quality Assessment: "Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q04  |  Category: Seasonal_Forecasts
Section: Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies > Methodology
---
This notebook provides a comprehensive assessment of the hit-rates for terciles on monthly anomalies of selected ECVs of the forecast systems by comparing the model predictions with the ERA5 reanalysis. The analysis is carried out for eight regions selected from those used in the IPCC-AR6 (see figure below) [[4]](https://doi.org/10.1017/CBO9781139177245.006).

The computation of the hit-score is straightforward: the monthly values of the selected ECV (e.g. temperature)  are firstly divided in terciles (below normal, normal, above normal). Then, the hit-score is computed by counting the number of events when the most populated tercile derived from the distribution of the seasonal forecasts corresponds to the observed tercile of the target month.

The analysis and results are organised in the following steps, which are detailed in the sections below:

**[](section-1):**
 * Import required packages
 * Define parameters describing regions, variables, and data
 * Plot the chosen regions
 * Define functions for the tercile analysis
 * Define plot function

**[](section-2):**
 * Download and process ERA5
 * Download and process seasonal forecast data

**[](section-3)**
 * Analysis of terciles
 * Compute hit-score and show a sample plot

**[](section-4):**
 * Hit-rate for terciles of 2m temperature
 * Hit-rate for terciles of other variables


EQC Quality Assessment: "Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q04  |  Category: Seasonal_Forecasts
Section: Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies > Analysis and results > 1. Choose the data to use and setup the code > Define parameters describing regions, variables, and data
---
The analysis of hit-scores is conducted over 5 different variables  for the 8 originating centres. The AR6 regions selected for the analysis are.

- NEAF: Northern-East Africa
- ENA: Eastern North America
- MED: Southern Europe/Mediterranean
- NES: Eastern Coast of South America
- SAS: Southern Asia
- SEA: South-Eastern Asia
- WNA: Western North America
- NWS: Western Coast of South America

A `common_request` is defined to cover the entire global domain and the entire hindcast period from 1993 to 2016. Specific data requests are then defined for the reanalysis and for the seasonal forecasts, so that all available months and lead times are retrieved for the analysis.

Detrend timeseries
Define centres with missing variables
Define data requests


EQC Quality Assessment: "Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q04  |  Category: Seasonal_Forecasts
Section: Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies > Analysis and results > 1. Choose the data to use and setup the code > Plot the chosen regions
---
For the bias analysis we use 8 out of the 46 AR6 regions defined in the `regionmask` package in order to assess the systematic errors of the modelling systems under different climatic regimes (tropical, extratropical, continental, maritime) with a number of cases manageable within the notebook.


EQC Quality Assessment: "Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q04  |  Category: Seasonal_Forecasts
Section: Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies > Analysis and results > 1. Choose the data to use and setup the code > Define the required functions
---
- The function `reindex_seasonal_forecast` is necessary in the tercile analysis to align the valid time of the forecast to the corresponding date in the reanalysis.

- The function `tercile_analysis` converts the value of both ERA5 data and seasonal forecast data into the corresponding tercile value, which corresponds to below normal, normal, and above normal. The processing of ERA5 and of seasonal forecasts is detailed below.

**Processing of ERA5**

1.	Compute the mean monthly temperature from the reanalysis over the selected AR6 regions.

2. If required, remove the linear trend to emphasise hit-scores associated to natural climate variability rather than to long-term tendencies. This step may be of interest for the analysis of hit-scores for temperature. By default, detrending is not applied in the analysis presented in this notebook. However, the code cell shows how to implement the detrending procedure.

3.	For each month, and for each region, compute the values associated to the 33-th and 66-th percentile of the corresponding monthly distribution

4.	Convert the series of mean monthly temperature anomalies over each region into the corresponding terciles, using the following conversion table:

- -1 = below normal, values below the 33-th percentile
   - 0 , values between the 33-th and the 66-th percentile
   - 1 = above normal, values above the 33-th percentile

**Processing of seasonal forecasts**

1.	Apply step 1. as for ERA5 for each seasonal forecasting system. In this case there is a different temperature anomaly for each originating centre, starting time and valid time for the forecast.

2. If required, apply step 2. as for ERA5 to remove linear trends.

3. Apply step 3. as for ERA5 to obtain the position of the terciles for each originating centre and for each starting time and valid time. In this case, the underlying distribution for the computation of the terciles includes all years in the hindcast as well as all ensemble members. The 33-th and 66-th percentiles depend on the lead-time, i.e. the difference between valid time and starting time, so that the analysis accounts for potential model drifts. In the present analysis, we consider a maximum lead-time of six months.

4. Apply step 4. as for ERA5 to convert temperature anomalies into the corresponding terciles for all regions, for all time months and valid time, and for each member of the ensemble forecast.

5. For seasonal forecasts the most likely tercile in the forecast is derived by computing the mode of the terciles (step 3) over all the ensemble members. The most likely terciles is an array whose dimensions are the originating centres, the region, the starting time and the valid time.

Stack starting_time and leading_month
Shift valid_time
Reindex: valid_time and starting_month
Spatial mean
Detrend timeseries
Compute anomaly
Reindex using valid year/month month
Get quantiles

(section-2)=


EQC Quality Assessment: "Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q04  |  Category: Seasonal_Forecasts
Section: Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies > Analysis and results > 2. Download and process data > Download and process ERA5
---
ERA5 data are converted to tercile values by applying the function `tercile_analysis`. The outcome of the download and transform process is a three-dimensional array containing tercile values, i.e. (-1, 0, 1) for ("_Below normal_, _Normal_, _Above Normal_") respectively, for each of the 8 regions, for 25 years and 12 months each year.

The final outcome of the tercile analysis for each variable is an array with the following dimensions:

> -  valid_year: 25
 > -  valid_month: 12
 > -  region: 8

A separate array is created for each of the 5 selected variables. All arrays are assembled into a single `xarray` dataset for the following computation and visualisations steps.

Get the reanalysis data


EQC Quality Assessment: "Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q04  |  Category: Seasonal_Forecasts
Section: Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies > Analysis and results > 2. Download and process data > Download and process seasonal forecasts
---
Similarly to ERA5 data, seasonal forecast data are converted to tercile values by applying the function `tercile_analysis`. However, in this case the outcome of the download and transform process is a five-dimensional array: besides the 8 regions, 25 years and 12 months corresponding to the valid time of the forecasts, two additional dimensions are necessary to account for the 8 originating centres and for the 12 possible starting months of the forecast. For each starting month, there are 6 tercile values that correspond to the length of the forecast taken into account, whereas the tercile values for the other six months are set to Not-a-Number.

The final outcome of the tercile analysis for each variable is an array with the following dimensions:

> -  valid_year: 25
 > -  valid_month: 12
 > -  starting_month: 12
 > -  region: 8
 > -  centre: 8

All arrays are assembled into a single `xarray` dataset containing all variables for the following computation and visualisations steps.

Get the seasonal forecast data - DETREND

(section-3)=


EQC Quality Assessment: "Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q04  |  Category: Seasonal_Forecasts
Section: Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies > Analysis and results > 3. Hit-score analysis > Analysis of terciles
---
Splitting the distribution of climate parameters into terciles is a simple way of converting probabilistic forecasts into binary information that a user of climate information can consult to make decisions.

For example, a user may ask whether the coming months will be warmer (or colder) than normal, and the answer would be "likely yes" or "likely no" depending on which is the most likely anomaly in the forecast. As a working hypothesis, it is reasonable to assume that specific actions are undertaken when a significant deviation from the norm is expected, for example planting a drought resistant but less productive maize variety if less rainfall than normal is expected, or purchasing in advance, and at a lower price, an extra stock of dispatchable energy for air conditioning systems if warmer than normal conditions are expected during summer.

For the analysis presented in this notebook, the forecasting system scores a hit if the observed monthly anomaly matches the corresponding forecast.

To illustrate how the hit-score analysis works, the three figures below show an example mean 2m temperature over South East Asia, using ECMWF System 5.1 as a sample seasonal forecast.

- The first figure shows a complete series of the terciles derived from ERA5 as a function of year and month. For example, the year 1998, during which a strong El Niño event has been registered, is characterised by a sequence of temperature anomalies above normal, starting from December 1997 to October 1998. Instead, the year 2008, a strong La Niña year, shows a sequence of below normal anomalies.

- The second figure shows, read from below to the top, the tercile analysis for the seasonal forecasts issued during 1998. Each row in the panel corresponds to a single six-month forecast issued at the corresponding starting month on the vertical axis. For the year 1998 most of the forecast starts with a temperature anomaly above normal and persists in this status until the end of the six-month period. Forecasts starting at the end of the previous year, which are represented in the top-left corner of the panel, also start with a warm anomaly that persists throughout the forecast. Since 1998 was a warm year in this region, most of the monthly anomalies of 1998 count as a hit for the forecasts issued in 1998 at all lead times.

plt.suptitle("ERA5")

```text
/data/common/miniforge3/envs/wp3/lib/python3.11/site-packages/dask/core.py:127: RuntimeWarning: divide by zero encountered in divide
  return func(*(_execute_task(a, cache) for a in args))
/data/common/miniforge3/envs/wp3/lib/python3.11/site-packages/dask/core.py:127: RuntimeWarning: invalid value encountered in multiply
  return func(*(_execute_task(a, cache) for a in args))
```


EQC Quality Assessment: "Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q04  |  Category: Seasonal_Forecasts
Section: Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies > Analysis and results > 3. Hit-score analysis > Compute hit-score and show a sample plot
---
By counting the number of hits for all available hindcasts and considering the hits on anomalies both above and below normal, it is possible to obtain a systematic assessment of the hit-rates for terciles on monthly anomalies. Similar hit-scores may also be considered, but for the categories with second- and third- highest probabilities. Other adjustments to the scores may be considered, as suggested by the [WMO Guidance](https://library.wmo.int/idurl/4/56227), by assigning for example a half hit if one of two categories with tied highest probability verifies.

In this assessment we consider the simplest possible definition:

$Hit \: rate = 100 * \frac{Number \: of \: hits}{Number \: of \: forecasts}$

The figure below, which should be read from the bottom to the top, shows in each row the hit rate on monthly anomalies of the forecast starting from the corresponding month indicated on the vertical axis.

For example, the forecasts starting in June (month 6) have a hit-rate above 90% for the first month of the forecast, which is reduced to a value between 80% and 90% for the second month (July). During the following months (August to November) the hit-rate decreases further but remains mostly above 70%.

Note that the horizontal axis is periodic and the tiles in the top left corner of the panel represent the hit-rates for terciles of the forecasts started in August or later during the year.

Compute hit rates
Define the custom colormap
This adjustment allow maz hit-rate values to be correctly represented with the color palette
hit_rate_mask = xr.where(hit_rate == 100., 99.9, hit_rate)

(section-4)=


EQC Quality Assessment: "Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q04  |  Category: Seasonal_Forecasts
Section: Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies > Analysis and results > 4. Plot and describe results > Hit-rate for 2 m temperature
---
The same analysis described in section 3 above. can be replicated for all forecasting systems and for all the selected sample regions.

Overall, the seasonal forecasts of 2m temperature have a significant, and potentially usable, hit-rate for terciles in regions closer to the tropics and under the direct influence of El Niño. The further away from the tropics, the lower the expected hit-rate on monthly anomalies.

Hit-rates smaller than 50% have been masked out, with the assumption that lower hit-rates would be of limited use for most practical applications. Although this working hypothesis can not be assumed to be strictly true for all purposes, it is a useful assumption for a discussion on the general characteristics of seasonal forecasts [[3]](https://doi.org/10.1002/wcc.523) [[5]](https://doi.org/10.1016/j.crm.2021.100375) [[6]](https://doi.org/10.1016/j.cliser.2023.100347) [[7]](https://doi.org/10.1016/j.cliser.2023.100346).

The figure below shows that, among the sample regions considered in this analysis, those with the higher and more persistent hit-rate are South East Asia (SEA) and North West South America (NWS). These areas are under the direct influence of ENSO and are expected to benefit from a significantly higher degree of predictability compared to other regions. Other areas with significant, and potentially usable, in practical applications are Northern-East Africa (NEAF) and (North East South America) NES. Other areas like Southern Asia (SAS) and Souther Europe/Mediterranean (MED) have a moderate hit-rate, limited mostly to short lead times, whereas NWS shows only scattered cases of hit-rate above 50% [[2]](https://doi.org/10.1038/s41612-023-00519-8) [[8]](https://doi.org/10.1175/BAMS-D-19-0019.1) [[9]](https://doi.org/10.1175/WAF-D-19-0106.1) [[10]](https://doi.org/10.1007/s00704-018-2421-9).

Compared to the similar assessment conducted ([](./seasonal_seasonal-monthly-single-levels_forecast-skill_q02.ipynb)), a significant aspect of the hit-rates for terciles is their weak dependence on the originating centre. Although minor differences exist in the performance of the different forecasting systems, the overall patterns of significant hit-rates and even their seasonality is very similar, e.g. higher hit-rates in North West South America during the period October-December associated to El Niño.

Create a colorbar for each row


EQC Quality Assessment: "Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q04  |  Category: Seasonal_Forecasts
Section: Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies > Analysis and results > 4. Plot and describe results > Hit-rate for other variables
---
A similar analysis can be conducted for other variables such as monthly accumulated rainfall, wind speed, solar radiation and dew point temperature.

* For **total precipitation** the hit-rate for terciles of monthly anomalies is mostly very small, except for the NES and SEA. Notably, the window of predictability is concentrated in specific seasons: March-May for NES and June-November for SEA [[11]](https://doi.org/10.1175/JHM-D-19-0095.1)

* For **surface solar radiation downwards** the only region with potentially exploitables hit-rates is South East Asia (SEA);  limited hit-rate reported over Europe. [[12]](https://doi.org/10.5194/essd-13-2701-2021) [[13]](https://doi.org/10.1016/j.renene.2019.03.134).

* For **10m wind speed** seasonal forecasts produce only limited cases of hit-rates above 60% and only at very short lead times. [[14]](https://doi.org/10.1016/j.renene.2019.04.135).

* For **2m dew point temperature** the pattern of significant hit-rates is clearly linked to the hit-rates of **2m temperature** coherently with an approximately constant pattern of relative humidity whereby the dewpoint temperature is directly related to local temperature by the Clausius-Clapeyron relation [[11]](https://doi.org/10.1175/JHM-D-19-0095.1).

Similarly to **2m temperature** hit-rates for terciles of seasonal forecasts for other variables have a weak dependence on the originating centres.

The analysis highlights how seasonal forecasts provide a significant hit-rate for terciles of anomalies observed at a monthly time scale mainly for some specific areas and variables.

Nevertheless, valuable information can be extracted by conducting tailored analysis for specific applications [[3]](https://doi.org/10.1002/wcc.523).

Overall, this analysis highlights how seasonal forecasts can not be used as an extension of weather predictions to derive information on expected climate anomalies at monthly or shorter time scales. The actual added value of the use of seasonal forecasts should be always assessed on the basis of the specific applications and decision making processes. Other use cases can be sensitive to different indicators of the quality of seasonal forecasts such as the chance above normal of multiple categories (see for example the [Met Office 3-month outlooks](https://www.metoffice.gov.uk/binaries/content/assets/metofficegovuk/pdf/business/public-sector/civil-contingency/3moutlook_jja_v2.pdf)) or the possibility of anticipating the likelihood of extreme events [[16]](https://doi.org/10.1038/s41467-023-42377-1).

Create a colorbar for each row


EQC Quality Assessment: "Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q04  |  Category: Seasonal_Forecasts
Section: Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies > ℹ If you want to know more > Key resources
---
The CDS catalogue entries for the data used were:
* Seasonal forecast monthly statistics on single levels: https://cds.climate.copernicus.eu/datasets/seasonal-monthly-single-levels?tab=overview
* ERA5 monthly averaged data on single levels from 1940 to present: https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels-monthly-means?tab=overview

More information on the verification of forecasts are available on the [Forecast User Guide](https://confluence.ecmwf.int/display/FUG/Forecast+User+Guide) at ECMWF. In particular, key concepts on the use of contingency tables are discussed in the section dedicated to [Usefulness of the forecast and Cost/Benefit Approach](https://confluence.ecmwf.int/display/FUG/Section+12.A+Statistical+Concepts+-+Deterministic+Data#Section12.AStatisticalConceptsDeterministicData-UsefulnessoftheForecast-ACost/BenefitApproach)

Code libraries used:
* [C3S EQC custom functions](https://github.com/bopen/c3s-eqc-automatic-quality-control/tree/main/c3s_eqc_automatic_quality_control), `c3s_eqc_automatic_quality_control`,  prepared by [B-Open](https://www.bopen.eu/)
* The `regionmask` package for the computation of regionalised statistics is available [here](https://regionmask.readthedocs.io/en/stable/index.html)


EQC Quality Assessment: "Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q04  |  Category: Seasonal_Forecasts
Section: Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies > ℹ If you want to know more > References
---
[[1]](https://doi.org/10.1016/j.cliser.2024.100496) Khosravi F,  Bruno Soares M, Teixeira M, Fontes N , Graca Antonio (2024), Assessing the usability and value of a climate service in the wine sector, Climate Services, Volume 34, 100496

[[2]](https://doi.org/10.1038/s41612-023-00519-8) Ludescher, J., Bunde, A. & Schellnhuber, H.J (2023).  Forecasting the El Niño type well before the spring predictability barrier. npj Clim Atmos Sci 6, 196.

[[3]](https://doi.org/10.1002/wcc.523) Bruno Soares M., Daly M, Dessai S (2018) Assessing the value of seasonal climate forecasts for decision-making. WIREs Clim Change. 9:e523.

[[4]](https://doi.org/10.1017/CBO9781139177245.006) Seneviratne, S. I., Nicholls, N., Easterling, D., Goodess, C. M., Kanae, S., Kossin, J., Zwiers, F. W. (2012). Changes in Climate Extremes and their Impacts on the Natural Physical Environment. In C. B. Field, V. Barros, T. F. Stocker, & Q. Dahe (Eds.), Managing the Risks of Extreme Events and Disasters to Advance Climate Change Adaptation: Special Report of the Intergovernmental Panel on Climate Change (pp. 109–230). chapter, Cambridge: Cambridge University Press.

[[5]](https://doi.org/10.1016/j.crm.2021.100375) MacLeod D, Kniveton D R, Todd M C,(2021) Playing the long game: Anticipatory action based on seasonal forecasts, Climate Risk Management, Volume 34, 2021, 100375,

[[6]](https://doi.org/10.1016/j.cliser.2023.100347) Terrado M, Marcos R,  González-Reviriego N, Vigo I, Nicodemou A, Graça A, Teixeira M, Fontes N, Silva S, Dell'Aquila A, Ponti L, Calmanti S, Bruno Soares M, Khosravi F, Caboni F, (2023) Co-production pathway of an end-to-end climate service for improved decision-making in the wine sector, Climate Services, Volume 30, 100347,

[[7]](https://doi.org/10.1016/j.cliser.2023.100346) Dell'Aquila, A. Graça, A. Teixeira M Fontes N, Gonzalez-Reviriego N, Marcos-Matamoros R, Chou C, Terrado M, Giannakopoulos C, Varotsos K, Caboni F, Locci R, Nanu M, Porru S, Argiolas S, Bruno Soares M, Sanderson, M. Bruno Soares, M Sanderson, M. (2023)  Monitoring climate related risk and opportunities for the wine sector: The MED-GOLD pilot service, Climate Services, 2023, 30, 100346

[[8]](https://doi.org/10.1175/BAMS-D-19-0019.1) Weisheimer, A., D. J. Befort, D. MacLeod, T. Palmer, C. O’Reilly, and K. Strømmen (2020) : Seasonal Forecasts of the Twentieth Century. Bull. Amer. Meteor. Soc., 101, E1413–E1426,

[[9]](https://doi.org/10.1175/WAF-D-19-0106.1) Gubler, S., and Coauthors (2020)  Assessment of ECMWF SEAS5 Seasonal Forecast Performance over South America. Wea. Forecasting, 35, 561–584,

[[10]](https://doi.org/10.1007/s00704-018-2421-9) Mohanty, U.C., Nageswararao, M.M., Sinha, P. et al. (2019) Evaluation of performance of seasonal precipitation prediction at regional scale over India. Theor Appl Climatol 135, 1123–1142


EQC Quality Assessment: "Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: forecast-skill_q04  |  Category: Seasonal_Forecasts
Section: Evaluation of the hit-rate of seasonal forecasts for tercile categories of monthly anomalies > ℹ If you want to know more > References
---
Nageswararao, M.M., Sinha, P. et al. (2019) Evaluation of performance of seasonal precipitation prediction at regional scale over India. Theor Appl Climatol 135, 1123–1142

[[11]](https://doi.org/10.1175/JHM-D-19-0095.1) Roy, T., X. He, P. Lin, H. E. Beck, C. Castro, and E. F. Wood (2020) Global Evaluation of Seasonal Precipitation and Temperature Forecasts from NMME. J. Hydrometeor., 21, 2473–2486

[[12]](https://doi.org/10.5194/essd-13-2701-2021) Lorenz, C., Portele, T. C., Laux, P., and Kunstmann, H.(2021) Bias-corrected and spatially disaggregated seasonal forecasts: a long-term reference forecast product for the water sector in semi-arid regions, Earth Syst. Sci. Data, 13, 2701–2722,

[[13]](https://doi.org/10.1016/j.renene.2019.03.134) De Felice M, Bruno Soares M, Alessandri A, Troccoli A,(2019) Scoping the potential usefulness of seasonal climate forecasts for solar power management, Renewable Energy, Volume 142, Pages 215-223,

[[14]](https://doi.org/10.1016/j.renene.2019.04.135) Lledó Ll., Torralba V.,  Soret A, Ramon J, Doblas-Reyes FJ (2019) Seasonal forecasts of wind power generation, Renewable Energy, Volume 143, 91-100

[[15]](https://doi.org/10.1098/rsif.2013.1162 ) Weisheimer A. and Palmer T. N. (2014) On the reliability of seasonal climate forecastsJ. R. Soc. Interface.1120131162

[[16]](https://doi.org/10.1038/s41467-023-42377-1) Dunstone, N., Smith, D.M., Hardiman, S.C. et al. Windows of opportunity for predicting seasonal climate extremes highlighted by the Pakistan floods of 2022. Nat Commun 14, 6544 (2023).


EQC Quality Assessment: "Assessing the impact of spatial scale and temporal trends on seasonal forecast quality"
Dataset: seasonal-monthly-single-levels [CDS]
Aspect: resolution_q06  |  Category: Seasonal_Forecasts
Section: Assessing the impact of spatial scale and temporal trends on seasonal forecast quality
---
Production date: 30.04.2025

Produced by: Johannes Langvatn (METNorway), Johanna Tjernström (METNorway)