ECMWF maintains a comprehensive range of verification statistics to evaluate the accuracy of its forecasts. Each year, a summary of verification results is presented to ECMWF’s Technical Advisory Committee (TAC). Their views about the performance of the operational forecasting system in 2023 are given in the box.
With the implementation of Cycle 48r1 of ECMWF’s Integrated Forecasting System (IFS) on 27 June 2023, the horizontal resolution of medium-range ensemble forecasts (ENS) has increased from 18 to 9 km. This is the same resolution as the current high-resolution forecast (HRES). Cycle 48r1 also brought changes to extended-range ensemble forecasts, notably an increase in members from 51 to 101, and many improvements to the data assimilation system as well as the forecast model. The largest forecast skill improvements are associated with the ensemble forecast because of the ENS resolution upgrade. ENS scores of surface variables are improved by 2% to 6%, upper‑air variables typically by around 1% to 3%.
The Cycle 48r1 resolution increase improves ENS tropical cyclone track and intensity forecasts, with position errors reduced by up to 10% and core pressure errors reduced by around 20%. Tropical cyclone track forecasts are improved because the 9 km model has a reduced translation speed bias. Intensity forecasts are improved because the higher horizontal resolution allows for a better representation of the strong horizontal gradients associated with tropical cyclones. This is also reflected in a strongly increased tropical cyclone intensity spread of the ENS, leading to a better spread–error relationship.
Compared to other centres, ECMWF has been able to consolidate its overall lead for upper-air parameters in the medium range thanks to the new Cycle 48r1. For surface parameters, such as 2 m temperature and precipitation, some other centres have, however, taken the lead in the short range. Nevertheless, for both 2 m temperature and 10 m wind speed, the number of large ENS errors has further decreased. Cycle 49r1, which is scheduled for implementation in autumn 2024, will bring substantial improvements near the surface due to the assimilation of 2 m temperature over land as part of the 4D‑Var data assimilation system and the introduction of the SPP scheme for model perturbations, among other changes. In terms of forecast skill for ocean wave parameters, ECMWF is leading compared to other global centres both in peak period and significant wave height.
ECMWF’s new experimental in-house machine-learning (ML) model AIFS (Artificial Intelligence/Integrated Forecasting System) improves on the HRES in some aspects, and its forecasts are competitive with the best from external machine-learning models. All of these models (both external and in-house) are trained on ERA5 reanalysis data.
The increase in ensemble size from 51 to 101 of the extended-range forecast has led to improvements in forecast performance across all parameters over the forecast range from week one to week four. The change in forecast frequency from bi‑weekly to daily has further increased the effective utility of the forecast. A major challenge for extended-range prediction is still posed by transitions in the large-scale flow pattern that lead to marked changes in temperature anomalies, such as the switch from warm to cold anomalies in Northern Europe in October 2023.
ECMWF’s seasonal forecast provided very good predictions of the transition early in 2023 from La Niña to El Niño conditions, even on the long-range (one year) timescale. The exceptionally warm northern hemisphere autumn season 2023 was well captured. However, the cold anomaly in northern Europe was missed.
Note that the TAC’s assessment of forecast performance in the box below is based on evaluation results that were available by September 2023, so it does not reflect the improvements due to Cycle 48r1 which became visible in the scores after that date.
The complete set of annual verification results is available in ECMWF Technical Memorandum No. 911 on ‘Evaluation of ECMWF forecasts, including the 2023 upgrade’, downloadable from https://www.ecmwf.int/en/publications/technical-memoranda.
The following are other sources of information about verification and forecasting system changes.
- Verification as part of ECMWF’s charts page: https://charts.ecmwf.int
- World Meteorological Organization (WMO) inter-comparison of global model forecast skill: https://wmolcdnv.ecmwf.int
- WMO ocean wave model intercomparison results: https://confluence.ecmwf.int/display/WLW/WMO+Lead+Centre+for+Wave+Forecast+Verification+LC-WFV
- List of ‘Known IFS Forecasting Issues’: https://confluence.ecmwf.int/display/FCST/Known+IFS+forecasting+issues
- IFS cycle changes since 1985: http://www.ecmwf.int/en/forecasts/documentation-and-support/changes-ecmwf-model
Assessment of ECMWF’s Technical Advisory Committee, 19–20 October 2023
With regard to its overall view of the performance of ECMWF’s operational forecasting system, the Committee:
a) congratulated ECWMF on the successful implementation of 48r1, the first model cycle to be implemented at the new Bologna facility;
b) welcomed early indications showing 48r1 having generally positive impacts on verification scores whilst recognizing that the impact of 48r1 is generally not yet reflected in many of the scores and metrics used in this assessment summary;
c) welcomed that some of the largest improvements in the 48r1 scorecards are for the ENS where the change in horizontal resolution to match HRES has brought positive impact, validating the decision to increase resolution;
d) congratulated ECMWF’s change in the extended forecast running daily at 48r1, noting this provides additional lead time for major events and transitions in the weeks 2 to 4 period;
e) recognized that in the absence of a new model upgrade in 2022 due to the move to Bologna, changes in scores often reflected interannual variability;
f) noted that in spite of there being no model upgrade in 2022, ECMWF maintained their lead over other centres in many metrics, especially upper air;
g) noted that the magnitude of the lead between ECMWF and other centres in the upper air has decreased for some scores, for example 850 hPa temperature CRPSS and verification against radiosondes in the tropics, but increased for others, for example 500 hPa anomaly correlation scores;
h) noted that outside of summer, ECMWF now lags behind some other centres for short-range scores in 2 m temperature and precipitation over Europe; this is thought to be linked to challenges in cold, stable airmasses and handling of light precipitation and other centres improving their own outputs;
i) acknowledged that regional biases in surface weather parameters are complex, for example in 2m temperature and dewpoint biases around Europe, and welcomed ongoing research here;
j) noted an increased bias in cloud cover introduced at 47r3 is likely to be addressed by 48r1 whilst a night-time continental positive bias in 10m wind scores is likely to be addressed at 49r1;
k) acknowledged that 47r3 has resulted in improved 850 hPa wind in the tropics but 850 hPa temperature here has degraded compared to analysis. Meanwhile, improvement in net short-wave radiation in the tropics has led to the smallest errors in this parameter ECMWF has ever had;
l) acknowledged that ECMWF maintained its long-term lead over other centres for significant wave height and its more recently developed lead for peak period scores;
m) noted some of the smallest errors so far in the handling of tropical cyclones, although there is still a slow bias for tropical cyclone propagation. Early indications suggest 48r1 offers some general improvements above 47r3;
n) noted that changes in some EFI verification scores, for example a drop in scores for precipitation, relative to last year are considered related to interannual variability rather than deficiencies in the forecast system;
o) acknowledged that improvements in CAMS following an update in 2022 have led to a return to previous verification scores. Noted, also, that the meteorological performance of some outputs in the Tropics performs comparably to HRES even though the resolution of CAMS here is 40 km;
p) noted that extended-range parameters continue to show improvements over climatology, more so at week 2 than weeks 3 and 4, and the highest ever real-time forecast precipitation skill was registered. Improvements for warm summer anomalies are greater than for cold winter anomalies, reflecting inherent forecast challenges;
q) acknowledged that in spring 2022 ECMWF seasonal output was too fast, with a transition from La Niña to neutral conditions, but by autumn 2022 ECMWF seasonal output captured the transition from La Niña to El Niño well. This is important given tropical Pacific SST [sea-surface temperature] anomalies are the biggest driver of global seasonal variation;
r) noted that the seasonal forecast for winter 2022–23 showed good skill in capturing the anomalously warm winter in Europe, capturing the idea of the warm anomaly being greater in northern and eastern Europe. Signals elsewhere in the northern hemisphere were also good even though the magnitude was not quite captured, for example over North America;
s) noted that the seasonal forecast for summer 2023 captured the warm European summer anomaly but did not capture the blocking transition that led to a shift in the location of the largest anomalies over the course of the summer. Further improvement in identification and magnitude of heat waves in the weeks 2 to 4 period would be welcomed;
t) appreciated the continued development of new diagnostics and very good support ECMWF provided to Member and Co‑operating States over the last year, with engagement via many mechanisms including online support, the annual UEF [Using ECMWF’s Forecasts], online seminars, new format of site visits and meteorological representatives at Member States.
