Uncertainty quantification for data-driven weather models

Data-driven machine learning methods for weather forecasting have experienced a steep progress over the last years, with recent studies demonstrating substantial improvements over physics-based numerical weather prediction models. Beyond improved forecasts, the major advantages of purely data-driven models are their substantially lower computational costs and faster generation of forecasts, once a model has been trained. However, in contrast to ensemble forecasts from physical weather models, most efforts in data-driven weather forecasting have been limited to deterministic, point-valued predictions only, making it impossible to quantify forecast uncertainties which is crucial for optimal decision making in applications. Our overarching aim is to evaluate and compare methods for creating probabilistic forecasts from data-driven weather models. The uncertainty quantification (UQ) approaches we compare are either based on generating ensemble forecasts from data-driven weather models via perturbations to the initial conditions, or based on statistical post-hoc UQ methods. The perturbation-based methods either leverage initial conditions from the ECMWF IFS ensemble, add random Gaussian noise to the deterministic initial conditions, or add random field perturbations based on past observations (Magnusson et al., 2009). The post-hoc approaches operate on deterministic forecasts and quantify forecast uncertainty using established post-processing methods, namely distributional regression networks (Rasp and Lerch, 2018) and isotonic distributional regression (Walz et al., 2022; Henzi et al., 2021). Using forecasts from Pangu-Weather (Bi et al., 2023), we evaluate these UQ methods over Europe for selected user-relevant weather variables, such as wind speed at 10 m, temperature at 2 m, and geopotential height at 500 hPa. We focus on daily initialised Pangu-Weather forecasts for 2022 with a forecast horizon of up to 7 days and compare their performance against ECMWF IFS ensemble forecasts. Our results suggest that Pangu-Weather predictions combined with UQ approaches yield improvements over the ECMWF ensemble forecasts for lead times of up to 5 days in terms of the Continuous Ranked Probability Score. However, it strongly depends on the variable of interest which of the UQ methods performs best, none of the different UQ methods performs best over all variables and lead times. Post-hoc UQ methods tend to perform better for shorter lead times, while initial condition perturbations are superior for longer lead times, with in particular the random field method showing promising results.   References: Bi, K., Xie, L., Zhang, H., Chen, X., Gu, X. and Tian, Q. (2023). Accurate medium-range global weather forecasting with 3D neural networks. Nature, 619, 533–538. Henzi, A., Ziegel, J. F. and Gneiting, T. (2021). Isotonic distributional regression. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83, 963–993. Magnusson, L., Nycander, J. and Källén, E. (2009). Flow-dependent versus flow-independent initial perturbations for ensemble prediction. Tellus A: Dynamic Meteorology and Oceanography, 61, 194. Rasp, S. and Lerch, S. (2018). Neural networks for postprocessing ensemble weather forecasts. Monthly Weather Review, 146, 3885–3900. Walz, E.-M., Henzi, A., Ziegel, J. and Gneiting, T. (2022). Easy Uncertainty Quantification (EasyUQ): Generating Predictive Distributions from Single-valued Model Output. Preprint, available at https://arxiv.org/abs/2212.08376.