A goal of the National Oceanic and Atmospheric Administration (NOAA) Warn-on-Forecast (WoF) project is to provide rapidly updating probabilistic guidance to human forecasters for short-term (e.g., 0-3 h) severe weather forecasts. Several case studies have shown that experimental WoF systems (WoFS) can produce accurate short-term probabilistic guidance for hazards such as tornadoes, hail, and heavy rainfall. However, without an appropriate probabilistic verification method for WoFS-style forecasts (which provide guidance for individual thunderstorms), a robust evaluation of WoFS performance has been lacking. In this dissertation, I develop a novel object-based verification method for short-term, storm-scale probabilistic forecasts and apply it to WoFS probabilistic mesocyclone guidance and further adapted to evaluate machine learning-based calibrations of WoFS severe weather probabilistic guidance.

The probabilistic mesocyclone guidance was generated by calculating grid-scale ensemble probabilities from WoFS forecasts of updraft helicity (UH) in layers 2—5 km (mid-level) and 0—2 km (low-level) above ground level (AGL) aggregated over 60-min periods. The resulting ensemble probability swaths are associated with individual thunderstorms and treated as objects. Each ensemble track object is assigned a single representative probability value. A mesocyclone probability object, conceptually, is a region bounded by the ensemble forecast envelope of a mesocyclone track for a thunderstorm over 1 hour. The mesocyclone probability objects were matched against rotation track objects in Multi-Radar Multi-Sensor data using the total interest score, but with the maximum displacement varied between 0, 9, 15, and 30 km. Forecast accuracy and reliability were assessed at four different forecast lead time periods: 0-60 min, 30-90 min, 60-120 min, and 90-150 min. In the 0-60 minute forecast period, the low-level UH probabilistic forecasts had a POD, FAR, and CSI of 0.46, 0.45, and 0.31, respectively, with a probability threshold of 22.2% (the threshold of maximum CSI). In the 90-150 minute forecast period, the POD and CSI dropped to 0.39 and 0.27 while FAR remained relatively unchanged. Forecast probabilities >60% over-predicted the likelihood of observed mesocyclones in the 0-60 min period; however, reliability improved when allowing larger maximum displacements for object matching and at longer lead times.

To evaluate the ability of machine learning (ML) models to calibrate WoFS severe weather guidance, the probability object-based method was generalized for identifying any ensemble storm track (based on individual ensemble updraft tracks rather than mesocyclone tracks). Using these ensemble storm tracks, three sets of predictors were extracted from the WoFS forecasts: intra-storm state variables, near-storm environment variables, and morphological attributes of the ensemble storm tracks. Random forests, gradient-boosted trees, and logistic regression algorithms were then trained to predict which WoFS 30-min ensemble storm tracks will produce a tornado, severe hail, and/or severe wind report. To provide a baseline against which to test the ML models’ performance, I extracted the probability of mid-level UH exceeding a threshold (tuned per severe weather hazard) from each ensemble storm track. The three ML algorithms discriminated well for all three hazards and produced far more reliable probabilities than the UH-based predictions. Using state-of-the-art ML interpretability methods, I found that the ML models learned sound physical relationships and the appropriate responses to the ensemble statistics. Intra-storm predictors were found to be more important than environmental predictors for all three ML models, but environmental predictors made positive contributions to severe weather likelihood in situations where the WoFS fails to analyze ongoing convection. Overall, the results suggest that ML-based calibrations of dynamical ensemble output can improve short term, storm-scale severe weather probabilistic guidance.