What is it about?
In addition to testing whether forecast model A is better than (or not) model B according to an intensity-based measure, such as RMSE, it is useful to test whether model A is better more often than model B. For example, if two football teams, A and B, play each other many times, you might want to know the average score differential from each game. But, really, it is how often A beats B that matters in the end. This paper also proposes using the power-divergence family of tests, and evaluates its accuracy (in the sense of type-I error size) and power empirically in the face of temporal dependence and contemporaneous correlation (when A and B are correlated with each other). It is found that the power-divergence test is robust to these dependencies, and therefore viable for use in the competing forecast verification domain.
Featured Image
Photo by Quino Al on Unsplash
Why is it important?
Often forecast model A is a modification of model B so that they might not differ by much according to measures like RMSE, and thus not be found to be statistically significantly different. However, if A is truly better, then perhaps it would win more "games" than B, even if by a close margin. This type of testing has not been prolific in forecast verification, and this paper attempts to change that, as well as propose an existing statistical test procedure that does not require resampling. The procedure is found through empirical testing to be robust to the types of deviations from the standard assumptions that are common in competing forecast verification.
Perspectives
Read the Original
This page is a summary of: Competing Forecast Verification: Using the Power-Divergence Statistic for Testing the Frequency of “Better”, Weather and Forecasting, June 2023, American Meteorological Society,
DOI: 10.1175/waf-d-22-0201.1.
You can read the full text:
Contributors
The following have contributed to this page