Challenges of Correct Validation

What is it about?

This article discusses examples of where the validation of machine learning methods can and does go wrong. There are shockingly high estimates of how many research results in the domain of machine learning may be flawed because of mistakes in validation. I argue in the paper that this needs urgent attention, in particular where non-experts apply machine learning techniques to their domain. I furthermore argue that also more research is needed into how to judge validation results because it is not obvious whether differences between the success of different methods are indeed always significant. This becomes particularly difficult in areas where recent advances have meant that competing methods distinguish themselves with success rates that differ by only a few percent or less.

Featured Image

Why is it important?

The correct validation of machine learning methods is of extreme importance because only correctly validated methods will perform as expected when applied to new problems or when used in real world applications. If we continue to see machine learning related papers presenting results that cannot actually be achieved in applications this runs the danger of completely discrediting the field.

Perspectives