What is it about?
The paper introduces a new method to improve the estimation of interrater agreement in classification tasks. The method (which relies on Bayesian modelling) is based on the estimation of the Belonging Threshold (BT) of each rater, that is the minimally sufficient degree of belonging of an object to a category needed by a rater to classify an object into a category. The role of the BT in the interrater agreement estimation is explored and discussed also by means of practical examples.
Featured Image
Why is it important?
It is shown how classical methods for the estimation of interrater agreement are affected by the BT. The proposed method is able to clearly improve the estimation of the "true" agreement between independent raters.
Perspectives
The method can be applied to any field in which classification is important such as medical dispatch, observation tasks, sport refereeing, economic choices. It will be extended to the case of polytomous classification. The method can be used to improve the training of the raters by focusing either on particular raters whose BT is far from optimal, or on specific classifications that result critical for most raters.
Andrea Spoto
Universita degli Studi di Padova
Read the Original
This page is a summary of: The role of raters threshold in estimating interrater agreement., Psychological Methods, October 2021, American Psychological Association (APA),
DOI: 10.1037/met0000416.
You can read the full text:
Contributors
The following have contributed to this page
