Numbers from ratings: Scientists' measurement concepts do not match what raters think

What is it about?

What is going on in raters' minds while ticking scales is very different from scientists' concepts of scientific measurement and quantification. Scientists' high theoretical aspirations for quantifying behaviour, mind and society place high demands on raters who are given a tool that does not allow them to fulfill this complex task. Instead, raters use mental shortcuts and focus on concrete examples that they can remember to make their judgements.

Featured Image

Why is it important?

When answering rating scales, raters have to make sense of the question at hand and come up with concrete examples to which they refer to make their judgement. The article shows that there are pronounced differences among people in the ways in which they do that. In fact, they often think about very different things when answering the same question. But scientists commonly ignore this diversity - in fact, they hardly ever ask! Strikingly, most raters do not have any quantitative ideas in mind when ticking rating scales - and yet researchers convert these ticks into real numbers and then treat them as scientific quantifications.

Perspectives