What's wrong with rating scales? A largely overlooked source of psychology's replication problems

What is it about?

Rating scales are popular methods to produce quantitative data. What seems simple, however, is in fact quite complex. This article explores the mental, empirical and methodological processes that are involved in rating methods in the side of both raters and researchers. These processes remain largely hidden and are therefore still hardly considered. Rating methods are convenient to use because they build on our language abilities—we can all understand the questions in a survey and don't need any training of how to use them. Given this, we tend to believe that we all understand them in the same way. But we don't. There are pronounced individual differences in how people understand the items in a questionnaire. Different people therefore have, for teh same item, different ideas in mind. Hence, we cannot know what specific raters actually considered in a given rating. This makes their responses incomparable. We also do not know how raters make their judgments and why they tick a particular box on the rating scale—why "agree" but not "strongly agree"? All this entails that we cannot trace the answers back to what raters actually aimed to express in their ratings. When researchers then assign numerical scores to the raters' responses, they assume these scores to have quantitative meaning. But in fact, raters' reasons for choosing specific answer boxes are often quite trivial and mostly not quantitative at all. All this differs fundamentally from measurement and seriously compromises the scientific value of rating data. The article summarises the main points in which rating scales differ from measurement instruments. It highlights many methodological problems with rating scales that are still largely overlooked and that constitute an important source of psychology's replication problems. Tackling these problems requires more rigorous methods for generating quantitative data in psychology, which are grounded in the principles of genuine measurement.

Featured Image

Photo by Олег Мороз on Unsplash

Photo by Олег Мороз on Unsplash

Why is it important?

Rating scales are widely used to study individuals. They are also used to make judgements and decisions about individuals (e.g., occupational assessments). The fundamental problems in their underlying rationales, however, cast serious doubts on their legitimacy for these purposes.

Perspectives