Psychology's Questionable Research Fundamentals (QRFs): Key problems beyond questionable practices

What is it about?

Psychology's crises in the replicability, generalisability and validity of its findings are currently believed to derive from Questionable Research Practices (QRPs). These involve hypothesising after the results are known (HARKing), analysing data relentlessly to obtain statistically significant results that support the researchers' hypotheses (p-hacking) or testing statistical associations of randomly combined variables without any theoretical hypotheses (fishing) and other questionable practices. For the meticulous method expert, these flaws are readily identifiable, as are their remedies—larger samples, more robust statistics, more data transparency (open science, preregistration). Thus, do psychology's crises arise just because psychologists are more prone to scientific misconduct than scholars in other disciplines? In our view, questionable research practices are just surface-level symptoms that distract from and obscure the root causes of psychology's crises—the Questionable Research Fundamentals (QRFs) of many of its theories, concepts, approaches and methods. In this article, we critically explore key problems of psychological measurement from independent perspectives involving various fields of expertise and lines of research that are well established but still hardly known in mainstream psychology. Our analyses are grouped into four main areas that cover different topics, problems and research questions and that require remediation, elaboration and further development. * Topic 1 Advancing quantitative psychology as a science Lucas Mazur reflects on psychology's struggle with its scientific status and on the problems, promises and perils of scientism. Aaro Toomela elaborates on what science actually is and the necessity to advance psychology's ontology, epistemology and methodology and to align them coherently to one another. Jack Martin reminds us of the inherent contextuality of human experience that makes up personhood and draws conclusions for quantitative and experimental psychology. * Topic 2 Psychometrics and psychological ‘measurement' and their fundamental differences to genuine analogues of physical measurement Jana Uher explores the conceptual problems of psychology's operationalist definition of ‘measurement' and quantitative data generation with rating ‘scales', and highlights incompatibilities in the epistemological framework on which psychometrics is built. Jörg-Henrik Heine and Moritz Heene locate the failed promises of psychological ‘measurement' in the impossibility to establish one–to–one relations between the phenomenological object domain and the mathematical metric space of positive real numbers. Paul Barrett concurs that, without meeting the axioms of quantity and the human mind's peculiarities, quantitative psychology cannot implement genuine measurement processes. Robert Mislevy uses a contextualised socio-cognitive approach to re-conceptualise the theoretical and philosophical framework that is necessary for making justified inferences from quantitative educational assessments in applied settings, while avoiding conceptual errors inherent in current conceptions. Jana Uher demonstrates that statistics and measurement are different scientific activities designed for different epistemic purposes. She specifies basic criteria and methodological principles that are epistemically necessary for establishing genuine analogues of measurement in psychology. * Topic 3 Psychologists' study phenomena and their means for investigating these phenomena (language, constructs) Jana Uher highlights the necessity to logically distinguish between the study phenomena (e.g., participants' beliefs) and the means used for their exploration (e.g., descriptions of beliefs in items). This requires an increased awareness of the complexities of human language (e.g., inbuilt semantics) and how these complicate scientific inquiry. Jan Ketil Arnulf demands a more critical reflection on the role of human language in scientific inquiry. He demonstrates that the inbuilt semantics of item statements, analysed through natural language algorithms, produces results similar to those obtained from empirical rating studies. Ron Weber analyses the ontology of construct–indicator and indicator–instrument relations and introduces novel ontological concepts to analyse the applicability of constructs and their operationalisations (indicators) to different subsamples of populations, highlighting their implications for instrument development. Topic 4 Generalising findings across unique individuals Jana Uher argues that psychology's default use of sample-level statistics to explore individual-level phenomena ignores the mathematical-statistical foundations of such inferences (ergodic theory), the non-ergodicity of psychology’s study phenomena and the peculiarities of complex living systems. Craig Speelman and Marek McGann highlight that the common sample-to-individual inferences build on the ergodic fallacy, thereby contributing to psychology's inferential and reproducibility problems, and they present pervasiveness analysis as an alternative approach. Jana Uher shows that, to avoid fallacies when making sample-to-individual inferences, psychology must advance profile-based (not group-based) approaches, implemented through individual-/person-oriented (not variable-oriented) analyses. This is essential for identifying actual commonalities and differences among individuals as well as for enabling causal analyses to unravel (possibly) underlying structures and processes. * Conclusions Just minimising scientific misconduct, as currently believed, and exploiting language-based algorithms (NLP, LLMs) without considering the intricacies of human language will only perpetuate psychology's crises. Instead, tackling psychology's Questionable Research Fundamentals (QRFs) requires critical self-reflection and a fundamental rethinking of doing science in psychology.

Featured Image

Photo by Iñaki del Olmo on Unsplash

Photo by Iñaki del Olmo on Unsplash

Why is it important?

Rather than trying to reinvent the wheel, mainstream psychology can and should capitalise on the advances already made over the last decades from different perspectives and fields of expertise. With our compilation of diverse perspectives on quantitative psychology's problems, we aim to set an example, to give new impetus to the current debates and to highlight important directions of future development that, as we believe, are necessary to rethink and advance psychology as a science.

Perspectives