Using AI language models with cloze tests to better understand human society and culture

What is it about?

Trained on what people naturally say and write in the real world, AI language models have learned and inherited human knowledge, thoughts, and feelings. With a proper method to extract such deep information, AI language models can be used as a measurement tool to study human mind and social psychology at the group level. My research here proposes and validates the Fill-Mask Association Test (FMAT) as a new method for more precisely and flexibly measuring social reality, social cognition, social bias, and social change. Valid and reliable, the FMAT contributes to an integrative framework for studying a broad spectrum of human psychological, cognitive, social, cultural, and historical phenomena in text and natural language. An R package “FMAT” has also been developed along with this article to simplify and standardize its workflow for easier use on many other questions.

Featured Image

Photo by Alexander Sinn on Unsplash

Photo by Alexander Sinn on Unsplash

Why is it important?

Earlier big-data text-analytic methods can measure concept popularity by counting words and capture conceptual association by quantifying semantic similarity between words. However, words without any contextual or relational information cannot tap into deeper and more complex theoretical constructs. The new FMAT method (1) adopts the propositional perspective on attitudes and social cognition, (2) leverages the propositional reasoning capability of BERT language models, and (3) allows for more concrete and fine-grained measurement of specific relations that can better represent theoretical constructs. With the FMAT, we can study social psychology at the group level in a more naturalistic, intelligent, and contextualized way. Overall, the FMAT can open up a new interdisciplinary field—computational intelligent social psychology.

Perspectives