Comparisons and recommendations for statistically pooling data from multiple cross-sectional studies

What is it about?

Integrative data analysis (IDA) is an alternative to meta-analysis that combines participant-level data from multiple studies for cumulative analysis. Two approaches, fixed effects models (FEM) and multilevel models MLM), have been used in psychological applications of IDA but have not been fully evaluated. Because IDA combines data from multiple studies, two different kinds of fixed effects can be studied in IDA: study-level and participant-level effects. Furthermore, between-study differences need to be modeled carefully. For IDA with cross-sectional data, we reviewed three FEMs and two MLMs and theoretically discussed whether and how they can estimate and test participant-level and study-level fixed effects with sufficient data. We also evaluated the performance of these models and different MLM estimation methods in a simulation study under realistic IDA conditions (e.g., fewer than 30 studies). Although two of the FEMs accurately estimate the fixed effects, they do not model between-study differences in participant-level effects, leading to incorrect inferences. A random-slopes MLM that accounts for differences in both study means and participant-level effects provided accurate inferences and estimates of the fixed effects and between-study differences. We found that MLMs can be feasible for IDA with as few as three to six studies using appropriate estimation methods. We illustrated the application of the five models and how they can provide different estimates and inferences in an empirical example. We conclude with recommendations to guide researchers when planning an IDA.

Featured Image

Photo by Clay Banks on Unsplash

Photo by Clay Banks on Unsplash

Why is it important?

While integrative data analysis of multiple studies is growing in popularity, there has been limited attention paid to statistical modeling approaches and their validity in this context. Given recent concerns regarding the so-called "replication crisis" in the social sciences, the use of multiple data sets in an appropriate analysis is a timely and important means to make more generalizable and well-powered inferences within psychological research. We provide missing statistical recommendations for appropriate modeling approaches for integrative data analysis.