Testing predicted variable clusters by comparing them with their optimized counterparts

What is it about?

Items of questionnaires about behavior, feelings and thoughts that represent psychopathological phenomena or personality characteristics should cluster in a characteristic way. If they do, these clusters will reveal psychopathological disorders, or subcategories within these disorders; or they represent personality traits. If one is able to predict the clustering, then testing the predicted cluster structure will be a rich source of feedback on the theory behind the prediction as well as on the construct validity of the test. To judge whether the prediction of the item clusters has been adequate, the most direct way would be to optimize them until they are in line with the empirical correlation matrix over the items. Then, the predicted clusters can be compared with the empirical ones in detail and the prediction of the cluster correlations can be tested. Investigating clustering between a great many variables is mostly done by means of factor analytic methods. So the terms factor loadings and factor structure will be used below. A direct way op optimizing factors is Procustes rotation. An indirect way is to perform confirmatory factor analysis (CFA) or - preferably - exploratory structural equation modeling (ESEM), and then optimize the tested model on the basis of the modification indices. Once the predicted factors have been optimized, they can be compared with the final ones. Feedback can be derived from looking at the factor loadings: Which items were unjustly assigned to the factor under inspection? Which items were unjustly not assigned to the factor now under inspection? Does the content of both correctly and incorrectly allocated items, judged in connection, demand a (slightly) different factor interpretation? Do the factors correlate as expected? Were high cross-correlations of items foreseen; did predicted cross-correlations not come true? If not, are the deviations serious, and if they are, do they falsify the theory or construct validity, or can they be interpreted differently? Contrasting predicted with optimized clusters (factors) thus provides a basis for classifying the items into hits, false positives and false negatives. Not only does this division greatly facilitates an evaluation of the items and reinterpretation of the factors, as we saw above, in addition it can be used to express the degree in which one’s predictions have been accurate (goodness of fit) in a value between 1 (perfect fit) and 0 (fit not better than random assignment). To illustrate this approach of testing clusters (factors) it was applied to a questionnaire on obsessive-compulsive disorder. Some readers may find this an additional reason for interest in the paper.

Featured Image

Why is it important?

The method that is most frequently used to test the goodness of fit of predicted factors is CFA. However, this provides only a global evaluation of the model in its predicted form, affected by various types of prediction errors simultaneously: in item allocation to factors, in factor correlations, in cross-loadings of items, and some others. The same holds for ESEM (Asparouhov & Muthén, 2009), although this has the advantage of providing secondary factor loadings. However, both methods have the option of a “data-driven re-specification of the predicted model”, in other words, optimizing predicted factors. Normally, such a re-specification is never continued to the end because that would run the risk of “capitalization on chance”, i.e., being unacceptably affected by imperfections in sampling and measuring. However, here capitalization on chance is not at stake because the optimized factors merely serve as sources of feedback. This feedback will lead to a revision or refinement of one’s theory, which - in turn - may lead to a new factor prediction, which need not be a servile copy of the optimized factors.

Perspectives