The problems of fit indices on replicated SEM studies

Abstract

There has been a research gap in examining fit indices under the context of reproducing the result of structural equation modeling (SEM) since a replication attempt revisited not many SEM studies. Two simulation studies were conducted to examine the distribution of fit indices of SEM on replicated samples. The first simulation chose three examples from social science literature to mimic replication attempts and found that the distribution of some indices shifted away from the original value. Specifically, the fit indices that use chi-square in their formulation consistently indicated a worse fit between the model and the data in a large proportion of replication attempts. Meanwhile, relative fit indices that use log-likelihood values such as AIC and BIC were less affected by replication, showing the distribution of replicated indices centered around the value from the original sample. The chi-squared family of fit indices showed an inferior fit than the original when one tries to replicate data using the observed moment matrix, even if the model fitted well to the original data. Using a baseline model log-likelihood, a new likelihood ratio LR0 that resists the fit-worsening effect of replications is suggested. The second simulation that varied model specification, model complexity, and sample size confirmed the finding from the first study and examined the performance of the LR0. The new likelihood ratio was much less affected by replication than the standard likelihood ratio, but its interpretability was limited. The diminishing effect on fit indices in replicated samples implies that one should interpret them carefully.