Revealing and resolving contradictory ways to reduce selection bias to enhance the validity of causal inferences from non-randomized longitudinal data

Abstract

The purpose of the current study is to understand the mechanism of selection bias in nonrandomized studies by exploring possible reasons for inconsistent and biased results between the simple gain scores and the residual gain scores from the human development perspective. Specifically, I test several possible reasons that lead to Lord's paradox (contradictory results from the two approaches), and test alternative models for solving the paradox to get consistent and hopefully less biased results using simulated data and data on treatments for mothers' depression from the Fragile Families (FFCW) longitudinal dataset. Monte Carlo simulations, based on Lord's paradox and reversed Lord's paradox, generated 97 conditions of 1000 simulated datasets by varying violations of assumptions of ANCOVA and simple gain score analyses. The FFCW data include two types of treatments at Wave 4 and mothers' depression severity at Wave 2, 3, 4, and 5. Seventeen covariates measured before treatment were used for propensity score matching. An alternative model, group-centered ANCOVA, was developed to compare results to another alternative model, a combination of mixture model and propensity score matching. The results indicated that Lord's paradox exists when the mean pretest score differs for the treatment group and the comparison group, and consistent results can be reached when that mean pretest difference is removed. Moreover, group-centered ANCOVA approach and the combination of mixture modeling and matching on either the pretest or the propensity score could be used to remove the pretest difference between the treatment group and comparison group. These two methods of removing the mean pretest difference between the two treatment conditions result in two distinct sets of consistent results, which are nearly as inconsistent with each other as the original inconsistency. Consistent results do not guarantee an unbiased result. This study helps to understand alternative ways to adjust for selection bias in nonrandomized studies. Under some conditions models based on simple gain scores may be less biased than models based on residual gain scores. When simple gain score are less biased, group-centered ANCOVA provides more statistical power than traditional analyses and other covariates can be added to minimize other suspected confounds.