Internal and External Validity: Two Faces of the Same Coin

Abstract

Internal and external validities have been traditionally presented as two different topics. The present paper brings them together by showing how the two are related when the samples have been previously selected on a covariate. Selected samples are common in the social sciences. We show in a computer simulation that the ANOVA fails both in terms of external and internal validities when performed in a selected sample. On the other hand, when we use an ANCOVA in a randomized experiment the ANCOVA retains both internal validity as well as external validity. The simulation was used to compare the performance of ANOVA and ANCOVA in random and nonrandom (selected) samples. When the covariates that were related to the selection process were included in the ANCOVA, the ANCOVA outperformed the ANOVA in many of the situations studied. When a covariate-by-treatment interaction was present, the treatment effects were overestimated by the ANOVAs and underestimated by the ANCOVAs that incorporated the interaction. In addition, with selected samples, the covariate-by-treatment interaction was hard to detect due to the lack of power. Researchers working with selected samples should conduct an ANCOVA using covariates that are related to both selection and the dependent variable.