Characteristics of biodata keys as a function of scaling method, sample size, and criterion.

Abstract

The present study investigated the impact of scaling method, key developmental sample size, and criterion characteristics on the incremental validity of scored biographical information (biodata) as a predictor of subsequent performance. Biodata surveys administered to air traffic control students at the FAA Academy between 1988 and 1990 were used to generate data that was scaled using three different measurement technologies: (a) vertical percent empirical keying, (b) factor analysis, and (c) multidimensional item response theory (MIRT). The resulting biodata scale scores were combined with an ability-based selection test to predict FAA Academy performance (defined as final score and pass/fail) and air traffic field facility OJT performance (defined as certification or "wash out"). Comparisons were made between each method, for samples ranging in size from 4,568 to 100 students, regarding the amount of shrinkage between developmental and cross-validation groups, magnitudes of resulting cross-validities, and classification accuracy. The results indicated that at large sample sizes (N ≥ 1,000) the differences between methods regarding the amount of shrinkage and the magnitude of cross-validity coefficients were negligible. At smaller sample sizes the amount of shrinkage increased with the empirical key enhanced model demonstrating a greater amount than the factor analytically derived keying model. Though differences were small, the incremental factor analytic model appeared to predict eventual performance, in most of the samples, better than the empirical or MIRT methods. This study contributes to the comparative scaling biodata literature by providing guidelines for future research and practice. Another outcome is exposure of MIRT methods to the biodata research community.