Construct Transparency and the Psychometric Properties of a Multidimensional Situational Judgment Test

Abstract

Reported internal consistencies of SJTs are often low, and empirical examination of the structural properties of SJT scores are rarely examined. This paper addressed the need for an empirical investigation of factors that affect the validity evidence of SJT scores by examining the extent to which construct transparency influenced the internal structure of a multidimensional SJT. Methods for increasing construct transparency that have previously been applied in the assessment center literature were adapted for use in the present study. Two conditions—experimental versus control—were used to investigate the influence of construct transparency on a recently developed multidimensional SJT designed to measure multiple sensemaking skills important to leadership. Various psychometric properties were examined and compared for the two conditions using a sample of 383 participants. The study found that there were no significant differences between the two conditions for either estimates of scale intercorrelations or scale reliabilities, and the patterns of correlations with external scales were very similar. Participants in the construct transparency condition received higher scores on all scales compared to participants in the control condition, and evidence of structural validity slightly favored the scores of the construct transparency condition. However, half of the scales lacked supportive validity evidence. The results of this study are discussed in terms of the extent to which multidimensional SJTs of sensemaking skills may be dependent upon case-based knowledge and experience as well as the usefulness of making the content of multidimensional SJTs transparent to test takers.