Refinement of a measure of data use practices

Abstract

The purpose of this study is to refine a measure of data use in early childhood schools, focusing on one section of a set of measures used to capture data use by teachers in schools serving children from birth to age 5. Data are defined as information collected purposefully by teachers or by others. Primary examples of data in early childhood settings include child assessment data, data on classroom quality, and data on families. A framework applied to refine items included learning from data at a low level to responding based on data at a high level. Items were developed to coincide with these levels.

Items were field tested through cognitive interviews and behavioral coding, and presented to panels of experts. These processes informed a version that was collected in the spring of 2019 with a sample of 359 teachers. The sample included early childhood teachers employed by a national network of early childhood programs that emphasize the use of data, early childhood teachers involved in similar evaluations, and teachers recruited from social networks. The purpose of this study was to review the psychometric qualities of six scales using factor analysis and item response theory (IRT) applications.

Results/conclusions: The study found that five of the six scales of the Informational Use set had acceptable psychometric qualities. Scales measuring the use of teacher-collected child assessment data and teacher-collected classroom quality data had items that had high estimates of discrimination and measured well across the latent spectrum. These are considered sufficient for use in research and professional development. Scales focused on other-collected child assessment data and other-collected classroom quality data had sufficient discrimination and bandwidth, but are considered primarily useful for descriptive purposes due to item fit issues that need additional study. One of the two Family scales is also sufficient for descriptive use. However, teacher-collected family data did not have sufficient evidence to recommend use as a scale. Model fit statistics had mixed results for all but one scale — the teacher-collected classroom data scale.