Analysis of Process Data in Educational Digital Assessments: New Similarity Measures and Outcome Analysis

Abstract

Process data can provide researchers with valuable insights into students’ behaviors during reading and while completing assessments. However, the complex and messy nature of process data makes the extraction of useful information from these data challenging. One approach that has been applied to process data is the comparison of process data using similarity measures, from which groups of similarly-performing students can be obtained. The current study presents two similarity measures developed to analyze educational process data with distinct properties. First, a time-series action-sequence similarity measure was developed to compare student behaviors during a digitally-based NAEP mathematics assessment. Clusters of students were created and interpreted based on this similarity measure and were employed as covariates in additive latent regression models to investigate their association with math ability and to demonstrate linear and nonlinear interactions between group membership and continuous covariates, such as response time, with the latent trait. This research revealed that a significant portion of item-level clusters were interpretable regarding various test-taking behaviors. Additionally, two item-level and all test-level cluster membership covariates were significantly associated with math ability, and the significance and nonlinearity of the effect of response time on math ability differed by cluster membership and accommodation status. Second, a scanpath trajectory spatio-temporal similarity measure for comparing students’ reading-process eye-tracking data, called STRESS, was developed to integrate the strengths of (a) scanpath similarity measures which involve aligning eye-tracking data on areas of interest and (b) trajectory similarity measures which can account for the dynamic spatio-temporal features of eye-tracking data. The validity of this STRESS measure was demonstrated through both qualitative analyses and a simulation study. R code for calculating both of these similarity measures is provided.