An examination of the validity of the VaNTH Observation System (VOS)

Abstract

This dissertation reports results of five studies designed to assess the validity of the VaNTH Observation System (VOS), which was developed to assess the implementation of “How People Learn” (HPL) (Bransford, Brown, & Cocking, 1999) framework innovations within the VaNTH Engineering Research Center. The content, convergent, and criterion-related validity of the Classroom Interaction Observation (CIO) and the content validity of the Global Ratings (GR) portions of the VOS were examined. For the CIO content validity study, the percent agreement between eleven educational experts’ and VOS observers’ classification of four individual HPL dimensions within twenty vignettes ranged from 68.2% to 78.6% at the least stringent level of analysis and was lower (25%) for HPL dimension combinations embedded in the vignettes. The GR content validity study revealed that content coverage is incomplete within the current GR. Data from the same eleven experts suggested that thirteen GR indicators could be grouped into subscales representing (1) knowledge-centeredness (Alpha = 0.59), (2) knowledge- and learner-centeredness (Alpha = 0.75), (3) knowledge-, learner-, and assessment-centeredness (Alpha = 0.91), and (4) learner- and assessment-centeredness (Alpha = 0.74). Two studies assessed the convergent validity of alternative methods of scoring and gathering CIO-based data. In the first convergent validity study, an HPL Index that used entire CIO code strings to calculate the amount of HPL instruction in classes was developed. The results from the Index were compared to the results derived by summing the percentage of instruction devoted to the four HPL dimensions (i.e., the current CIO assessment method). The comparison revealed that both indices index the degree of “HPLness” in a class (r = 0.79). The HPL Index, however, provides a more meaningful summary of the amount of HPL instruction in a class than the current method. In the second convergent validity study, an alternate CIO data gathering method showed similar instructional profiles, suggesting that the current method of data gathering accurately reflects the amount and type of interactions that transpire in a class. The final study confirms that the HPL Index distinguishes pedagogy in known HPL and non-HPL (traditional) courses. Implications of the findings are discussed.