Evaluating Math Recovery: The Impact of Implementation Fidelity on Student Outcomes

Abstract

In this dissertation, I report an analysis of the relationship between student outcomes and fidelity of implementation of Math Recovery, an unscripted, pullout, tutoring program intended to increase the mathematics achievement of low-performing first graders. The work I describe was conducted as part of a larger evaluation study of the intervention. Two research questions guided the conduct and analysis of the larger evaluation study: 1) Does participation in Math Recovery raise the mathematics achievement of low performing first-grade students? 2) If so, do participating students maintain the gains made in first grade through the end of second grade? The work documented in this report was driven by a third question: 3) What is the relation between fidelity of implementation of the program and student outcomes at the end of the school year in which students received tutoring? I detail the process of developing and testing instruments for assessing implementation fidelity of Math Recovery. In doing so, I provide a concrete account of how recent conceptions and standards of fidelity assessment were applied to the FOI assessment of MR, including (a) identifying the intervention’s program theory and core components; (b) creating operational definitions of the intervention’s core components; (c) developing coding instruments; (d) hiring and training coders; (e) instituting a sampling frame sufficient for generalizing fidelity findings to the study population; and (f) determining the reliability and validity of the instruments. In addition to reporting the findings of my analyses, I provide a discussion of key aspects of the work that were particularly helpful in assessing implementation fidelity of Math Recovery in order to illustrate how these steps might be accomplished in fidelity studies of other unscripted interventions.