| dc.description.abstract | The purpose of this program evaluation is to provide the Center for Teaching at Vanderbilt University with a comprehensive evaluation of their Teaching Certificate program, which is in its third year of existence. As a joint project between the Graduate School and the Center for Teaching, Vanderbilt's Teaching Certificate program aims "to help graduate students, professional students, and post-doctoral fellows develop and refine their teaching skills through three cycles of teaching activities, each consisting of inquiry, experimentation, and reflection phases." (Vanderbilt University) Much like other teacher preparation programs described in the body of this document, the Teaching Certificate program combines workshops, teaching observation and feedback experiences, reading groups, a literature review, and reflective essays to achieve its outcomes. However, one element that sets the Vanderbilt program apart from similar programs is the required project that highlights the Scholarship of Teaching and Learning (SoTL). The three cycles of inquiry, experimentation, and reflection increasingly emphasize teaching as a scholarly activity as defined by Boyer as one of four domains of scholarship. Specifically this evaluation seeks to answer three questions. The questions are:
1. What do participants learn in the program, including knowledge, skills, and attitudes?
2. How do they apply what they learn when teaching at Vanderbilt or in faculty positions obtained after leaving Vanderbilt?
3. What knowledge, skills, and attitudes regarding teaching do Vanderbilt departments and programs want their doctoral students to possess upon graduation?
The deliverable for this project is an assessment of student learning and of the program's strengths and weaknesses in order to give the Center for Teaching useful information to improve the program and thereby improve the experience for the participants. This project consists of two phases: participant analysis and stakeholder analysis. The participant analysis stage primarily addresses the first two questions stated above and focuses on documents and interviews with actual program participants, both those currently in the program as well as the few who had completed all requirements. During the stakeholder analysis phase, external stakeholders defined as Directors of Graduate Study at Vanderbilt were interviewed in order to identify skills, abilities, and attitudes that they deem as important for their graduate students. This phase of the evaluation specifically addresses the third question stated above. Questions were used that elicited information about stakeholders' perceptions of the value of teaching preparation for their students as well as the departments' actual efforts or lack thereof in preparing their graduate students for teaching responsibilities which they may encounter as a faculty member.
For the participant phase, the investigators created an evaluation rubric in order to examine program documents. This rubric operationalized four of six stated program objectives. The remaining two program objectives not evaluated rely on "end-of-pipeline" analysis of participant performance once they have obtained full-time employment after graduation. Given that there are very few program finishers who have graduated from the university and moved into faculty roles, these two program objectives were not assessed. The rubric created to assess the four program objectives employed a 5-point scale and was used to gauge the acquisition of knowledge, skills and attitudes of teaching as a scholarly activity. This quantitative approach allowed the investigators to assess the magnitude of knowledge gained regarding the four program objectives being evaluated. Based on the results of the document analysis a common interview protocol was developed in consultation with the Center for Teaching in order to extract more information than was obtained from the document analysis. This qualitative approach sampled participants from each of the three cycles and interviewed them in order to establish how participant content knowledge increased throughout the program. The second phase of this evaluation, the stakeholder analysis, sampled Directors of Graduate Study from across the campus and interviewed them using a common interview protocol in order to identify expectations and attitudes of graduate preparation for teaching.
It is important to note that this program evaluation contains important limitations that stem from the lack of program finishers, the lack of operationalized objectives, and the open-ended electronic portfolio reporting system. In addition, there were inconsistencies in documentation from one participant to another which probably impacted the level of reliability with the outcomes.
Three of the four objectives evaluated in the quantitative analysis suggest an increase in knowledge, skills, and attitudes of participants' learning with regard to the following: undergraduate learning, analysis of their own teaching, and engagement with their own teaching in a community of scholars. With the fourth objective, which is primarily a Cycle 3 activity, participants showed no significant difference at the end of Cycle 2 in approaching their own teaching as a scholarly activity. Based on these results, further information needed to be acquired by way of qualitative analysis to determine if the initial results were an accurate representation of the participants change. Based on results from the participant interviews, graduate students appear to fulfill the four program objectives evaluated and are able to approach their teaching as a scholarly activity and learn from their own teaching and from others' teaching. Participants self-report that they have gained knowledge and skills from their participation in the program and the analysis of data demonstrates an increase in knowledge, skills, and attitudes as they relate to teaching as a scholarly activity.
From interviews with the stakeholders, there is sufficient evidence of tension that exists between research and teaching at Vanderbilt. Departmental activities on training graduate students to teach vary widely but conform somewhat to trends in high and low consensus fields. The applied and natural sciences and some social sciences tend to focus more training on research skills while many humanities areas devote more resources to teaching in addition to that of research skills. However, many departments base their success on student placement after graduation with a high desire to be at research-intensive universities, even though many departments see their students at teaching-intensive institutions or in industry.
In conclusion, the investigators determined that there was a substantial increase in knowledge, skills, and attitudes in the scholarship of teaching and learning by program participants. Participant experience tends to vary widely depending on the department attitudes regarding teaching as evidenced from participant interviews and DGS interviews. The role of the DGS tends to be marginalized in many departments as a service duty rather than a professional role dedicated to strengthening graduate student education in research and teaching. As a result, it is important that the Center for Teaching be relied upon to fulfill the need for training in pedagogy in order to fully prepare doctoral students for professional employment. In addition, teaching opportunities should be increased in many departments in order to provide graduate students with more substantive experiences in teaching to enhance their profile as they seek professional employment. In comparison to other similar programs across the country, the Teaching Certificate program is on a positive trajectory to establish best practices in educating and evaluating teaching as a scholarly activity for the higher education community at large. A concern, however, is that of data management, which is important for quality evaluation of participant learning. Using operationalized objectives, variables should be evaluated on the basis of how participants are gaining knowledge, skills, and attitudes throughout the program. Inadequate data management threatens the significance of assessment in this program. Recommendations resulting from this program evaluation are listed below.
1. Program objectives should be operationalized in order to provide consistent evaluation of the increase in participant knowledge, skills, and attitudes.
2. When prompting participant reflection, the use of leading questions in the portfolio allows for more consistent reporting of outcomes in the various program cycles, which leads to greater validity when evaluating participant performance.
3. Possibly have participants revise their teaching statements more regularly than just at the end of the program, which should integrate their statements with every teaching activity undertaken.
4. Participants seem to enjoy a great deal of structure in the schedules, thus the use of soft deadlines or typical times to complete tasks can aid in efficient time management.
5. It is critical to effectively track the progress through the program in order to measure gains effectively, thus having students regularly self-report progress ensures accurate record keeping.
6. The portfolio system is clearly critical to evaluate participant progress, which justifies having a simple yet sophisticated system to handle self-reporting, tracking, and evaluation of participant activities.
7. Stakeholders are important to the continued success of the program and key faculty should be identified and approached as supporters of the program. Additionally, DGS's deemed potential supporters should be well-educated on the program in an effort to continue to have a stream of applicants who become participants.
8. A possibility could be to modify the participants' academic transcripts to note this significant accomplishment and to add credibility to the program and its participants with regard to SoTL.
9. The CFT should spearhead a concerted effort in partnership with the Graduate School to integrate teaching into the overall graduate student experience considering that so many end up in teaching positions.
10. The population of post-doctoral fellows is increasing and becomes an area of interest for gaining program participants, thus marketing efforts should be increased to this demographic.
11. A major benefit of this program is the 'high touch' approach to participant activities and this high level of service should be continued.
12. More consistent program evaluation is important to maintaining this important and critical program to the graduate student experience at Vanderbilt. The next formal evaluation should occur when more participants finish the program and gain full-time faculty employment in order to evaluate the two program objectives not assessed in this study.
The program has a strong foundation on which to build, and the ongoing efforts of the Center for Teaching staff to improve the program will no doubt make it a leader in its field and a model program for other institutions to emulate. | en_US |
