2019 Impact and Outcome Measures
MSU faculty are currently engaged in case study research as a strategy to assess the effectiveness of completers in the classroom and impact on P-12 student learning. A multiple case study was conducted during the 2017-2018 academic year to assess completer outcomes 2.5 years after completing an Educator Preparation Program (EPP). Case study was used as an alternative to value-added measures (unavailable at the state level) to holistically examine complex attributes of quality teaching (Merriam, 1998; Yin, 2014). The Reflective Experiential Teacher framework (Mayville State University Teacher Education Handbook, 2018) was used to design methods and guide analysis. The constructivist framework incorporates the knowledge, skills, and dispositions of the ten core teaching standards of the Interstate Teacher Assessment and Support Consortium (InTASC) (CCSSO, 2013) divided into the four CAEP Standard 1 categories of: Learner and Learning, Content, Instructional Practice, and Professional Responsibility. The categories “provide a scheme for the valid evaluation of any teacher’s core competencies and the reliable means of recording and compiling overall teacher performance” (ND DPI, 2015, p. 4).
This case study was part of a larger investigation of the role EPPs play in graduates’ teaching effectiveness, among a myriad of other complex factors, and the role graduates effectiveness has on student learning. Data was collected to answer the questions, “Do P-12 students in the classrooms of TPP completers demonstrate expected levels of learning and development?”, What factors do graduates perceive impact students’ learning and development?”, and “How can a TPP improve program quality based on results of P-12 student learning in classrooms in which graduates are teaching?”.
Mixed methods data collection in this descriptive study included: graduate and supervisor surveys, self and supervisor evaluation of skills and dispositions, interviews, document review, student engagement surveys, achievement data, and pre and post assessment data. Descriptive statistics were used to describe data for each case. Within each case, the constant comparative method of qualitative data analysis was used (Glaser & Strauss, 1967) to construct codes, categories, subcategories or themes through continuous comparison of data (Merriam, 1998, p. 159).
Results indicated a majority of students from participants’ classrooms demonstrated learning growth, and participants viewed their teaching as effective. Implications for programs include pre-service opportunities to identify and respond to authentic student engagement, and opportunities to build collective teacher efficacy. Additional insights describe pre-service teacher training to measure student progress and measure impact based on classroom assessments. The investigation provides a replicable case study design for teacher educators to examine relationships between teacher preparation, program graduates, and P-12 student outcomes.
In addition, state data on the effectiveness on student outcomes of MSU graduates compared to all ND teachers is also available. SLDS data is statewide and longitudinal. However, only test data from grades 4-8 resulted in a small N, and numbers cannot be disaggregated by area of preparation. There is also a small student N as the K-12 students had to take the NDSA in math for two consecutive years. There is a causal leap between student achievement and teacher effectiveness evident in presentation of the results; the testing results are one test on one day. Curricular choices/decisions compared to NDSA test are unknown. Additionally, only reading and math scores are available, and only reading growth distributions were provided, but not proficiency levels. Overall the information does little to inform EPP program improvement.
MATH Growth %ile Distribution: Students of MSU first year teachers (n = 41) included in the report, demonstrated a growth percentile distribution in Math in the 57th percentile over the 2015-2016 and 2016-2017 school years. The range was 28-76. This distribution was higher than all ND non-first year teachers (50th percentile) and other ND first year teachers (48th percentile).
MATH Proficiency Levels: Proficiency levels of students of MSU first year teachers (n = 41) included in the report, declined from 34.1% of students proficient or advanced to 26.8% proficient or advanced. This trend is consistent of students on non-first year teachers in North Dakota (44.6% to 39.0%).
READING Growth %ile Distribution: Students of MSU first year teachers (n = 41) included in the report, demonstrated a growth percentile distribution in Math in the 57th percentile over the 2015-2016 and 2016-2017 school years. The range was 28-78. This distribution was higher than all ND non-first year teachers (50th percentile) and other ND first year teachers (46.5 percentile).
Prepared manuscripts are under review for publication and are available for stakeholder review upon request.
Anderson, S.K., Hagen, B., Whitsel, C., Dulski-Bucholz, A., Smith, K., & Willeson, A. (2019). Leveraging case study research: A mechanism for measuring teaching effectiveness. Mid-Western Educational Researcher pp. 1-43. (Accepted)
Anderson, S.K., Hagen, B., Whitsel. C., Smith, K.D., & Duffield, S. (2019). Graduate impact on student learning: A descriptive case study. (Under Peer Review)
3. Satisfaction of employers and employment milestones (Component 4.3)
Findings on employer satisfaction are collected from surveys administered to supervisors of first-year MSU teachers during the spring of 2018. The supervisor survey asks those who supervise first-year teachers to assess the novices’ readiness for the teaching profession. The survey asks supervisors to assess the quality of graduates’ instructional practices, abilities to work with diverse learners, abilities to establish positive classroom environment, and levels of professionalism. The survey is administered to direct supervisors of teacher preparation graduates employed in schools approximately one year after the teachers completed their preparation programs.
The Division of Education has set an acceptable target of a 2 or higher mean on all indicators (all tends to agree or agree); employers are satisfied with the completers’ preparation for their assigned responsibilities in working with P-12 students as indicated with a tendency to agree. An ideal target has been set at a 3 or higher mean on all indicators (all agree). Results of the survey are reviewed annual by the Division of Education Faculty.
2017-2018 Summary of Findings: Overall, all acceptable targets were met with ratings above 2.0 in three categories (Instructional Practice, Learning Environment, and Professionalism). Supervisor feedback indicated candidates were ready for the responsibilities they encounter on the job. There was one supervisor who marked tends to disagree for multiple items related to one completer.
Four supervisors indicated they were not able to observe the ability to differentiate instruction for ELL. Most ideal targets were met with a rating of 3.0 or higher (for Instructional Practice, Learning Environment, and Professionalism).
Response rate for supervisors declined this year from 71% in Spring 2017 to 42% in Spring 2018. It is unclear why this occurred this year. Supervisor surveys are dependent on TTS response rates. Perceptions of supervisors in response to prompts are hard to determine. Such as, what do they view as "long term planning" v. how does the EPP interpret that. The supervisor marked “not able to observe: in the mental health and ELL student questions. It is unclear if graduates do not have opportunities to practice strategies for students with mental health and ELL students or if the supervisor has not observed it during the teacher evaluation process.
4. Satisfaction of completers (Component 4.4)
Findings on completer satisfaction are collected from surveys administered to completers the academic year following their graduation. All completers are invited to complete the survey, but those who are teaching complete an additional section to rate the quality of their preparation. The survey is administered approximately one year after the graduates completed their preparation programs.
The Division of Education has set an acceptable target of a 2 or higher mean on all indicators (all tends to agree or agree) and approximately commensurate means with the ND state aggregate; completers perceive their preparation as relevant to the responsibilities they confront on the job, and that the preparation was effective as indicated with a tendency to agree.
An ideal target has been set at a 3 or higher mean on all indicators (all agree) with results above the ND aggregate. Results of the survey are reviewed annual by the Division of Education Faculty.
2017-2018 Summary of Findings: Overall completers perceive their preparation as relevant to the responsibilities they confront on the job, and that the preparation was effective. 88% had applied for a teaching license in ND and 16% in MN; graduates had also applied in Washington, Canada (2), and Wisconsin. 37.5% stated they were well prepared for their interview, and 58.3% they were somewhat prepared-95.8% received job offers. 61.9% indicated a formal induction/mentoring program was available, and 38.1% that it was not. All but 1 completer indicated they would recommend the MSU program to others. Differentiating instruction and classroom management were identified as the area completers felt the most need for professional development. This is commensurate with prior surveys, the state aggregate, and the individual item results.
Completers rated their preparation at or near the ND aggregate on most measures. All but 4 items on the survey had a mean of 3 or above indicating a tendency to agree. This was a decrease from 8 items reported by the previous cohort below a mean of 3. The items that were rated under 3 with a tendency to disagree include the following:
1. Differentiate instruction for students with IEPs and 504 plans. (2.9)
2. Differentiate instruction for students with mental health needs. (2.7)*
3. Differentiate instruction for gifted and talented students. (2.9)*
4. Differentiate instruction for English-language learners. (2.8)*
*indicates an item carried over from the prior cohort
Completers indicate overall they were prepared to enter the teaching field. The survey is a valid and reliable tool utilized across 63 IHE. Comparison to state level aggregate is available. The response rate has remained relatively high at 63% (prior year 2016-2017 was 73.2%). The state aggregate response rate was 44%.
Not all graduates complete the survey. Language and terminology of the EPP is not always commensurate with the language of the survey (i.e., self-assessment). Teaching diverse learners and differentiating instruction continues to be an area that completers identify as needing professional development. We are not using the predictive capabilities or the correlational information of the TTS to capacity.
5. Program completers for 2017-2018: 39
6. Ability of completers to meet licensing: Institutional Pass Rate on Praxis Exams: 97% Average GPA: 3.73
7. Ability of completers to be hired in education positions for which they have prepared-placement patterns of completers 2016-2017:
8. Student loan default rates & other consumer information:
To assure appropriate stakeholders are involved in the continuous improvement process, annual findings are brought for feedback and discussion to the Teacher Education Advisory Board, which includes alumni, employers, practitioners, and school and community partners. Action plans are shared and revised based on stakeholder feedback. Additionally, information on annual reporting measures and program outcomes is included in all MOUs, shared via the MSU Foundations Office, and also through the Division of Education Facebook Page.
Important Finding #1: Update Mission and Vision (5.3)
Finding: Teacher candidates in early childhood, elementary, secondary and the MAT programs are meeting, and often exceeding, acceptable targets for the knowledge, skills and dispositions of effective teaching. Assessment evidence outlined across measures in TaskStream confirms that candidates are progressively building pedagogical and content knowledge throughout teacher training in the categories of learner and learning, content, instructional practice and professional responsibility. With positive candidate outcomes clearly evident through disaggregated course and program data analysis, the broader focus of the Division as a whole was considered.
The 2017-2018 academic year brought progressive changes in admission requirements and programming in response to accreditation standards, rapid changes in teacher licensure requirements at the state level, changing K-12 content standards and state-wide vision planning, a focus on personalized learning, and an increasing shortage of qualified teachers. In addition, the first full cycle of courses for the 18-month Master of Arts in Teaching (MAT) program concluded, and the Division was also able to reflect on the updated MSU institutional mission (November 2016) in relation to the mission, vision and goals of 13 teacher training programs, including the teacher leadership focus of the MAT program. This review led to the conclusion to revise the Division mission and vision statements to support forward program planning. With content program reviews due to the state licensing board and a full self-study report for national accreditation due the upcoming year, the time was right to visualize and articulate a possible future for teacher training programs at Mayville State. The mission and vision must give a common language to the collective work of training effective teachers as the threaded through all programs.
The Division Chair, Student Placement & Data Management Coordinator, and CAEP Accreditation Coordinator drafted a revised mission and vision statement. The draft was presented to faculty in advance of a full-day faculty retreat to reflect and comment on. At the faculty retreat, two groups were formed to consider the versions in small groups. Suggestions for improvement were made, discussion was held around the nuances of wording, and weighed against the lived experience of teaching and learning in the EPP through faculty perception. The final draft was unanimously approved by Division faculty and unanimously approved by the Teacher Education Committee with no changes. The new mission and vision are included below.
Mission: The Division of Education is committed to creating a collaborative culture that frames the development of teacher candidates’ knowledge, skills and dispositions through intentional, dynamic, integrated, and diverse teaching and learning experiences that support personalized, professional growth to positively impact learners.
Vision: To be a leader in educator preparation through transformative, relevant programming, learner-centered pedagogy, and partnerships that enable unique opportunities for quality experiences supportive to learning and leadership.
Action: Implementation of the new vision will begin through strategic goal setting for the next 7 year accreditation cycle. This process will be based on the annual review of candidate competencies and components of the quality assurance, feedback from state content experts and the Teacher Education Advisory Committee, and results of the self-study report. Program area teams in Elementary Education, Early Childhood, Special Education, Secondary Education (through the Teacher Education Committee) and MAT will use the new mission and vision statements as a practical guide for creating plans, setting goals and objectives, making decisions, and coordinating and evaluating choices through the 2020 accreditation site visit process.
Important Finding #2: Clinical Placements/Partnerships (2.1 & 2.3)
Finding: The Reflective Experiential Teacher conceptual framework is based upon a belief that teacher candidates develop the ability to reflect on and apply current research findings, theoretical knowledge, and effective teaching practices through relevant classroom experiences. As such, clinical experiences are monitored annually for appropriate depth, breadth, diversity and coherence with program goals. Recent growth in the number of candidates in early childhood, elementary and secondary math teacher training programs, the addition of the MAT program, and distance/online expansion has rapidly increased the need for more placements in P-12 schools. As of fall 2018, the Student Placement & Data Management Coordinator arranged for 185 placements across 152 schools.
Simultaneously, quality placements with trained P-12 clinical educators has become increasingly difficult. Local schools within a drivable radius for MSU students are small, thus there is a small pool of clinical placement locations. In addition, schools are employing individuals with alternate access teaching licenses who are not qualified to work with teacher candidates due to teacher shortage, further limiting the options for placements. Alternate licensing also means candidates are admitted to MSU programs but have already been hired and are working as teachers; this had changed the scope and sequence of placements to supply mentorship to new teachers who have not been trained. Schools are also encouraging paraeducator and support staff to complete teacher training, thus current employees need time within their contracts to both meet the needs of their employer as well as the goals and objectives of planned clinical experiences to demonstrate application of pedagogy. Also, schools are working with major initiatives that require focused time by their faculty/staff that limit capacity for teacher training. Finally, some partner schools have additional requirements beyond teacher training expectations, such as addition hours, community involvement, specialization of content, co-teaching, and personalized learning models that change expectations for candidates.
The Division needs to be flexible and responsive to the changing needs of school partners as school faculty and improvement efforts are also undergoing continuous improvement. The increase in program enrollment does create a challenge for finding quality clinical placement locations, and the division is working to find creative solutions to this challenge.
Action: The Division will investigate innovative and technology-based placement options with organizations and schools to address the increased need for quality clinical experiences. These may include options such as undergraduate teaching for MAT candidates, teaching online through the ND CDE, virtual classroom observations, live-time video feeds into classrooms, co-teaching placements, and the addition of clinical faculty to oversee processes.