Case Study:

Preparation for Teacher Effectiveness Project

West Virginia University

Table of Contents

Introduction .............................................................................................................................. 2 Purpose ..................................................................................................................................... 3 Participants ............................................................................................................................... 4 Case Study Forms ...................................................................................................................... 6 Case Study Lesson Plans and Results ......................................................................................12 Results from the Focus Group .............................................................................................. 15 Questions for the Focus Group ................................................................................................20

Next Steps ............................................................................................................................... 29 Appendix .................................................................................................................................30

Educator Evaluations - Case Study Year One.............................................................................21 Educator Evaluations - Case Study Year Two.............................................................................28

INTRODUCTION

The EPP conducted a case study with graduates from our teacher preparation programs (TPPs) to obtain supplemental evidence of our completers’ impact on P-12 student growth for component 4.1. Participants provided a lesson/unit plan, which they had implemented, and joined a focus group interview that asked for their perceptions about assessment practices. A request for their P-12 learners’ results on state achievement tests was made, but the participants did not have thisinformation.

Participants also provided their Educator Evaluation completed by an administrator, which provides supplemental evidence about their application of professional knowledge, skills, and dispositions for component 4.2.

This case study project is an initial inquiry into the assessment practices undertaken by our EPP’s first-year teachers. IRB was not sought; however, guidelines were followed and participants completed consent forms. The case study is titled “The Preparation for Teacher Effectiveness Project.”

2

PURPOSE

The purpose of the case study is to examine the assessments administered by the EPP’s completers who are first-year teachers, so that the P-12 student growth that occurred within the completers classrooms is evaluated. Additionally, a focus group was conducted with the first-year teachers to gain their perceptions about assessment practices. Participants were also asked to provide a copy of their Educator Evaluation that was completed by an administrator.

Our completers’ perceptions about assessment practices are important to understanding our first-year teachers’ preparedness for implementing assessments and for using test results to change instructional practices.

The questions the case study sought to address include:

• How do first-year teachers use results from state achievement tests?• Do first-year teachers feel prepared to create classroom assessments?• How do first-year teachers use the results from classroom assessments?• Do first-year teachers feel that there are areas of improvement they would like to address

when using classroom assessments?

Participants in the case study were asked to provide the artifacts listed below and take part in a focus group interview. The CAEP Standard and Component associated with each item is identified.

Artifacts and Activities Requested from Case Study Participants

Association to Standard and Component

1.) A lesson/unit plan implemented with their students where progress was evaluated through a pre- and post-test.

The lesson/unit plan results provide supplemental assessment scores to evaluate student growth. (Standard 4.1)

2.) State standardized test evaluations they had received for the P-12 students in their class.

The state standardized test from the completer’s classroom supplements the data that the state provided for state achievement test results. (Standard 4.1)

3.) A copy of the participant’s Educator Evaluation that was conducted by an administrator.

The Educator Evaluation provides supplemental information to evaluate the completer’s application of professional knowledge, skills, and dispositions. (Standard 4.2)

4.) Participants were asked to join a focus group interview to learn about their perceptions of assessment practices.

The focus group provided information about the first-year teachers’ preparedness for implementing assessments and for using test results to change instructional practices.

3

PARTICIPANTS

Case Study Year One

Recruitment of 2015-2016 graduates

During the 2016-17 academic year, messages were sent to seven school counties where 2015-2016 graduates of our EPP were employed. Five of the counties are local school districts where the EPP places student teachers. Two additional counties that have large towns and where more of the EPP’s completers were employed were also contacted.

An administrator in each county was contacted, and the information about the project was shared with him or her. Forms were sent to the administrators (Bookmark: Case Study Forms), who then shared them with their employees who graduated from our EPP. A document that described the project was included as well as a consent form.

The EPP received information from nine graduates who were working in three different public school counties in West Virginia. However, only two of the graduates provided a consent form, so only those two individuals were included in the study.

Teacher Position

Teacher Preparation Program Completed

Licensure West Virginia County Public School

Kindergarten Teacher

5 YR Teacher Education TPP Elementary Education, K-6

County A

School Counselor

5 YR Teacher Education TPP School Counseling MA

Spanish 5-Adult School Counseling

County B

The two graduates provided these supplemental items:

• Results from a lesson plan• A copy of their Educator Evaluation• Participation in a Focus Group

Neither of the graduates provided results from a state achievement test for their students.

Case Study Year Two

Recruitment of 2016-2017 Graduates

When preparing for the second year of the case study, the EPP employed a new strategy in an effort to increase the participation rate. During spring 2017 when the exit survey was administered to our graduates at the end of their teacher preparation programs (TPPs), it was explained to the graduates that the EPP would be in contact with them during the next year if they were employed in West Virginia public schools. Graduates provided their contact information so that the EPP could be in communication with them during the following year.

4

During fall 2017, the EPP was in communication with the administrators from five school districts that are nearby to the EPP. The Assistant Dean sent information (Bookmark: Case Study Interest Form) about the case study project to the local school districts who, in turn, met with their first-year teachers during mentor meetings. If a first-year teacher was interested in participating in the project, s/he completed an interest form. The school district administrators collected the interest forms and sent them to the Assistant Dean. Two of the five local school districts collected interest forms from our graduates and sent them to the Assistant Dean.

The Assistant Dean called the graduates who had completed an interest form. The project and the three items needed from the graduate during that academic year were explained. Graduates indicated if they were still interested in participating in the case study project. The identifying information of those who stated they had an interest was kept. A graduate student worker helped the Assistant Dean to maintain contact with the graduates throughout the school year.

The EPP received responses from 24 first-year teachers employed in two local school districts. Of those teachers, only two graduates were able to complete all activities for the case study project:

Teacher Position

Teacher Preparation Program Completed

Licensure West Virginia County Public School

9th-12th Grade Mathematics

5 YR Teacher Education TPP Mathematics Education, 5 - Adult

County C

7th-8th Grade General Science

5 YR Teacher Education TPP Elementary Education, K-6 General Science, 5-9

County D

The two graduates provided these supplemental items:

• Results from a lesson plan• A copy of their Educator Evaluation• Participation in a Focus Group

Neither of the graduates provided results from a state achievement test for their students.

5

Case Study Forms

6

WestVuginia University. COLLEGE OF EDUCATION AND HUMAN SERVICES

Hello,

My name is Laura Porter and I am the Assistant Dean of West Virginia University from the College of Education and

Human Services.

I am writing because I understand that you are completing your first-year as a beginning teacher in Monongalia County Public Schools. The first year of teaching is often met with new experiences and new challenges. So, I want to first say,

"thank-you" for your hard work and perseverance in educating the students in our state of West Virginia.

The education profession is examining how first-year teachers have been prepared to meet the expectations of the

classroom and their effectiveness as teachers with their students. I am interested in learning about specific measures

related to teacher effectiveness that would allow WVU to determine and report to our accrediting organization the quality

of the teacher preparation that our graduates received.

I am writing to ask if you would share the following items with me before the end of this school year. This information

will be used to demonstrate the growth your students have shown during this school year and your effectiveness as a

teacher. This information will be handled with extreme confidentiality. Your name and the names of your student will not be used.

The items that I am requesting include:

I. A unit or lesson plan that includes pre-test and post-test scores with these components: a.) the number of students

in the class, b.) the number of students who completed the pre-test/post-test, c.) the lesson/unit goal/objectives, d.)the date(s) it was administered, e.) the pre-test scores without student names, f.) the maximum possible pre-test

score, g.) the post-test scores without student names, and h.) the maximum possible post-test score.2. Any standardized test evaluation that the students in your classroom complete such as: Smarter Balance tests,

Early Learning Profile, Fitnessgram, Alternative Special Education Assessments.3. Your completed Educator Evaluation conducted by an administrator in your county.

After receiving these items, I will contact you and/or a group of Monongalia County teachers who participated in this

request to schedule a focus group or telephone interview to clarify any of the information that was obtained.

I hope that you will consider working with me on this project. The information gained will assist WVU in learning how to

make improvements to its teacher preparation programs as well as report on the outstanding work that first-year teachers

in West Virginia are completing. If you are interested, please complete the Consent Form sheet so that I can contact you directly for the items requested and answer any questions that you may have.

Thank you,

Laura Porter, EdD

Assistant Dean, Student Services & Accreditation College of Education and Human Services West Virginia University ljporter@mail.\vvu.edu 304-579-9937

OFFICE OF THE DEAN

PO Box 6122 I 802 Allen Hall

Morgantown, VW 26506-6122

11.1304.293.3334 D 304.293. 7565 Equal Opportunity/Affirmative Action Institution

Academic Years: 2016-2017 and 2017-2018

7

CONSENT FORM: PREPARATION for TEACHER EFFECTIVENESS PROJECT

Project Leader– Dr. Laura Porter, Assistant Dean, Student Services & Accreditation Contact Information: 304.293.4415 (w) / 304.579.9937 (c) / ljporter@mail.wvu.edu

I. PROJECT ITEMSThis project involves obtaining items that will be used to demonstrate the growth your students have shownduring this school year and your effectiveness as a teacher. This includes:

1. A unit or lesson plan that includes pre-test and post-test scores with these components: a.) the number ofstudents in the class, b.) the number of students who completed the pre-test/post-test, c.) the lesson/unitgoal/objectives, d.) the date(s) it was administered, e.) the pre-test scores without student names, f.) themaximum possible pre-test score, g.) the post-test scores without student names, and h.) the maximumpossible post-test score.

2. Any standardized test evaluation that the students in your classroom complete such as: Smarter Balancetests, Early Learning Profile, Fitnessgram, Alternative Special Education Assessments.

3. Your completed Educator Evaluation conducted by an administrator in your county.

After Dr. Porter receives these items, your participation in a focus group or telephone interview is requested to clarify any of the information that was obtained.

II. CONFIDENTIALITY OF RECORDSYour records will be confidential. Identifying information, such as names, places, institutions, will be replacedwith a pseudonym. Only Dr. Laura Porter will have access to the identifying information. You will not beidentified in any reports.

III. QUESTIONSIf you have questions about the procedures of this project, please contacting Dr. Laura Porter(ljporter@mail.wvu.edu or 304.293.4415) during the workday.

IV. SIGNATURESBy signing this consent form, you affirm that you have read this consent form, you agree to take part in thisproject and to have Dr. Laura Porter contact you. You will receive a copy of this consent form.

_____________________________ ______________________________________ Participant (Print Name) Grade/Subject Area (i.e., Pre-K, Elem Ed-3rd,

Special Ed 2nd -4th, English 6th) _____________________________ ______________________________________ Signature School/County Where you are Employed

_____________________________ ______________________________________ e-mail Contact Phone Contact

_____________________________ ______________________________________ Date WVU Program Completed

Academic Years: 2016-2017 and 2017-2018

8

TIMELINE FOR PROJECT ARTIFACTS

NOVEMBER Consent Forms Complete

NOVEMBER—FRIDAY, APRIL 20TH Unit Plan with Pre/Post Testing is sent (email to Laura Porter and Reno Swiger)

MAY 1ST Education Evaluation sent (email to Laura Porter and Reno Swiger)

END OF MAY/ BEGINNING OF JUNE Participation in Focus Group

FALL 2018 Send Standardized Achievement Test Information (email to Laura Porter and Reno Swiger)

Email Contact: Dr. Laura Porter: ljporter@mail.wvu.edu Reno Swiger: reno.swiger@mail.wvu.edu

Academic Year 2017-2018

9

Case Study Interest Form

10

Hello First Year Teacher from WVU ~

Congratulations on starting your teaching career at Monongalia County Public Schools. West

Virginia University wishes you every success in your new position.

West Virginia University is also writing to ask that you maintain contact with us during your first

year of teaching. The education profession is examining how first-year teachers have been

prepared to meet the expectations of the classroom and their effectiveness as teachers with

their students. Your contact with us during your first year of teaching will help us to learn about

the preparation you received at WVU.

The information you provide to us will assist WVU in learning how to make improvements to its

teacher preparation programs as well as report on the outstanding work that first-year teachers

in West Virginia are doing.

Please take a moment to provide your contact information. Leave the completed form with the

leader of your First Year Mentor Program.

Sincerely,

Laura Porter

Laura Porter, Ed.D.

Assistant Dean, Student Services & Accreditation

College of Education and Human Services

West Virginia University

Allen Hall, 802-A

Morgantown, WV

304-293-4415

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Name: ___________________________________________________

Please provide contact information where we can best reach you:

e-mail: ___________________________________________________

alternate e-mail: ___________________________________________

phone number: ____________________________________________

I have a position at this school: _______________________________

I have been assigned to this grade level/content area: _____________

Academic Year 2017-2018

11

LESSON PLANS AND RESULTS

The purpose of the case study is to examine the assessments administered by the EPP’s completers who are first-year teachers, so that the P-12 student growth that occurred within the completers’ classrooms is evaluated. Participants in the case study were asked to provide a lesson/unit plan implemented with their students where progress was evaluated through a pre- and post-test. The state provided achievement data on standardized tests for the EPP’s completers. The lesson/unit plan results provide supplemental assessment scores to evaluate student growth for Standard 4.1.

Case Study Participants 2015 -2016 Completers

Completer #1 – Kindergarten Teacher

The kindergarten teacher sent copies of the lesson plans where she integrated the county-wide progress-monitoring periodic benchmark test used with the students in her classroom to address the English Language Arts Standards for kindergarten.

Results from the two progress monitoring tasks.

Two tasks: 1.) World Fluency Task (WFT) 2.) Phoneme Segmentation Task (PST)

1.) Thirteen Students completed the WFT. • 10 students had a higher percentile ranking on the WFT post-assessment.

2.) Thirteen Students completed the Phoneme Segmentation Task (PST). • Five students had higher percentile rankings on the PST post assessment.• Six students did not increase their percentile ranking on the PST post

assessment; however, final scores for the six students were above the 50th

percentile range. The manual for this CBM assessment states if the studentscores above the 50th percentile range, the evaluation can say with somedegree of confidence that the particular skill area is not an issue.

Based on these results, the majority of students who completed each of the tasks (i.e. WFT and PST), showed improvement or maintained their ability to perform a task at the time of the post assessment.

Completer #2 – School Counselor

The school counselor sent copies of a teacher-constructed pre- and post-test assessment that she used to measure a group of 4th and 5th graders’ ability to show improvements in organizing their work spaces and identifying organizational strategies. The assessment was part of a lesson plan that addressed the West Virginia Student Success Standards.

12

The counselor noted that the results of the pre- and post-test assessment were inconclusive; however, the students provided more in-depth responses during their post-test. The responses showed that students were able to reflect on their time in the group and the progress that they made. The counselor deduced that based on these responses, the students had made improvement with their organization strategies.

Performance Gap Reduction (PGR) Analysis

When examining the lesson plans and assessments that the first group of case study participants provided, it became difficult to make comparisons of student growth. This occurred because each lesson plan was measuring a different objective and was implemented with a different grade level of students. In order to evaluate student growth from the different lesson plans provided by the first-year teachers, the performance gap reduction (PGR) scale analysis was employed for the second group of case study participants. Waggoner, Carroll, Merk, & Wietzel (2015) explain that it is often difficult to measure a teacher’s impact on student learning, because there are difficulties in gathering student learning data that are comparable across teachers and programs. The PGR scale method is an approach to targeting and measuring student growth and to rating teacher impact on that growth. The Maine Department of Education explains the benefits of using the PGR scale analysis as follows:

• Provides for equity and comparability in establishing teacher impact rating for cohortswith low, high, or widely varying pre-assessment scores.

• Eliminates the variability in quality and rigor of growth targets set by individualteachers.

The steps used to complete the performance gap reduction (PGR) scale analysis are found in Appendix A.

Case Study Participants 2016 -2017 Completers

Completer #3 – Secondary Mathematics Teacher

The 9th – 12th Secondary Mathematics Teacher sent the results of a pre- and post-test for an Applications of Logarithms lesson plan. The objective of the lesson was for students to have an understanding of logarithm properties and to solve equations involving logarithms. The results of the pre-test showed that most students were unable to answer any of the questions. The mathematics teacher changed her instructional plans and extended the unit by including more days of practice and review. She incorporated examples that helped the students to understand how logarithms were used in their daily lives. The results of the post-test showed that students performance increased substantially. The PGR Impact Scale (See Appendix A) was calculated based on the pre- and post-test scores. The mean growth index reduced the mean performance gap by 116%, which indicates the teacher had a high impact in reducing the performance gap for this lesson.

13

Completer #4 – Upper Elementary Grade Science Teacher

The 7th – 8th grade general science teacher sent the results of a pre- and post-test for a Chemical and Physical Engagement lesson plan. The objective of the lesson was for the students to identify the difference between chemical and physical changes. The results of the pre-test showed that the students had forgotten the basic information about chemical and physical changes that they had learned in previous grade levels. The general science teacher adapted her instruction by providing more hands-on examples for the students. She noted that her students performed better when they learn through doing. The results of the post-test showed that, overall, students did make improvements and were able to demonstrate their understanding of chemical and physical changes. The PGR Impact Scale (Bookmark: Appendix A) was calculated based on the pre- and post-test scores. The mean growth index reduced the mean performance gap by 43%, which indicates the teacher had a low to moderate impact in reducing the performance gap for this lesson.

Summary:

The participants from both case study groups provided lesson plans that included assessments to meet the lesson objective. This information is limited by the small sample size; however, the results indicate that our completers are prepared to construct and use assessments. In addition, the completers used the results from the assessments to modify their instructional practices.

Based on these results, a consideration for the EPP is further study among completers in different licensure areas to note their perceptions about assessment practices. Additionally, the information gained from using the performance gap reduction (PGR) scale analysis with the second group of case study participants was helpful in determining their impact on student growth. Another consideration for the EPP is continuing this approach with a broader group of completers.

References:

Waggoner, J., Carroll, JB., Merk, H., and Weitel, B. (2015). Critics and Critical Analysis: Lessons from 19,000 P-12 Students in Candidates Classrooms, Teacher Education Quarterly, Winter 2015, pp. 97 – 108.

Maine Department of Education, Student Learning and Growth: Approaches to Measuring Teacher Effectiveness. Retrieved from: (http://www.maine.gov/doe/effectiveness/general/index.html)

14

FOCUS GROUP AND RESULTS

First-year teachers who participated in each year of the case study were asked to join in a focus group interview. The focus group was scheduled after the participants had sent copies of their lesson plans that included an assessment. The focus group was conducted with the first-year teachers to gain their perceptions about assessment practices. Our completers’ perceptions about assessment practices are important to understanding first-year teachers’ preparedness for implementing assessments and for using test results to change instructional practices.

Framework for Understanding Assessment Data

State and national assessments continue to be implemented across the United States to meet federal requirements. The West Virginia Department of Education (WVDE) is charged with the development of the Every Student Succeeds Act (ESSA) State Plan. It must meet the various requirements of the federal legislation, which includes describing the academic standards and assessments that are consistent with the ESSA regulations (refer to the WVDE website, http://wvde.state.wv.us/essa/development.html). Therefore, first-year teachers will be expected to implement state-required assessments. Their perceptions of these assessment practices are important to understanding their readiness to use assessments to inform instruction.

Questions for the focus group were derived from the Student Assessment Inventory for School Districts, Listening to Teachers: Sample Focus Group and Survey Materials, (Achieve, 2015: https://www.achieve.org/files/AchieveStudentAssessmentInventory_ListeningtoTeachers. pdf). The questions chosen for the EPP’s focus group and the full compilation of survey questions from Achieve.org are found in the Bookmark: Questions for Focus Group.

School evaluation practices involve end-of-year summative assessments, periodic benchmark assessments, and teacher constructed assessments. Improvements in student achievement are noted when assessment practices are linked to curriculum and instruction (Cauley and McMillian, 2010). Black, et al. (2003) noted that students taught by teachers who use assessment for learning achieved in six to seven months what otherwise took one year. Will (2017) reported that students clearly demonstrate growth when they are active participants in the assessment process.

This case study explored first-year teacher perceptions about assessment practices and student achievement, because teachers’ support and enthusiasm is crucial for the success of any evaluation model (Marzano, R.J., Toth, M., & Schooling, P., 2012). When studying teacher perceptions about classroom assessment practices, Yao (2015) found that teachers embraced the notion of assessment for learning but often perceived assessment as an end product or an assessment of learning. Militello, M., Bass, L., Jackson, K. T., & Wang, Y. (2013) studied teacher and principal perceptions about assessment data use. Findings noted that differences existed among principals and teachers. Principals use data to evaluate the school, make improvements, and model best practices of data use. Whereas teachers used data to improve instruction and outcomes for students.

15

Studying the P-12 student growth of completers who were prepared by the EPP was accomplished by examining the different lesson plan assessment results that each completer provided and understanding the perceptions of the completers towards assessment practices used during their first-year of teaching. The focus group interviews explored the completers’ perceived usefulness of the assessment practices (summative, periodic benchmark, and teacher-constructed formative tests). Perceptions about their preparedness to implement assessments, to use test results for instruction, and the meaningfulness of the assessment practices for their students were investigated. The case study is a beginning inquiry about the effectiveness of the assessment practices employed by our first-year teachers.

Conducting the Focus Group: Year One and Year Two

The process for conducting the focus group was the same for both groups of graduates. The 2015-16 graduates sent items for the lesson plan portion of the case study project during the 2016-17 academic year. The focus group was completed during December 2017. The 2016-2017 graduates sent items for the lesson plan portion of the case study project during the 2017-2018 academic year. The focus group was completed during May 2018.

For the December 2017 and May 2018 focus group, the Assistant Dean and a graduate student met one of the completers at the school where she taught. The other completer was called via cell phone and also accessed using FaceTime. The focus group interview was recorded as the completers responded to the questions found in the Bookmark: Focus Group Questions. The graduate student transcribed the audio recording. Then the Assistant Dean and the graduate student reviewed the transcribed text and coded the text to identify common themes that the emerged from the completers’ responses (Ryan & Bernard, 2005).

Themes emerged from the perceptions of the completers in both focus groups about assessment practices, their preparedness for implementing assessments, for using test results to change instructional practices, and the meaningfulness of assessments for their students. The participants’ responses are used to answer the research questions.

Question One:

• How do first-year teachers use results from state achievement tests?

Over the course of the two years for the case study, state test achievement practices havechanged. The state was administering the Smarter Balance assessments during the 2016-2017 academic year for the first-year teachers who participated in the first focus group.

The counselor monitored the Smarter Balance assessment results for all grades at herschool during 2016-2017. Her analysis of state assessment data included considering thewhole child. She referenced the impact of Maslow’s Hierarchy of Needs on students’

16

achievement. When viewing assessment results, she asked herself if the students were having their basic needs met so that they could perform at their best.

The kindergarten teacher explained that the required state benchmark assessment at the kindergarten level during her first year of teaching was confusing to many teachers in the building. The instrument changed the following year, and further changes to the instrument were being discussed. The frequent changes to the assessment may have resulted in a lack of confidence in its purpose.

During the 2017-2018 academic year, the state made changes to the achievement tests administered. At the time of the focus group interview in May 2018, the 9th- 12th grade mathematics teacher and the 7th – 8th grade science teacher had not yet seen the results of the state achievement tests their students had completed. Both of these first-year teachers were waiting for more information from their principals about what was expected from the state test results.

To answer this question more fully, further administrations of the state achievement test and data reports are needed. When this occurs, examination of the first-year teachers’ understanding of the state achievement results can occur.

Question Two:

• Do first-year teachers feel prepared to create classroom assessments?Participants across both focus groups shared that they had received preparation aboutclassroom assessments during coursework in their teacher preparation program. Twoparticipants discussed using the “Backwards Design” approach that had been explainedduring their preparation program. Another participant reported that she had learned howto structure test questions during a course in her undergraduate program.As beginning teachers, the participants all shared that they developed their understandingof assessment by collaborating with their peer colleagues. Some respondents participatedin Professional Learning Communities and discussed assessment practices with grade-level partners. Other respondents used online communities and websites that containedassessment resources.The results of this focus group show that participants felt prepared to include classroomassessments in their instruction. Participants also discussed how they continue to add totheir understanding about how to develop classroom assessments by collaborating withtheir peers.

Question Three:

• How do first-year teachers use the results from classroom assessments to informinstruction?

17

All of the participants discussed the importance of using assessment for instruction. Analyzing data from assessments helped to determine the type of instruction to use with students. For example, the kindergarten teacher used data results to group students with similar phonemic awareness skills. After reviewing pre-test results, the science teacher recognized that more hands-on experiences were needed to refresh the students understanding of the difference between chemical and physical changes. The mathematics teacher shared that she extended the amount of time for the unit because the pre-test results indicated that the students were struggling with the application of logarithms.

Participants shared that assessment data not only monitored student growth, it also made them accountable to student learning. The counselor noted that after addressing organizational strategies with her 4th and 5th graders, the results of her post-test indicated that students were more reflective when responding to questions. The kindergarten teacher explained that the growth her students made with phonemic skills verified that she was appropriately instructing their developmental levels. The mathematics teacher reported that students showed improvement on the post-test, which also indicated that her instruction was effective. The science teacher felt that the increase in the students’ performance on the post-test occurred, because she had worked through the misconceptions that became apparent on the pre-test.

The responses from the case study participants indicates that our first-year teachers understand the importance of classroom assessments and use the results from assessments to inform their instructional practices. Participants gave examples of how they were able to measure student growth by incorporating assessments into their instruction.

Question Four:

• Do first-year teachers feel there are areas of improvement they would like to addresswhen using classroom assessments?

The participants also shared some areas they would like to continue to develop when using classroom assessments based on their participation in the case study. Two participants shared that the focus of their first-year of teaching had been organizing lessons and using suggestions from other teachers. After completing a year of teaching, they felt more confident in planning lessons and units. They planned to incorporate more pre- and post-tests into their lesson and unit plans because they felt it was helpful to monitor student growth. Another participant shared that she wanted to include more opportunities for students to share what they learned about their performance from assessment results.

These responses indicate that our first-year teachers understand the benefit of using assessments. Participants gave examples of how they wanted to improve their assessment practices and gave examples of how they could incorporate this goal in the future school years.

18

Summary

The participants from both focus group interviews responded with statements that indicated that they had a good understanding of the value of implementing assessment practices with their students. This information is limited by the small sample size. Still, the results from the focus group interviews demonstrate that our completers are prepared to use assessments, have developed strategies to improve teacher-made assessments, have a strong understanding of the benefits of using assessments to inform instruction, and are interested in expanding their skills related to constructing and using assessments. Based on these results, a consideration for the EPP is further study among completers in different licensure areas to note their perceptions about assessment practices. The EPP also plans to triangulate the data responses from the case study participants with the employers' feedback from the Supervisor Survey.

References

Will, M. (2017). What happens when students are the test designers? Ed Week, 36,(32), 7-10.

Black, P., Harrison, C., Lee, C., Marshall, B., & William, D. (2003). Assessment for learning: Putting it into practice. Open University Press: Buckingham, UK.

Cauley, M. C., McMillan, J. H. (2010). Formative assessment techniques to support student motivation and achievement. Clearing House: A Journal of Education Strategies, Issues and Ideas, 83(1), 1-6.

Marzano, R. J., Toth, M., & Schooling, P. (2012). Examining the role of teacher evaluation in student achievement: Contemporary research base for the Marzano causal teacher evaluation model. Retrieved June 18, 2018, from Learning Sciences International: https://www.learningsciences.com/wp/wp-content/uploads/2017/06/Role-of-Teacher-Evaluation-2012.pdf.

Militello, M., Bass, L., Jackson, K. T., & Wang, Y. (2013). How data are used and misused in schools: Perceptions from teachers and principals. Education Sciences, 3, 98-120. DOI: 10.3390/educsci3020098.

Ryan, G. W., & Bernard, H. B. (2005). Techniques to identify themes in qualitative data. Retrieved June 18, 2018, from: http://academia.uat.edu.mx/pariente/Lecturas/Techniques%20to%20Identify%20Themes%20in%20Qualitative%20Data.pdf

Yao, Y. (2015). Teacher perceptions of classroom assessment: A focus group interview. SRATE Journal, 24(2), 51-58.

19

Questions for Focus Group with WV Teachers Prepared at WVU

Purpose: To understand the perceptions about the value of state assessments and the value of their

own classroom common assessments from first-year teachers.

Opening Questions

-What grade(s) and/or subject(s) do you teach?

-How long have you worked in this county?

Questions about County/State Required Assessments

-Based on your understanding of (identify a state assessment), how well does this assessment align to

content standards? How do you know?

-What is your understanding of the purpose of the (identified state assessment)? What is it designed to

measure?

-What are the ways you use (the identified state assessment)?

-To what degree do you find (the identified state assessment) useful? (This is where the moderator may

prompt: strong alignment to standards, timely results, helpful reporting, helps inform instruction, etc.

Then a follow-up may be: how did you learn to use assessment for this purpose?)

-What did you learn about your student's achievement from this assessment?

-What did your student's learn about their achievement from this assessment?

Questions about Classroom Common Assessments

-How did you develop the classroom assessment that was shared with (name of EPP).

-Could you explain how you learned to create classroom assessment?

-What are the way you use this classroom assessment?

-To what degree do you find it useful? (This is where the moderator may prompt: strong alignment to

standards, timely results, helpful reporting, helps inform instruction, etc. Then a follow-up may be: how

did you learn to use assessment for this purpose?)

-What did you learn about your student's achievement from this assessment?

-What did your student's learn about their achievement from this assessment? (follow-up questions)

Questions for the focus group were derived from the Student Assessment Inventory for School

Districts: listening to Teachers: Sample Focus Group and Survey Materials, p.8 (Achieve,

2015: https://www.achieve.org/publications/listening-teachers-sample-focus-group-and-survey-

materials

20

Completers' Evaluations

Supplemental Evidence for Component 4.2

Case Study Year One

First-Year Teachers 2016-2017

Completer #1 - Kindergarten Teacher

Completer #2 - School Counselor

Completers who participated in the first year of the case study project also provided

a copy of their evaluations by an administrator during their first year of teaching.

The items that follow are the evaluations for Completer #1 and #2. The results show

that the completers received strong ratings overall, which is an indicator of their

abilities to apply the professional knowledge, skills, and dispositions that address

P-12 student learning.

21

Name:· Supervisor /Mentor:

School:

Standard

1. Plans, prepares, and implements

learning experiences that make the

subject matter meaningful for

students.

2. Delivers instruction based on best

practices in ways that engage

students.

3. Demonstrates knowledge of the

content and its inter-relatedness

within and across disciplines.

4. Creates an environment that

manages student behavior and

emphasizes the importance of

learning.

5. Collaborates with school colleagues

and other professionals to best serve

the needs of students

6. Supports the

cognitive/social/emotional

development of students from all

backgrounds to impact learning.

7. Gathers and interprets assessment

data for improvement of teaching

and learning.

8. Demonstrates competency in the

knowledge and implementation of

technology standards/instruction.

9. Engages in reflective practice to

improve individual teaching

competencies and student learning.

10. Observes professional

responsibilities in an ethical manner

and adheres to established

regulations, policies, rules, and laws

(WV Employee Code of Conduct).

Distinguished Proficient Basic Unsatisfactory

V

/

V

✓

Completer #1Case Study - Year OneMentor/Supervisor Observation of WVU Graduates

22

CounseJor: I • -

School Year 2017

Standard 1 : Program Planning, Design and Management - The professional school counselor collaboratively plans, manages and designs the comprehensive, developmental school counseling program. DISTINGUISHED

Standard Element 1.1 The school counselor assumes leadership in planning, designing and advocating for a balanced, comprehensive-school counseling program aligned with the state model.

® Distinguished O Accomplished O Emerging O Unsatisfactory

Standard Element 1.2 The school counselor manages the school counseling program and documents alignment with the WV School Counselor Performance Standards. @ Distinguished O Accomplished O Emerging O Unsatisfactory Standard 1 Comments

This year, the counseling program was built from scratch based on the SCA national model and WV state model. A comprehensive plan and

strategic plan write up were placed in the principals mailbox within the first eek of school. Throughout the year, the program was evaluated by the

counselor at different times to ensure it was running in adherence with the ational and state models. The amount of time the counselor was spending in

direct service of students vs. indirect service of students was monitored. This ear, delivering guidance curriculum lessons took the large majority of time.

Moving foiward, making time spent on individual counseling and group counseling more proportionate with the time spent in the classroom will be a focus. Throughout the 1ear, documentation has been kept in all areas of the counseling program. A daily log in a calenruµ- was used, but also the WV counseling log was used to track time and ensure a comprehensive program

as running well. A counseling curriculwn map was used to plan out the ear oflessons, successfully meeting the WV Student Success Standards. In

addition, three times this year, the Counselor Performance Standards were ed to help the counselor reflect and evaluate the program. This helped

e the ro on track and focused throu out the entire school ear.

Standard 2: Program Delivery - The professional school counselor facilitates delivery of the WV Comprehensive Developmental School Counseling Program for all students. DISTINGUISHED

Standard Element 2.1 The school counselor facilitates the implementation of an integrated comprehensive guidance curriculum. @ Distinguished O Accomplished O Emerging O Unsatisfactory

Standard Element 2.2 The school counselor utilizes research-based and best practices as reflected by state school counseling protocols. @ Distinguished O Accomplished O Emerging O Unsatisfactory

Standard Element 2.3 The school counselor provides a continuum of interventions and responsive services. ® Distinguished O Accomplished 0 Emerging 0 Unsatisfactory

Completer #2Case Study - Year One

23

24

25

26

Standard 6 worked diligently to reach her goal. Their 14 of the 16 Career Comments Clusters represented. Students were highly engaged. I believe the 2 weeks of

lessons combined with the Cafe made a huge impact on the students. Students continue to set goals and speak of what they want to do when they go to college or make a career choice.

Standard 7: Professional Conduct

Professional Conduct Standard 1 Policy and Procedure - The counselor demonstrates professional conduct as defined in law, policy and procedure at the state? district, and school level.

@ Meets Standard 0 Below Standard . 0 Unsatisfactory Professional Conduct Standard 2 Attendance - The counselor demonstrates professional conduct as defined in law, policy and procedure at the state, district, and school level.

@ Meets Standard O Below Standard O Unsatisfactory Professional Conduct Standard 3 Schedule - The counselor demonstrates professional conduct as defined in law, policy and procedure at the state, district, and school level.

@ Meets Standard O Below Standard O Unsatisfactory Professional Conduct Standard 4 Respect - The counselor demonstrates professional conduct as defined in law, policy and procedure at the state, district, and school level.

Standard 7 Comments Evaluator Commendations and Recommendations

@ Meets Standard O Below Standard O Unsatisfactory always conducts herself in a professional and positive manner

when workin with her colle es students and families. This was _ first year as a school counselor. She is to be commended for the amazing job she did getting to know our students and families. She works diligently to meet tlle needs of all our students. Whtm . ot in the classroom, she is working with individual students or groups. She

often takes her lunch and.planning iime_to work with students. Thank you for going above and beyohd.:.Career Cafe, Buddy Benches, food bags,

adershi Team Master Gardner Ciub arid overseein the ardens ...

S39286 accepted evaluation

27

Completers' Evaluations

Supplemental Evidence for Component 4.2

Case Study Year Two

First-Year Teachers 2017-2018

Completer #3 - 9th-12th Grade Math Teacher

Completer #4 - 7th -8th Grade General Science Teacher

Completers who participated in the second year of the case study project also

provided a copy of their evaluations by an administrator during their first year of teaching.

The items that follow are the evaluations for Completers #3 and #4. The results

show competencies related to the West Virginia Professional Teaching Standards

that were observed by the evaluator. The positive comments cited throughout both

evaluations indicate the efforts taken by each individual and the progress made in

developing their professional practices.

28

NEXT STEPS

The case study, Preparation for Teacher Effectiveness Project, was an initial inquiry into our completers’ preparation with assessment practices. Results of the case study project are limited based on the sample size. The first-year teachers who participated were enthusiastic about the project and discussed their interest in using assessments, constructing assessments, and modifying their instruction based on assessment results. This is understandable in that participants who took the time to join the project would most likely be interested in sharing their knowledge and furthering their skills.

Based on these results, a consideration for the EPP is further study among completers in different licensure areas to note their perceptions about assessment practices. The EPP also plans to triangulate the data responses from the case study participants with the employers' feedback from the Supervisor Survey. Additionally, the information gained from using the performance gap reduction (PGR) scale analysis with the second group of case study participants was helpful in determining their impact on student growth. Another consideration for the EPP is continuing this approach with a broader group of completers. Involving a greater number of completers would provide the EPP with more information about the assessment practices candidates are prepared to use. The use of the PGR scale analysis may be an approach to discuss with our clinical partners. It may be of interest to include this approach with their other efforts when measuring student growth.

29

APPENDIX

Steps for Calculating the Performance Gap Reduction (PGR)

1. Determine the maximum possible score on the assessment.2. Calculate pre-assessment scores.3. Determine the performance gap for each student. This is the difference between the

maximum score possible and the pre-assessment score.4. Determine the mean performance gap (this is the mean performance gap for entire class).5. Calculate the post-assessment score.6. Determine mean growth gain (difference between pre-and post-assessment scores) for

each individual student.7. Determine the class mean growth (this is the mean growth gain for entire class).8. Determine the percent performance gap reduction (this is calculated by: course mean

growth/mean performance gap).9. Determine the teacher’s rating on the PGR impact scale.

Rating on PGR Impact Scale

• Mean growth index reduces mean performance gap by at least 75% = High• Mean growth index reduces mean performance gap by at least 50% = Moderate• Mean growth index reduces mean performance gap by at least 25% = Low• Mean growth index reduces mean performance gap less than 25% = Negligible

Reference:

Maine Department of Education (2018). Student learning and growth--Approaches to measuring teacher effectiveness. Retrieved June 18, 2018, from General information on development and implementation of PEPG system components: http://www.maine.gov/doe/effectiveness/general/index.html

30