colorado department of education teacher rubric … · colorado department of education teacher...

Colorado Department of Education Teacher Rubric Comparison Study:

Final Report

February 14, 2013

cultivating learning and positive change

www.magnoliaconsulting.org

&

ZBarley Consulting, LLC

CDE Rubric Comparison Study Final Report, February 14, 2013 ZBarley Consulting, LLC and Magnolia Consulting, LLC

i

Acknowledgements

We would like to thank the Colorado Department of Education (CDE) and members of CDE’s Educator Effectiveness team—Dr. Katy Anthes, Dr. Jean Williams, Dawn Pare, and Tricia Majors—for their commitment to facilitating a transparent, inclusive, and comprehensive process for developing and implementing the teacher evaluation rubric as part of the CDE Model Evaluation System. Their dedication and relationships with educators across the state made recruitment for this study possible. We also want to recognize the valuable contributions of study evaluators who gave their time, expertise, and critical insights to the observation process. We are grateful to district and school administrators who opened their doors to this study and allowed us the honor of working with their teachers. At the core of this work are the teachers who participated, and we are thankful for their willingness to engage and reflect on the quality and utility of professional growth tools such as the teacher evaluation rubric. Finally, we want to acknowledge the great work by Bethy Leonardi, who administered participant surveys, tracked data collection activities, and prepared and analyzed data for reporting. The authors, Stephanie B. Wilkerson, Ph.D. Magnolia Consulting, LLC 5135 Blenheim Rd. Charlottesville, VA 22902 (ph) 855.984.5540 (toll free) http://www.magnoliaconsulting.org Zoe Barley, Ph.D. ZBarley Consulting 116 Jackson Street Denver CO 80206 720.318.1171


ii

Table of Contents Introduction ................................................................................................................................... 1 CDE Teacher Evaluation Rubric .................................................................................................... 2 Study Design ................................................................................................................................. 5

Participant Recruitment & Characteristics .................................................................................................. 5

Data Collection Methods ........................................................................................................................... 6

Participant Survey ................................................................................................................................................... 7 Evaluator Focus Group ........................................................................................................................................... 7 Teacher Evaluation Ratings .................................................................................................................................... 7

Procedures .................................................................................................................................................. 7

Teacher Observation Process ..................................................................................................................... 8

Study Findings ............................................................................................................................ 10 How Did the 4-Point and 5-Point Rubric Ratings Compare? ................................................................... 10

Rubric Modal Differences ..................................................................................................................................... 12 Rubric Distribution Differences ........................................................................................................................... 14

Rating Distribution Summary ................................................................................................................... 16

Which Rubric Provided Clearer and More Actionable Feedback? .............................................................. 16

Which Rubric Did Participants Prefer? ..................................................................................................... 19

Study Limitations ........................................................................................................................ 20 Summary and Discussion ........................................................................................................... 21 Appendix A: 4-Point and 5-Point Rubrics .................................................................................... 24 Appendix B: Teacher Quality Standards I, II, and III and Elements ............................................. 46 Appendix C: Participant Surveys ................................................................................................. 47 Appendix D: Evaluator Focus Group Protocol ............................................................................. 51

List of Figures

Figure 1. Average frequencies of ratings by performance level and rubric. ..................................................... 11 Figure 2. Comparison of rubric distributions for total number of ratings at or above Proficient. ................... 12 Figure 3. Participant ratings of rubrics providing clear and actionable feedback. ........................................... 18 Figure 4. Participant ratings of preferred rubric. ............................................................................................ 19


iii

List of Tables

Table 1. Excerpt from the 5-point Teacher Evaluation Rubric ........................................................................ 2 Table 2. Excerpt from the 4-Point Evaluation Rubric ..................................................................................... 4 Table 3. Study Sample .................................................................................................................................... 6 Table 4. Frequency and Chi-square Values by Element and Rating Level ..................................................... 10 Table 5. Professional Practices by Mode and Rubric for Element 2b ............................................................. 13 Table 6. Professional Practices by Mode and Rubric for Element 3c ............................................................. 13 Table 7. Professional Practices by Mode and Rubric for Element 3g ............................................................. 14 Table 8. Statistically Significant Chi-Square Results of Rating Distributions ................................................. 15 Table 9. Teacher Survey Ratings of 4-Point and 5-Point Rubrics .................................................................. 17 Table 10. School Evaluator Ratings of 4-Point and 5-Point Rubrics ............................................................. 17 Table 11. Evaluator Ratings of 4-Point and 5-Point Rubrics ......................................................................... 18


1

Introduction

The Colorado Department of Education (CDE) Educator Effectiveness team has the goal of promoting effective teaching across the state, and in support of that goal is developing the Colorado Model Evaluation System for principals and teachers1. The model system provides a resource bank of supporting documents including evaluation rubrics for principals, assistant principals, and teachers as well as user guides for each. The CDE Model Evaluation System reflects five key priorities as stated in the Teacher’s User Guide:

1. Data should inform decisions, but human judgment will always be an essential

component of evaluations.

2. The implementation and assessment of the evaluation system must embody continuous improvement.

3. The purpose of the system is to provide meaningful and credible feedback that improves performance.

4. The development and implementation of educator evaluation systems must continue to involve all stakeholders in a collaborative process.

5. Educator evaluations must take place within a larger system that is aligned and supportive.

The development and implementation of the Model Evaluation System teacher rubric is taking place in four phases: (1) development and testing of the evaluation tools, including the creation of the evaluation rubric and its corresponding user guides, during the 2011-12 school year; (2) piloting the teacher rubrics in 27 districts during the 2011-12 and 2012-13 school years; (3) conducting a validation study with statewide rollout and monitoring of pilot sites during the 2013-14 school year; and (4) full statewide implementation, also scheduled for the 2013-14 school year.

As part of the development process for the CDE Model Evaluation System, the CDE Educator Effectiveness team is examining the merits of using a teacher evaluation rubric with a 4-point versus a 5-point rating scale when anchored to the same standard of proficiency. The CDE contracted with two external and independent research firms--Magnolia Consulting, LLC, and ZBarley Consulting, LLC—to design and facilitate a comparison study of 4-point and 5-point rubrics. Researchers conducted this study from September to December 2012.

Magnolia Consulting and ZBarleyConsulting established a firewall between themselves

and CDE to ensure that the study was conducted independently and objectively. The firewall prevented CDE from accessing, analyzing, or interpreting the data during the study period. Researchers conducted informal progress reports with the CDE team during the study period through telephone and email correspondence to inform them of study progress and any potential study implementation issues.

1 Senate Bill 191 provided legislative direction to assessing teacher effectiveness. 2 For two elements the majority of ratings deviate from Partially Proficient or Proficient. For Element 1c, most teachers


2

CDE Teacher Evaluation Rubric

The CDE Framework to Evaluate Teachers includes six Teacher Quality Standards leading to a designation of a performance standard for each teacher of Ineffective, Partially Effective, Effective, or Highly Effective. The current teacher evaluation rubric is used to assess professional practices for five of the Teacher Quality Standards. (The sixth Teacher Quality Standard is Student Growth.) Each of the five standards has supporting elements that define the mandatory items that each district’s teacher evaluation system must address. Each element within the standards describes professional practices of daily instruction that teachers would be expected to demonstrate. Table 1 presents an excerpt from the 5-point rubric to show the nesting of professional practices within elements and elements within standards. Educators are assessed on their performance of the professional practices within each element using a 5-point rating system for each Teacher Quality Standard:

• Not Evident. Teacher does not meet state performance standard. • Partially Proficient. Teacher does not meet state performance standard. • Proficient. Teacher meets state performance standard. • Accomplished. Teacher exceeds state standard. • Exemplary. Teacher significantly exceeds state standard.

Table 1. Excerpt from the 5-point Teacher Evaluation Rubric Quality Standard I: Teachers demonstrate mastery of and pedagogical expertise in the content they teach. The elementary Teacher is an expert in literacy and mathematics and is knowledgeable in all other content that he or she teaches (e.g., science, social studies, arts, physical education, or world languages). The secondary Teacher has knowledge of literacy and mathematics and is an expert in his or her content endorsement area(s).

Not Evident Partially Proficient

Proficient (Meets State Standard) Accomplished Exemplary

Element a: Teachers provide instruction that is aligned with the Colorado Academic Standards; their District's organized plan of instruction; and the individual needs of their students. There is inadequate evidence that the Teacher: O Plans instruction

on a daily basis. O O Includes a

defensible progression of learning in instructional plans.

O Uses instructional objectives that are appropriate for all students.

The Teacher: Develops lesson plans based on: O Colorado

Academic Standards.

O District’s plan of instruction.

O Student needs.

. . . and The Teacher: Aligns instruction with: O Student learning

objectives. O District plan for

instruction. O Colorado Academic

Standards. Collaborates with

other school staff to vertically and horizontally articulate the curriculum.

. . . and Students: Advance to the

next level within the curriculum or next higher course in sequence.

O Interact with the rigorous and challenging content in meaningful ways.

. . . and

Students: Discuss gaps in their learning with: O Teacher. Families and

significant adults.

O Professional Practice is Observable during a classroom observation. Professional Practice is Not Observable during a classroom observation.


3

The rubric is intended for professional practices to build cumulatively beginning with

those represented under Not Evident and moving through each rating level towards Exemplary. Therefore, the Proficient rating should not be construed as a midpoint from which raters look for practices falling above or below “average,” but rather as a midpoint in a progression toward exemplary teaching. As teacher practices reflect greater proficiency, evidence of teacher effectiveness should be apparent in student actions and behaviors. Therefore, many of the professional practices under Accomplished and Exemplary focus on students.

The rubric includes professional practices that are observable and non-observable during

classroom instruction (see the bottom row of Table 1 for symbols denoting observable and non-observable professional practices). For many of the professional practices, including those non-observable, evaluators and teachers need to refer to artifacts (e.g., lesson plans, parent feedback, student work, formative and summative assessment of student work, etc.) and other sources of evidence before assigning a performance level. This reflects the intention for both evaluators and teachers to use the rubric to create a comprehensive and holistic evaluation process of teacher practice. It acknowledges that observations alone do not provide sufficient information about teacher effectiveness.

For purposes of this study, researchers modified the current 5-point rubric to include

only four points: Not Evident, Partially Proficient, Proficient, and Exemplary (see Table 2). In creating the rubric, researchers’ intent was to have the 4-point rubric comparable to the 5-point rubric to the greatest extent possible so that the primary comparison would be between having four or five points. Therefore, for the 4-point rubric the professional practices for Not Evident, Partially Proficient, and Proficient are the same as in the 5-point rubric. For each element, researchers referred to the professional practices for the 5-point Accomplished and Exemplary performance rating levels and revised them to create a set of practices for Exemplary in the 4-point rubric. The process resulted in keeping the number, level, and elevation of the practices in the 4-point Exemplary category the same as in the 5-point Exemplary category (see Appendix A for the 5-point and 4-point rubrics used in the study).


4

Table 2. Excerpt from the 4-Point Evaluation Rubric Quality Standard I: Teachers demonstrate mastery of and pedagogical expertise in the content they teach. The elementary Teacher is an expert in literacy and mathematics and is knowledgeable in all other content that he or she teaches (e.g., science, social studies, arts, physical education, or world languages). The secondary Teacher has knowledge of literacy and mathematics and is an expert in his or her content endorsement area(s).

Not Evident Partially Proficient Proficient (Meets State Standard) Exemplary

Element a: Teachers provide instruction that is aligned with the Colorado Academic Standards; their District's organized plan of instruction; and the individual needs of their students. There is inadequate evidence that the Teacher: O Plans instruction on a

daily basis.

O Includes a defensible progression of learning in instructional plans.


The Teacher: Develops lesson plans based on: O Colorado Academic

Standards. O District’s plan of

instruction. O Student needs.




Standards. Collaborates with


. . . and Students:

O Interact with the

rigorous and challenging content in meaningful ways.

Identify gaps in learning in student journals and learning logs.


One other modification to the rubrics was made for purposes of the study. Of the five

Teacher Quality Standards in the 5-point rubric, only the first three were included in the study (see Appendix B for the Teacher Quality Standards I – III). The professional practices in the Teacher Quality Standards IV and V are made up almost entirely of non-observable professional practices (73 out of 79). For these practices an evaluator must obtain artifacts and other materials from the teacher, school, or district, and typically discuss them with the teacher being evaluated to determine the teacher’s progress toward achieving the professional practice. The study evaluators were not educators within the participating districts and would not easily have had access to these materials. Obtaining and rating them would have imposed a further burden on the participating schools and teachers.


5

Study Design The purpose of this study was to compare a 4-point and 5-point version of the CDE

teacher evaluation rubric to determine if one rubric or the other provided a more effective rating structure of educator effectiveness. The study used a mixed-method approach to address the following research questions:

1. Which rubric version enables greater differentiation among educator performance, and

thus yields a more robust distribution of educator evaluation ratings? 2. Which rubric version provides clearer and more actionable feedback for educator

improvement?

The study strictly focused on the comparison of the two rubrics and not on other aspects of measuring teacher effectiveness using the Model Evaluation System. Thus it did not address participants’ feelings toward SB 191 or towards evaluation of teaching practice in general. The districts, schools, and teachers who participated in the study are not part of the CDE validation pilot study and therefore, did not receive training on the 5-point rubric or an informational orientation to SB 191 and the Model Evaluation System. Evaluators for this study included educators, principals, and instructional coaches experienced in conducting teacher observations. All study evaluators received training on the rubrics. Each study evaluator used both rubrics in each of two observations, alternating which rubric they used first. Each study evaluator then conducted a feedback session with each teacher and his/her school evaluator who had been invited to sit in on the observations. The feedback sessions focused on comparing results from the two rubrics.

The data for this report come from teacher observations, surveys of teachers, school

evaluators, and evaluators, and a focus group with the evaluators. The data are limited by a small sample size (49 teachers) and characteristics of the districts/schools participating; that is, no urban districts or large schools were included in the study. The lessons observed were based on convenience and were not purposively or randomly sampled by subject being taught. The data also are limited by observation of only two lessons. The study did not involve teacher development of professional growth goals with evaluators. The evaluators were unfamiliar with student characteristics in the classrooms they observed and did not have access to a comprehensive body of evidence for the non-observable professional practices as a typical school evaluator would. Based on time limitations and access to comprehensive non-observable data, researchers selected only three of the five Teacher Quality Standards for this study (see Appendix B). For these reasons, this study focuses on element-level ratings, not standard-level ratings. Given the limitations of the study there would be insufficient evidence to rate the participating teachers at the standard level or overall.

Participant Recruitment & Characteristics

CDE identified and recruited eight evaluators with experience conducting teacher evaluations and familiarity with the state Model Evaluation System. Study researchers provided evaluators with a review of both rubrics and a full orientation to the study. Evaluators received a


6

stipend for their participation as well as reimbursement for travel expenses to sites. The teaching experience of the evaluators ranged from 15 to 35 years with an average of 19 years.

CDE and researchers recruited a purposive sample of five school districts across the state. Districts were recruited based on the following characteristics:

1) Willingness to dedicate additional staff time to participate in observations and data

collection activities. 2) Not already involved as a pilot site and thus not familiar with the 5-point rubric. 3) Proximity of trained evaluators to districts to optimize teacher evaluation time and

minimize evaluator travel to sites.

Within each district, a sample of schools was obtained for a total school sample size of 13 schools. Within each school a random sample of regular classroom teachers from those who volunteered was obtained across grades and content areas, for a total sample of 49 teachers. In most cases researchers randomly selected teachers from the list of volunteers sent by the principal using statistical software that permits a random selection of cases. In one small district all of the teachers— 6 elementary, 4 middle, and 4 high school teachers—participated. In two other districts it was possible to include all teachers who volunteered as participants. See Table 3 for a presentation of the final sample of 13 schools and 49 teachers across five districts. Districts ranged in size from 178 to 3,124 students and were either classified as rural or remote. In each school the school-level evaluator who typically conducts evaluations was invited to observe the study activities (i.e., classroom observations and teacher feedback sessions) and reflect on differences between the two rubric versions. Table 3. Study Sample

District 1

(3 schools) District 2




(2 schools) Total

(13 schools) Evaluators 2 2 2 2 1 8 HS teachers 0 4 4 0 1 9 MS teachers 8 4 4 0 1 17 Elementary 4 4 6 8 1 20 Total teachers

12 12 14 8 3 49

Based on feedback from 32 teachers, teaching experience ranged from less than one year of teaching to 26 years of experience. Eleven teachers had fewer than 5 years of experience, 12 had from 5 to 15 years of experience, and 9 had more than 15 years of experience.

Data Collection Methods

Teachers, study evaluators, and school evaluators all served as important data sources for this study. They represented different, yet integral, perceptions of how each rubric is applied for providing clear and actionable feedback for educator effectiveness. Researchers used the following data collection activities to address the study’s research questions.


7

Participant Survey

Researchers developed four online surveys: a teacher survey, two evaluator surveys and a school evaluator survey. The purpose of the surveys was to assess participants’ perceptions of the clarity, accuracy, fairness, ease of use, limitations, credibility, and production of actionable feedback of the 4-point rubric versus the 5-point rubric. Evaluators received a post-observation survey after each observation. All participants received post-feedback surveys after the final teacher feedback sessions (see Appendix C). Researchers conducted descriptive analyses of survey data as well as inferential analyses (t-tests) to assess statistical differences in perceptions between the 4-point and 5-point rubrics, where applicable. Evaluator Focus Group Researchers developed a protocol to conduct a focus group including all eight evaluators (see Appendix D). The purpose of the focus group was to engage evaluators in dialogue and reflections on the differences in usefulness of the 4-point versus the 5-point rubrics in observing teacher practice and providing teachers with accurate, actionable feedback to improve their practice. The focus group allowed researchers to probe deeper on evaluators’ survey findings and any issues and concerns that surfaced during the study. Researchers conducted a systematic analysis of the qualitative data to capture the perceptions and experiences of the evaluators. Teacher Evaluation Ratings Each evaluator observed each teacher twice with the 4-point and the 5-point rubrics, for a total of four completed rubrics per teacher. Given the study’s limited timeframe, these observations were not intended to capture teacher growth from the first to second observation, but rather served as two data points for the same rubric and same teacher. Because the study’s purpose was to compare the distribution of ratings between the 4-point and 5-point rubrics, rather than differences in ratings from the first to second observation, researchers added together ratings for the first and second observations for each teacher and then compared the 4-point and 5-point totals for the final analyses.

Following the User’s Guide for the teacher rubric, a teacher’s rating for an element reflects the highest performance rating level for which an evaluator has observed—or checked—all professional practices at that level and at all levels below that level. Therefore, for each element, a teacher receives a rating of Not Evident, Partially Proficient, Proficient, Accomplished or Exemplary on the 5-point rubric and a rating of Not Evident, Partially Proficient, Proficient, or Exemplary on the 4-point rubric.

Procedures

Researchers conducted a 60-minute webinar with evaluators to review the protocol for conducting observations, management of evaluation data, and expectations for participation in study data collection activities. Researchers also developed and recorded a ten-minute webinar for teachers to watch prior to participation. Researchers developed and provided supporting


8

materials for evaluators to conduct their work, including a User’s Guide for the Rubric Comparison Study, a pre-observation protocol, a post-observation protocol, summary reports to provide the teacher and school evaluator on the results of each observation, and a final feedback report form. Each evaluator used each rubric, the 4-point and the 5-point, for each teacher observed, half starting with the 4-point rubric and then completing the 5-point, the other half starting with the 5-point and then completing the 4-point. For the second observation, evaluators started with the rubric they used second during the prior observation. Teacher Observation Process

Teacher observations focused on Teacher Quality Standards I-III of the evaluation

rubrics, including observable and non-observable professional practices. The teacher evaluation process consisted of four key activities described below.

Pre-Observation

Teachers completed a pre-observation planning form with information about the lesson to be observed and the characteristics of the learning environment. Teachers voluntarily completed the 5-point rubric as a self-assessment prior to their first observation and selected at least two practices that they would like feedback on. Teachers determined what evidence they believed would be useful in contributing to the assessment of the practices such as lesson plans, student work, etc. Together teachers and evaluators in the pre-observation conference clarified the objectives, background, and context for the lesson to be observed. The evaluator may have suggested additional evidence that would aid in rating the non-observable practices. Together they finalized the schedule of the lesson to be observed and a schedule for the post-observation conference.

Across the two observations, evaluators found the information collected through the pre-observation protocol useful 81% of the time. On the final survey, 94% of teachers reported completing the self-assessment and 87% selected practices for observation. Teacher Observation 1

Evaluators conducted the first classroom observation (at least a full lesson) with the school evaluator observing. Evaluators used both rubrics as they observed—half using the 4-point rubric first and half using the 5-point first. On the same day as the observation, each evaluator met with the teacher for a brief post-observation conference to clarify the teacher’s assessment of the lesson just concluded. It is important to note that because completion of the rubric requires reflection and additional evidence from non-observables, evaluators did not provide evaluation ratings or actionable feedback for professional growth after these observations. The evaluators completed a brief survey concerning the use of the rubrics for the first observation. Teacher Observation 2

The second classroom observation followed the same protocols as the first classroom observation. Based on availability of evaluators and teachers, the time between the first and second observations ranged from one to four weeks. Both pre- and post-processes were the


9

same. Evaluators who had used the 4-point rubric first during the initial observation used the 5-point rubric first for the second observation, and visa versa. The evaluators again completed a brief survey concerning the use of the rubrics. Integrated into both school visits 1 and 2 were opportunities to connect with the school evaluator to get further information on the non-observable aspects of the rubric for the specific teachers. Of the six school evaluators who responded to the survey, 17% joined in the observations of all participating teachers in a school and 83% participated in some of the observations of participating teachers.

On the post observation survey 47% of the evaluators responded that they always

collected artifacts for the observations, 33% usually collected them, and 20% seldom collected them. Nonetheless, evaluators said about two-thirds of the time there were non-observables for which they had insufficient evidence to rate.

Post Observation Feedback to Teachers

This third school visit focused on providing feedback to the teachers on the two sets of ratings generated from the 4-point and 5-point rubrics. Using the rubric data gathered from their observations, evaluators completed summary reports of the element ratings for both the 4-point and 5-point rubrics and developed a feedback session report based on the selection of approximately four professional practices to discuss with the teachers (which included the two practices selected by teachers prior to the observations). The feedback sessions lasted an average of 39 minutes and the school evaluator was present about half of the time. Immediately after the feedback session the teacher and school evaluator were asked to complete a brief online survey about the two rubrics and whether each provided clear, accurate, and actionable feedback.


10

Study Findings

This section presents findings based on the study’s two central research questions related to the robustness in distribution ratings for each rubric and their ability to yield clear and actionable feedback for teacher growth.

How Did the 4-Point and 5-Point Rubric Ratings Compare?

The first research question for the study was Which rubric version enables greater differentiation among educator performance, and thus yields a more robust distribution of educator evaluation ratings? To compare the rating distributions between rubrics, researchers ran frequency counts of teacher ratings for each element—that is, the number of teachers with a rating at each level (e.g., Not Evident, Partially Proficient, Proficient) for each element. Because the purpose was to compare the distribution of ratings between the two rubrics, rather than for each observation, researchers examined the frequencies of element ratings for the 4-point and 5-point rubrics for the two observations combined. Researchers also calculated the mode for each element to identify which performance level teachers received most often.

Table 4 presents the number of teachers receiving a rating at each performance level for each element. Frequencies highlighted in blue indicated the mode for the 4-point rubric and those highlighted in red reflect the mode for the 5-point rubric. The bottom three rows present the total frequency count, the average frequency count across elements and the range for each performance level by rubric. Table 4. Frequency and Chi-square Values by Element and Rating Level (4-pt mode, 5-pt mode)

Element Not evident Partially Proficient Proficient Accomplished Exemplary

4pt 5pt 4pt 5pt 4pt 5pt 5pt 4pt 5pt 1a 1 0 51 52 43 19 25 3 2 1b 23 22 61 61 14 9 6 0 0 1c 56 60 29 26 11 9 1 2 2 1d 1 0 49 50 44 36 12 4 0 1e 7 6 74 78 9 13 1 8 0 1f 3 1 19 16 70 51 26 6 4 2a 3 1 23 25 18 4 18 54 50 2b 5 3 29 28 57 27 31 7 9 2c 1 0 67 64 30 16 16 0 2 2d 4 3 53 50 38 39 4 3 2 2e 2 2 62 61 19 20 4 15 11 2f 3 1 38 39 34 16 36 23 6 3a 5 3 64 67 28 24 4 1 0 3b 2 1 51 52 44 40 5 1 0


11

Element Not evident Partially Proficient Proficient Accomplished Exemplary

4pt 5pt 4pt 5pt 4pt 5pt 5pt 4pt 5pt 3c 1 0 40 42 37 55 1 20 0 3d 22 22 58 56 17 11 8 1 1 3e 3 1 51 54 36 18 24 8 1 3f 9 10 65 63 6 12 3 18 10 3g 1 0 40 42 54 30 24 3 2 3h 3 3 75 74 18 19 2 2 0

Total 155 139 999 1,000 627 468 251 179 102

Average 7.75 6.95 49.95 50.0 31.35 23.4 12.55 8.95 5.1

Range 1-56 0-60 19-75 16-78 6-70 4-55 1-31 0-54 0-50 Note: Shading marks modes for the 4-point and 5-point rubrics fall outside the same performance level.

When looking at the total frequency counts across elements (third row from the bottom

in Table 4) and the average frequencies for each performance level (second row from the bottom), findings indicate that most often teachers received a rating of Partially Proficient.2 This was comparable between the 4-point and 5-point rubrics, with the 4-point rubric having a total of 999 ratings (mean of 50 ratings) compared to 1000 ratings (mean of 50 ratings) for the 5-point rubric. The Not Evident frequency ratings were similar between rubrics. Figure 1 presents a visual depiction of the average rating frequencies by rubric and performance rating level.

Figure 1. Average frequencies of ratings by performance level and rubric.

2 For two elements the majority of ratings deviate from Partially Proficient or Proficient. For Element 1c, most teachers received ratings of Not Evident on both rubrics. The element focuses on teachers demonstrating knowledge of mathematics and promoting student learning in specific mathematics domains, regardless of the class they teach. Therefore, reading, science, social studies, and physical education teachers are held accountable to professional practices related to mathematical concepts. For Element 2a, most teachers received ratings of Exemplary on both rubrics, which focuses on teachers fostering a predictable learning environment with positive relationships among students and adults.

7.0

50.0

23.4 12.6

5.1

7.8

50.0

31.4

9.0

0

10

20

30

40

50

60

Not evident Partially Pro6icient

Pro6icient Accomplished Exemplary

Average frequency

Element Performance Rating Levels

5 point 4 point


12

The total number of ratings at or above the Proficient performance level was 821 for the 5-point rubric and 806 for the 4-point rubric. Although these totals were comparable, the distribution of ratings across the Proficient, Accomplished (5-point only) and Exemplary performance levels varied by rubric (see

Figure 2). There were a higher number of ratings at the Proficient performance level for the 4-point rubric (627) than the 5-point rubric (468), 251 ratings at the Accomplished performance level for the 5-point, and 179 and 102 ratings at the Exemplary level for the 4-point and 5-point rubrics, respectively.

Figure 2. Comparison of rubric distributions for total number of ratings at or above Proficient. Rubric Modal Differences

The 4-point and 5-point modes for each element fall within the same performance level for all elements except for three. This suggests that the performance rating level teachers’ received most often was consistent between the two rubrics in most cases. For example, if the most frequently received rating on the 4-point scale was Partially Proficient, that same rating was likely to be the most frequently received rating on the 5-point scale as well. The three elements where the modes for the two rubrics fall outside the same performance level are shaded in Table 4 and more fully described in the tables below.

As shown in Table 5, Element 2b reads, “Teachers demonstrate a commitment to and

respect for diversity, while working toward common goals as a community and as a country.” Teachers most often received a rating of Proficient on the 4-point rubric and Accomplished on the 5-point rubric. The Proficient and Exemplary professional practices are the same for the 4-point and 5-point rubrics. With the inclusion of the Accomplished category on the 5-point rubric, evaluators were able to identify evidence related to students respecting the backgrounds of fellow students more often than they were able to observe students actively seeking a variety of perspectives to complete group assignments. As such, the 5-point rubric included an incremental progression of student behavior that allowed evaluators to recognize a preliminary impact of teacher effectiveness.

627

468 251

179

102

0 100 200 300 400 500 600 700 800 900

4-point rubric

5-point rubric

Rating Frequency

Proficient Accomplished Exemplary


13

Table 5. Professional Practices by Mode and Rubric for Element 2b Element 2b: Teachers demonstrate a commitment to and respect for diversity, while working toward common goals as a community and as a country.

Proficient 4-point (mode = 57)

Accomplished 5-point (mode = 31)

Exemplary 4-point & 5-point

The Teacher establishes routine processes that result in: O A strong sense of community among students. O Effective interactions among students. O Respect for individual differences. O Positive social relationships.

Students: O Respect the

backgrounds of fellow students.

Students: O Actively seek a

variety of perspectives to complete group assignments.

Note: The Proficient and Exemplary professional practices are the same for the 4-point and 5-point rubrics. The 4-point rubric does not include the Accomplished performance level.

As shown in Table 6, Element 3c states, “Teachers demonstrate a rich knowledge of current research on effective instructional practices to meet the developmental and academic needs of their students.” Teachers most often received a rating of Partially Proficient on the 4-point rubric and Proficient on the 5-point rubric. Both the 4-point and 5-point rubrics share the same professional practices under Partially Proficient, and Proficient, but had slightly different professional practices under Exemplary. For the 5-point rubric, evaluators only gave a rating once above Proficient. For the 4-point rubric, there were 37 ratings at Proficient and an additional 20 Exemplary ratings, compared to no teachers receiving the Exemplary rating on the 5-point rubric. Comparing the professional practices under Accomplished on the 5-point rubric and Exemplary on the 4-point rubric, one can see that the two practices under Exemplary on the 4-point rubric might be more straightforward to observe than the three practices under Accomplished. For instance, evaluating if students connect lesson objectives to prior knowledge in a significant and meaningful way, requires a high-level of inference from the evaluator, whereas students responding to a lesson that supports their growth might require a lower-level of inference based on what evaluators can observe in the classroom.

Table 6. Professional Practices by Mode and Rubric for Element 3c Element 3c: Teachers demonstrate a rich knowledge of current research on effective instructional practices to meet the developmental and academic needs of their students.

Partially Proficient 4-point (mode = 40) 5-point (freq = 42)

Proficient 5-point (mode = 55) 4-point (freq = 37)

Accomplished 5-point

(freq = 1)

Exemplary 4-point

(freq = 20) 5-point

(freq = 0) The Teacher: O Makes lesson

objectives clear to the students.

O Employs a variety of instructional strategies.

O Provides instruction that requires critical thinking, problem solving, and performance skills.

O Checks for student understanding of content.

The Teacher: O Facilitates

learning by supporting students as they learn new material.

O Sets the expectation that students will reflect on and communicate about their learning.

Students: O Articulate the

importance of the lesson objective.

O Connect lesson objective to prior knowledge in a significant and meaningful way.

O Describe their level of performance in relation to lesson objectives.

Students: O Recognize and

respond to instruction that supports their growth.

O Communicate with the Teacher about their learning.

Students: O Apply skills

and knowledge learned in the classroom.

O Articulate the ways in which they learn most effectively.

Note: The Partially Proficient and Proficient professional practices are the same for the 4-point and 5-point rubrics.


14

As shown in Table 7, Element 3g states, “Teachers communicate effectively, making

learning objectives clear and providing appropriate models of language.” Teachers most often received a rating of Partially Proficient on the 5-point and Proficient on the 4-point. This is an interesting distribution because the professional practices for Partially Proficient and Proficient are the same for both rubrics. On the 5-point rubric there were 30 ratings of Proficient and 26 ratings above Proficient (24 of those at the Accomplished level). The ratings from the 5-point rubric were distributed more evenly across three performance levels, reflecting a more incremental progression of student behaviors to observe.

Table 7. Professional Practices by Mode and Rubric for Element 3g Element 3g: Teachers communicate effectively, making learning objectives clear and providing appropriate models of language. Partially Proficient

5-point (mode = 42)

Proficient 4-point

(mode = 54)

Accomplished 5-point

(freq = 24)

Exemplary 4-point

(freq = 3) 5-point

(freq = 2) The Teacher: O Models

effective communication skills.

O Sets expectations and employs strategies so students can communicate effectively.

The Teacher: O Models and teaches

effective skills in listening, presenting ideas, and leading discussions.

O Provides opportunities for students to practice communication skills.

Students: O Apply effective

written and oral communication skills in their work.

O Demonstrate a respectful and sensitive approach toward fellow students and Teachers.

Students: O Demonstrate

effective oral communicat-ion skills with fellow students and the Teacher.

O Model formal communicat-ion in academic settings.

Students: O Participate in

teams in ways that build trust and ownership of ideas among team members.

O Model formal communicat-ions in academic settings.

Note: The Partially Proficient and Proficient practices are the same for the 4-point and 5-point rubrics. The 4-point rubric does not include the Accomplished performance level. Rubric Distribution Differences

The aforementioned cases describe differences in modes when occurring outside of the same performance level. Researchers conducted chi-square analyses to examine if statistically significant differences existed between the distribution of ratings across performance levels for the 4-point rubric and the 5-point rubric. The chi-square analyses consider the number of units for each performance level and then compare the units within each level for the 4-point and the 5-point rubrics (e.g., number of ratings of Proficient on the 4-point compared to the number of Proficient ratings on the 5-point). Because the 4-point rubric does not have the Accomplished category, researchers used a conservative, unbiased approach and removed this performance level from the chi-square analyses.3

3 An alternate approach might have been to combine the Accomplished and Exemplary 5-point ratings; however, that would skew the comparisons since Exemplary in the 4-point rubric was constrained to have only the number of professional practices (49) as in the Exemplary category of the 5-point rubric, whereas Accomplished adds 47 more practices to the 5-point for a total of 96. These differences would not reflect differences in the distributions, but differences in the rubric structures. Therefore, the statistical comparisons of rating distributions between the two rubrics are for the Not Evident, Partially Proficient, Proficient, or Exemplary performance levels.


15

Of the 20 elements, the five elements in Table 8 had statistically significant differences in the distribution of ratings between the 4-point and 5-point rubrics. The differences in ratings occurred in the Proficient and Exemplary categories, or in two cases, in both. For three elements (2a, 2f, and 3e) differences in ratings were likely offset by the large number of Accomplished ratings on the 5-point rubric, in which case one would expect fewer ratings at the Proficient or Exemplary levels. For the other two elements (1e and 3c, described previously), there was only one rating above Proficient on the 5-point scale. For these two elements, this points to differences in the wording and progression of professional practices from Proficient to Exemplary on the 4-point and Proficient to Accomplished on the 5-point, whereby the evidence supporting the 4-point professional practices related to students (as framed) might be slightly easier to observe in a classroom setting.

Table 8. Statistically Significant Chi-Square Results of Rating Distributions

Element Chi-Square

Value Rating

Frequency 5-point

Accomplished Rating Frequencya 4-point 5-point

1e. Teachers Develop Lessons That Reflect The Interconnectedness Of Content Areas/Disciplines.

8.905* Exemplary

1 8 0

2a. Teachers foster a predictable learning environment in the classroom in which each student has a positive, nurturing relationship with caring adults and peers.

8.412*

Proficient

18 18 4

2f. Teachers create a learning environment characterized by acceptable student behavior, efficient use of time, and appropriate intervention strategies.

9.858*

Proficient

36 34 16

Exemplary 23 6

3c. Teachers demonstrate a rich knowledge of current research on effective instructional practices to meet the developmental and academic needs of their students.

24.566**

Proficient

1 37 55

Exemplary

20 0

3e. Teachers establish and communicate high expectations for all students and plan instruction that helps students develop critical thinking and problem solving skills.

9.364*

Proficient

24 36 18

*p < .05, **p < .01 a The Accomplished category, not included in the chi-square analysis, only pertains to the 5-point rubric


16

Rating Distribution Summary For five of the 20 rating distributions (25%) across elements there were statistically

significant differences between the 4-point and 5-point rubrics. The differences between the frequency distributions occur within the Proficient and Exemplary performance levels. For these five elements, the inclusion of the Accomplished category on the 5-point rubric could be attributed to the difference between rating distributions. In three cases, there were a substantial number of teacher ratings at the Accomplished level, thus offsetting the number of ratings at the Proficient and Exemplary levels on the 5-point rubric. In two cases, the study evaluators might have more easily observed and identified the professional practices under Exemplary in the 4-point rubric than the professional practices under Accomplished in the 5-point rubric.

Which Rubric Provided Clearer and More Actionable Feedback?

As part of the Model Evaluation System the teacher rubric is intended not only as a measure of teacher effectiveness, but also as a tool to engage teachers in their professional growth over time. Accordingly, the purpose of the rubric is both summative and formative as teachers receive clear and actionable feedback to guide their professional development. As such, the second study research question is: Which rubric version provides clearer and more actionable feedback for educator improvement?

Findings for this research question are based on the perceptions and experiences of

evaluators collected through post-observation surveys, a post-feedback session survey, and a focus group. All eight evaluators responded to the surveys4 and participated in the focus group. Researchers also collected additional feedback through online surveys of teacher participants and school evaluators. Response rates were 74% for teachers and 55% for school evaluators.

Across the three participant groups—teachers, school evaluators, and evaluators—

respondents rated if they Strongly Agree (4), Agree (3), Disagree (2), or Strongly Disagree (1) with four statements related to each rubric providing (a) clear feedback, (b) actionable feedback, (c) accurate ratings, and (d) a clear pathway for professional growth. Thus a mean rating of 2.5 would indicate that, on average, participants neither fully agreed nor disagreed with the statement. Overall, the mean value for the 5-point rubric exceeded that of the 4-point mean on all four teacher ratings, two of three school evaluator ratings, and three of the four evaluator ratings (see Table 9, Table 10, and Table 11). These differences in ratings for the 4-point rubric and 5-point rubric were only statistically significant for three items on the teacher survey (see Table 9). This indicates that teachers perceived the 5-point rubric as providing more actionable feedback for growth, reflecting their professional practices more accurately, and promoting a clearer pathway for professional growth than the 4-point rubric. It is important to note that averages for the latter two items fell between Disagree and Agree ratings, whereas the ratings regarding clear or actionable feedback fell between Agree and Strongly Agree. 4 One evaluator completed the post-feedback session survey twice, so findings are based on nine responses, rather than eight. All surveys were anonymous to encourage candid responses, so researchers were unable to identify or remove the extra survey received.


17

Table 9. Teacher Survey Ratings of 4-Point and 5-Point Rubrics

Item 4-point Mean

5-point Mean t Approx.

df p

value I received clear feedback on professional practices based on the rubric.

3.16 3.32 1.976 30 .057

I received actionable feedback for growth based on the rubric. 3.13 3.4 3.247 29 .003**

My ratings of the rubric reflected my professional practices accurately.

2.45 2.77 2.752 30 .010**

The scale of the rubric promotes a clear pathway for professional growth.

2.58 2.97 2.834 30 .008**

*p < .05, **p < .01

In general, school evaluators’ ratings on items reflect comparable levels of agreement to teacher and evaluator ratings. School evaluators perceived no difference in the feedback teachers received based on the 4-point and 5-point rubrics. Table 10. School Evaluator Ratings of 4-Point and 5-Point Rubrics

Item 4-point Mean

5-point Mean t Approx. df p

value Teachers received clear feedback on professional practices based on the rubric. †

3 3 - - -

Teachers received actionable feedback for growth based on the rubric.

3.17 3 1.0 5 .363

Teachers received accurate feedback. Item not included on survey The scale of the rubric promotes a clear pathway for professional growth.

2.5 2.83 1.0 5 .363

*p < .05, **p < .01 † The correlation and t value cannot be computed because the standard error of the difference is 0.

Although evaluators’ agreement ratings were higher for the 5-point rubric than the 4-point rubric on items related to actionable feedback, the accuracy of ratings, and promoting a clear pathway for growth, these differences were not statistically significant. Similarly, evaluators had slightly higher ratings in favor of the 4-point rubric providing clearer feedback than the 5-point, but the difference was not statistically significant.


18

Table 11. Evaluator Ratings of 4-Point and 5-Point Rubrics

Item 4-point Mean

5-point Mean t Approx.

df p value

The scale of the rubric allowed me to provide clear feedback on professional practices.

3.0 2.75 -.80 7 .451

The scale of the rubric allowed me to provide actionable feedback for growth.

2.78 3.11 2.0 8 .081

The scale of the rubric allowed me to rate professional practices accurately.

2.56 2.89 1.155 8 .282


2.33 2.78 1.315 8 .225

*p < .05, **p < .01 In addition to statements requiring ratings of agreement or disagreement, participants

were asked to choose between the two rubrics. On the post-feedback session surveys, teachers, evaluators, and school evaluators indicated which of the two rubrics provided clearer and more actionable feedback. As shown in Figure 3, 52% of teachers, 44% of evaluators, and 67% of school evaluators selected the 5-point rubric compared to 16% of teachers, zero evaluators, and 17% of school evaluators who chose the 4-point rubric. Additionally, 29% of teachers and 56% of evaluators indicated that neither rubric provided clearer and more actionable feedback across elements, suggesting they did not perceive a large enough difference between the rubrics to choose one over the other. Of the school evaluators responding, 17% indicated they did not have enough information to determine if one rubric provided clearer and more actionable feedback than the other.

Figure 3. Participant ratings of rubrics providing clear and actionable feedback.

0 10 20 30 40 50 60 70 80 90

100

No difference 4-point 5-point

Per

cen

t

Which rubric provided clearer and more actionable feedback?

School Evaluator (n = 6) Teacher (n = 30) Evaluator (n = 9)


19

Which Rubric Did Participants Prefer?

Finally, all three participant groups indicated which rubric they would prefer to use in

teacher performance evaluations. Evaluators had the opportunity to respond to this question on the post-observation survey, the post-feedback session survey, and during the focus groups. As shown in Figure 4, 45% of teachers and 50% of school evaluators, prefer the 5-point rubric, whereas 16% and 17%, respectively, prefer the 4-point rubric. On both the post-observation and final feedback surveys, evaluators were evenly divided between preferring the 4-point and 5-point rubrics. Among teachers, 13% indicated either rubric would be okay for teacher performance evaluations.

Figure 4. Participant ratings of preferred rubric.

A percentage of teachers (26%) would prefer neither rubric for teacher performance

evaluations. Also preferring neither rubric were 33% of school evaluators. When interpreting this finding, it is important to consider that the educators who participated in the study were unfamiliar with the Model Evaluation System and only experienced isolated classroom observations independent of a comprehensive professional growth plan. A portion of the responses may have been from lack of information about the intent of SB 191, the Model Evaluation System process and how the rubric represents one tool among many in supporting teachers’ professional growth.

Teachers and school evaluators also provided comments to explain their choices.

Comments from six teachers and five school evaluators who preferred the 5-point rubric explained that it gave more room for growth and improvement and provided more information

0

10

20

30

40

50

60

70

80

90

100

Either Neither 4 point 5 point

Per

cen

t

Which rubric do you prefer? School Evaluator (n = 6) Teacher (n = 31) Evaluator Final (n = 9) Evaluator Observation (n = 16)


20

about instruction. Two teachers who preferred the 4-point rubric indicated that it left less room for subjective judgments and was more concise. Three teachers who preferred neither rubric believed the rubrics did not address their instructional focus, such as physical education, art or special education. Six teachers commented that many more observations are necessary to make these judgments. Four school evaluators mentioned difficulties in observing each and all of the practices that are on the rubrics.

Eleven percent of evaluator responses on the final survey and 25% of evaluator

responses on the post-observation survey indicate that they would prefer neither rubric in conducting teacher performance evaluations. Their responses to several open-ended items about rubric preferences are described below.

Additional Study Evaluator Comments

Evaluators provided additional comments on the post-feedback session survey regarding the strengths and limitations of each rubric. Evaluators reported strengths of the 4-point rubric as being shorter (i.e., fewer professional practices to observe), more concise, and having a faster track to Exemplary. Others noted, however, that a limitation of the 4-point is that it is difficult for a teacher to improve beyond Proficient and thus to attain Exemplary. Strengths of the 5-point rubric included the increased steps for growth, the additional practices described, and better representation of the changes sought in order to improve teaching by focusing on student learning behaviors. The only limitation of the 5-point rubric that evaluators mentioned was its length.

In the focus group evaluators confirmed these strengths and limitations. They noted that the 5-point rubric provided more opportunities for rating teachers who are in the upper range of the rating scale; one evaluator commented that the 4-point scale was like having a grading scale with no “B.” Evaluators did not perceive a difference between rubrics in rating teachers who were Proficient or less than Proficient. As to which rubric they would recommend for use, three said the 4-point because it is simpler and made no difference for the less accomplished teachers. Five evaluators favored the 5-point, with three evaluators explaining it is preferable because it allows for more professional growth. Two evaluators reiterated that the changes in student learning behaviors currently sought are better represented in the 5-point rubric. Finally they noted that not all principals they worked with are prepared to do teacher observations of this type and that it will be important for training to be provided.

Study Limitations

As noted earlier, there are limitations to this study that readers should have in mind. First, given the timeline of the study, teacher evaluation ratings only represented teacher performance at mid-year. The study did not capture a more comprehensive and cumulative body of evidence that typically would be collected over an entire school year and through multiple observations. Second, because the study only ran through mid-year, it was challenging for evaluators to collect information pertaining to the non-observable professional practices in the rubrics. This might have influenced the accuracy of the ratings on those practices. The


21

constraints of this study precluded researchers from controlling for individual evaluator differences such as communication style, personality, and depth and clarity of actionable feedback to teachers. However, researchers requested that evaluators only speak from the information and professional practices in the respective rubrics and not deviate from the framework of the rubrics. Evaluators varied which rubric was discussed first among the teachers each evaluator observed. By having each evaluator use both rubrics the evaluators were able to make comparisons without the potentially contaminating influence of evaluator differences in personality, communication style, or personal understandings of the nature of feedback.

The limited focus of the study, simply comparing the scale of the two rubrics and not

including other aspects of the Model Evaluation System, might have raised other issues for teachers who were not trained on the System and might have had negative feelings about classroom observations or teacher evaluations from past experiences. It is not known if and how such responses might have influenced the study results.

Finally, budget limitations precluded a larger sample size that would have permitted a

greater variety of teachers at each school level. Survey response rates varied: teachers (74%), school evaluators (50%), and study evaluators (100%). Response rates of less than 80% can result in findings that might not fully represent the targeted sample, and therefore could result in biased survey estimates.5

Summary and Discussion

This study compared a 4-point and 5-point version of the CDE teacher evaluation rubric to measure if one version provides a more effective rating structure of educator effectiveness and clearer and more actionable feedback for guiding professional growth. CDE will use the findings from this study to guide decisionmaking for rubric refinement. CDE will use the following guideposts related to robust distributions and clear and actionable feedback for determining if one of these rubrics is more effective than the other.

Robust distribution: The underpinning hypothesis assumes there will be no difference in the distribution of ratings between the 4-point and the 5-point scales. Chi-square analyses will determine if statistically significant differences exist using the standard p value of <. 05. Should differences in rubric distributions not be statistically significant, then both rubrics will be thought to yield robust distributions.

A robust distribution, in this case, is defined by a rubric that enables greater differentiation among educator effectiveness. A distribution of teacher performance ratings is robust if it distinguishes among teachers at different performance levels and with different professional growth needs. Analyses revealed statistically significant differences between the 5 Johnson T, Owens L. Survey response rate reporting in the professional literature. Paper presented at the 58th Annual

Meeting of the American Association for Public Opinion Research, Nashville, May 2003. Available at: http://www.srl.uic.edu/publist/Conference/rr_reporting.pdf. ,Accessed January 21, 2013.


22

distributions on the 4-point and 5-point rubrics on five of the 20 elements. In reviewing the distributions for these five elements, researchers found the differences between the frequencies occur within the Proficient and Exemplary performance levels. Across all five elements this could be attributable to the inclusion of the Accomplished category on the 5-point rubric. In three of the five cases, differences are likely a result of a substantial number of teachers rated at the Accomplished level, thereby yielding a more diverse distribution of ratings across the Proficient and Exemplary levels for the two rubrics. In two cases, differences in rating distributions could be attributed to the nature of professional practices under Accomplished in the 5-point rubric and Exemplary in the 4-point rubric, whereby professional practices in the latter might be identified more easily during a classroom observation than the former.

Upon further examination of rating distributions across the 4-point and 5-point rubrics,

findings reveal the distribution for both rubrics is highest at the Partially Proficient performance level with rubrics having an almost equal number of ratings at this level (4-point = 999; 5-point = 1,000). The next highest frequencies are at the Proficient level and fewer frequencies are at the Not Evident and Exemplary levels. Both rubrics were sensitive to detecting teachers performing at both ends of the scales. It is of note that the number of ratings at or above the Proficient level was comparable between the two rubrics, with 821 ratings for the 5-point rubric and 806 ratings for the 4-point rubric. However, the distribution of those ratings across the Proficient, Accomplished (5-point only), and Exemplary rating levels varied. Because of the inclusion of the professional practices at the Accomplished rating level on the 5-point rubric, teachers had more opportunities to demonstrate educational effectiveness on a wider continuum of growth than on the 4-point rubric.

These findings suggest that the inclusion of Accomplished in the 5-point scale does not

positively bias ratings of teachers, but rather provides a more sensitive and incremental measure of their progress and growth beyond the Proficient level. Because the professional practices shift from requiring teacher-oriented evidence to student-oriented evidence past the Proficient level on both rubrics, the Accomplished level in the 5-point rubric provides more instances to measure gradual impacts of teacher effectiveness on students.

Clear and actionable feedback: Results from the teacher, school evaluator, and

evaluator surveys show that 65% or more of the respondents indicate that one of the rubrics yields better feedback. Results from the focus group supports this finding, or if the survey responses are not definitive, results from the focus group will provide differential evidence if differences are reported.

Findings from the surveys indicate that 52% of teachers, 67% of school evaluators, and

44% of evaluators believe the 5-point rubric provides clearer and more actionable feedback than the 4-point rubric.6 Of those responding, it is important to note that 29% of teachers and 33% of evaluators indicated neither rubric provided clear and actionable feedback. However, there were statistically significant differences in teachers’ ratings of the 4-point and 5-point rubrics. Specifically, teachers rated the 5-point rubric higher than the 4-point rubric on providing actionable feedback for growth, reflecting their professional practices accurately, and promoting a clear pathway for professional growth.

6 Due to the small sample size for this study, caution is warranted in interpreting these findings.


23

When asked which rubric evaluators preferred to use for teacher performance evaluations, they were divided between the 4-point and 5-point rubrics on the survey, but favored the 5-point rubric (5 to 3) during the focus group. The difference in rubric preferences between the survey and focus group data could be attributed to the opportunity for study evaluators to further process their experiences during the focus group dialogue. Those supporting the 4-point rubric thought it was simpler and more concise, whereas those favoring the 5-point rubric thought it provided more opportunities for rating accomplished teachers who were above the Proficient level, and it was more conducive to promoting professional growth. Asked the same question, more teachers and school evaluators said they preferred the 5-point rubric than the 4-point rubric.

The results of this study comparing a 5-point rubric for evaluating teacher effectiveness

to a 4-point rubric suggest that either rubric could be used to differentiate teacher performance. The 5-point rubric, however, allows for a broader differentiation of teachers and steps to improvement of practice, and includes professional practices that have the expectation that teachers will influence student learning practices.


24

Appendix A: 4-Point and 5-Point Rubrics

Rubric for Evaluating Colorado’s Teachers

FOUR POINT RUBRIC Effective Teachers in the state of Colorado have the knowledge, skills, and commitments needed to provide excellent and equitable learning opportunities and growth for all students. They strive to support growth and development, close achievement gaps and to prepare diverse student populations for postsecondary and workforce success. Effective Teachers facilitate mastery of content and skill development, and employ and adjust evidence-based strategies and approaches for students who are not achieving mastery and students who need acceleration. They also develop in students the skills, interests and abilities necessary to be lifelong learners, as well as for democratic and civic participation. Effective Teachers communicate high expectations to students and their families and utilize diverse strategies to engage them in a mutually supportive teaching and learning environment. Because effective Teachers understand that the work of ensuring meaningful learning opportunities for all students cannot happen in isolation, they engage in collaboration, continuous reflection, on-going learning and leadership within the profession. The Teacher Quality Standards outline the knowledge and skills required of an effective Teacher and will be used to evaluate Teachers in the state of Colorado. All School Districts and BOCES shall base their evaluations of licensed classroom Teachers on the full set of Teacher Quality Standards and associated detailed Elements included below, or shall adopt their own locally developed standards that meet or exceed the Teacher Quality Standards and Elements. Quality Standard I: Teachers demonstrate mastery of and pedagogical expertise in the content they teach. The elementary Teacher is an expert in literacy and mathematics and is knowledgeable in all other content that he or she teaches (e.g., science, social studies, arts, physical education, or world languages). The secondary Teacher has knowledge of literacy and mathematics and is an expert in his or her content endorsement area(s).


Element a: Teachers provide instruction that is aligned with the Colorado Academic Standards; their District's organized plan of instruction; and the individual needs of their students.

There is inadequate evidence that the Teacher: O Plans instruction on a

daily basis.



The Teacher: Develops lesson plans based on: O Colorado Academic

Standards. O District’s plan of

instruction. O Student needs.




Standards. Collaborates with other

school staff to vertically and horizontally articulate the curriculum.

. . . and Students:

O Interact with the

rigorous and challenging content in meaningful ways.

Identify gaps in learning in student journals and learning logs.



25

Quality Standard I: Teachers demonstrate mastery of and pedagogical expertise in the content they teach. The elementary Teacher is an expert in literacy and mathematics and is knowledgeable in all other content that he or she teaches (e.g., science, social studies, arts, physical education, or world languages). The secondary Teacher has knowledge of literacy and mathematics and is an expert in his or her content endorsement area(s).


Element b: Teachers demonstrate knowledge of student literacy development in reading, writing, speaking and listening.

This section describes professional practices that should be demonstrated by ALL TEACHERS, regardless of grade level or subject taught.

There is inadequate evidence that the Teacher: O Emphasizes literacy

connections while teaching content other than reading, English, and/or language arts.

O Has knowledge of how to integrate literacy across content areas.

The Teacher: Makes complex reading accessible to students by: O Making necessary

adjustments to content.

O Integrating literacy skills and knowledge into lessons.

O Demonstrates a deep understanding of literacy content and skills.

. . . and The Teacher: Provides literacy instruction that enhances: O Critical thinking and

reasoning. O Information literacy. O Collaboration. O Self-direction. O Innovation.

O Focuses lessons on the

reading of complex materials.

. . . and Students: O Communicate orally

and in writing at levels that meet or exceed expectations for their age, grade, and ability level.

O Exceed expectations in critical thinking and problem solving skills.

O Listen attentively to

the Teacher and peers, understand what is communicated and apply it to the current lesson.

Element c: Teachers demonstrate knowledge of mathematics and understand how to promote student development in numbers and operations, algebra, geometry and measurement, and data analysis and probability.

This section describes professional practices that should be demonstrated by ALL TEACHERS.

There is inadequate evidence that the Teacher: O Includes math topics in

discussions that do not have math as the primary focus.

O Promotes and encourages students to make explicit math connections across content.

The Teacher: O Emphasizes to students

why they need to learn math content and skills.

O Uses instructional strategies that require students to apply and transfer mathematical knowledge to different content areas.

. . . and The Teacher: O Emphasizes

interdisciplinary connections to math.

. . . and Students: O Discuss ideas and

solutions to challenging mathematical problems.

O Are able to explain using

mathematical vocabulary their approach to solving a problem.


26



Element d: Teachers demonstrate knowledge of the content, central concepts, tools of inquiry, appropriate evidence-based instructional practices and specialized character of the disciplines being taught. There is inadequate evidence that the Teacher: O Breaks down concepts

and teaches each part using appropriate, effective strategies and/or tools.

O Uses appropriate instructional resources.

O Employs a variety of

instructional strategies to address student need.

The Teacher: Provides explanations of content that are: O Accurate. O Clear. O Concise. O Comprehensive.

O Uses instructional

materials that are accurate and appropriate for the lesson being taught.

O Maximizes learning opportunities.

. . . and The Teacher: O Designs lessons to

assure that student learning objectives are addressed.

O Engages students in a variety of explanations and multiple representations of concepts and ideas.

O Uses a variety of inquiry methods to explore new ideas and theories.

. . . and Students: O Identify central

concepts and discuss them in the context of the discipline being taught.

O Apply newly learned content to unique situations and different disciplines.

O Are able to discuss intellectually challenging ideas and content

Element e: Teachers develop lessons that reflect the interconnectedness of content areas/disciplines.

There is inadequate evidence that the Teacher: O Monitors learning

during instruction.

O Highlights key concepts and connects them to other powerful ideas.

O Implements

instruction that communicates a purpose for learning.

The Teacher: Establishes an environment and uses instructional strategies to assure that instruction: O Addresses the full

spectrum of learning needs, skill levels, and learning styles.

O Articulates content and interdisciplinary connections.

. . . and The Teacher: O Carefully and clearly

builds interdisciplinary connections for students.

O Provides instructional strategies that include literacy, numeracy, and language development across content areas.

. . . and Students:

O Accelerate and

advance their learning through connecting the current lesson with other lessons.



27



Element f: Teachers make instruction and content relevant to students and take actions to connect students’ background and contextual knowledge with new information being taught. There is inadequate evidence that the Teacher: O Motivates students to

make connections to their learning.

O Selects instructional materials and strategies with regard to relevance, central contexts, or foundational evidence base.

O Consistently and appropriately links content and prior knowledge.

The Teacher: Designs lessons and units and uses instructional strategies that: O Helps students connect to

their learning by linking curriculum with prior knowledge, experiences, and /or cultural contexts.

O Employs appropriate services, resources, and materials to facilitate student engagement.

O Is developmentally appropriate.

. . . and The Teacher: O Motivates students to

make connections to prior learning.

O Designs lessons and materials to assure that student learning objectives are addressed in ways that are meaningful for diverse learners.

. . . and

Students connect to their learning by:

O Asking questions and

solving problems that are meaningful to them.

O Placing class learning objectives in their own learning experiences.

O Associating new content with their backgrounds and contextual knowledge.



28

Quality Standard II: Teachers establish a safe, inclusive and respectful learning environment for a diverse population of students.


Element a: Teachers foster a predictable learning environment in the classroom in which each student has a positive, nurturing relationship with caring adults and peers. There is inadequate evidence that the Teacher creates a classroom environment in which: O Diversity is

acknowledged and used to further student learning.

O The importance of student and family background is considered in developing lesson plans.

O Students build positive relationships with each other.

The Teacher creates a classroom environment that: O Emphasizes mutual

respect for and understanding of all students.

O Encourages positive relationships between and among students.

O Is conducive for all students to learn.

. . and The Teacher: O Creates a classroom

environment which values diverse perspectives.

O Models empathy and respect for diversity.

O Sets common goals for all

students in order to build unity.

. . and Students: O Engage in respectful and open dialogue with each other and their Teacher.

Element b: Teachers demonstrate a commitment to and respect for diversity, while working toward common goals as a community and as a country. There is inadequate evidence that the Teacher creates a classroom environment in which: O Student diversity is

valued. O Student and family

background characteristics are considered in developing lessons.

The Teacher: O Uses instructional

approaches and materials that reflect students’ backgrounds.

O Acknowledges the value of each student’s contributions to the quality of lessons.

. . . and The Teacher establishes routine processes that result in: O A strong sense of

community among students.

O Effective interactions among students.

O Respect for individual differences.

O Positive social relationships.

. . . and Students: O Actively seek a variety of perspectives to complete group assignments.



29



Element c: Teachers engage students as individuals with unique interests and strengths. There is inadequate evidence that the Teacher: O Has high expectations for

all students.

O Uses data for instructional decision making.

O Holds students

accountable for their learning.

O Considers student

interests in planning lessons.

The Teacher: O Monitors students for

level of participation. O Encourages students to

share their interests. O Challenges students to

expand and enhance their learning.

O Acknowledges students

for their accomplishments.

. . . and The Teacher: O Asks appropriately

challenging questions of all students.

O Scaffolds questions. O Gives wait time

equitably. O Flexibly groups students. O Ensures that all students

participate with a high level of frequency.

. . . and Students: O Express their individual

interests within the context of the lesson while participating in the class.

O Encourage fellow students to participate and to challenge themselves.

O Select challenging

content and activities when given the choice in order to stretch their skills and abilities

Element d: Teachers adapt their teaching for the benefit of all students, including those with special needs, across a range of ability levels. There is inadequate evidence that the Teacher: O Adapts lesson plan to

address individual student needs.

Uses recommendations made by specialists and colleagues to understand student needs.

The Teacher: Designs instruction to

address specific learning needs of all students.

O Monitors the quality of

student participation and performance.

. . . and The Teacher: Solicits input from

colleagues and specialists to understand students’ learning needs.

O Uses multiple strategies to teach and assess students.

O Adapts instructional strategies to meet student needs.

O Challenges and supports all students to learn to their greatest ability.

. . . and Students:

O Articulate an awareness

of their learning needs and abilities.

O Advocate for themselves

in seeking teacher support.

O Recognize other students’

learning needs when working together.

O Support fellow classmates by adapting joint learning experiences to meet a variety of ability levels.



30



Element e: Teachers provide proactive, clear and constructive feedback to families about student progress and work collaboratively with the families and significant adults in the lives of their students. There is inadequate evidence that the Teacher: O Establishes a classroom

environment that is inviting to families and significant adults.

The Teacher: O Maintains appropriate and

respectful relationships with students, their families, and significant adults.

Uses a variety of methods to initiate communication with families and significant adults.

Is sensitive to the diverse

family structures.

. . . and The Teacher: Partners with families and

significant adults to help students meet education goals.

Coordinates information

from families and significant adults with colleagues who provide student services.

Seeks services and

resources to meet the diverse needs of students.

. . . and Families and Significant Adults:

Demonstrate a clear

understanding of their student’s progress and needs.

Partner with the Teacher and the school for the benefit of their students

Element f: Teachers create a learning environment characterized by acceptable student behavior, efficient use of time, and appropriate intervention strategies. There is inadequate evidence that the Teacher: O Has rules to guide

students to behave appropriately in the classroom.

O Holds students accountable for school and/or class rules.

O Provides structures or

transitions at the beginning of each class.

The Teacher: O Puts procedures in place

to avoid interruption to instructional time.

O Posts class rules where

they are readily available to all students.

. . . and The Teacher: O Makes maximum use of

instructional time.

O Holds students accountable for adherence to school and class rules.

O Maintains a safe and

orderly environment.

. . . and Students: O Self-monitor to stay on

task during class periods and avoid interruptions to their work.

O Accept responsibility for

their behavior and use of time.



31

Quality Standard III: Teachers plan and deliver effective instruction and create an environment that facilitates learning for their students.


Element a: Teachers demonstrate knowledge of current developmental science, the ways in which learning takes place, and the appropriate levels of intellectual, social, and emotional development of their students. There is inadequate evidence that the Teacher: O Understands how to

differentiate instruction.

O Modifies content to

assure that students are able to work at their ability levels.

O Understands the

interrelatedness of students’ intellectual, social, and emotional development.

The Teacher: O Provides instruction

that is developmentally appropriate for all students.

Studies emerging research to expand personal knowledge of how students learn.

. . . and The Teacher: O Adapts lessons to

address students’ strengths and weaknesses.

O Applies knowledge of current developmental science to address student needs.

Collaborates with

colleagues with experience in developmental science to improve the quality of lessons.

. . . and Students: O Participate in the

classroom in ways that make the best use of their time and effort.

O Offer suggestions to the Teacher regarding ways to adapt lessons to make them more engaging, challenging, and relevant.

Demonstrate in student

work an understanding of their learning styles and abilities

Element b: Teachers plan and consistently deliver instruction that draws on results of student assessments, is aligned to academic standards, and advances students’ level of content knowledge and skills. There is inadequate evidence that the Teacher: Uses assessment

feedback to guide adjustments to instruction.

O Has explicit student outcomes in mind for each lesson.

The Teacher: O Instructs and

assesses required skills.

O Advances students’ content knowledge and skills.

O Aligns instruction

with academic standards and student assessment results.

. . . and The Teacher: O Monitors instruction

against student performance and makes real-time adjustments.

O Encourages students to take academic risks.

O Makes sure students

meet learning objectives while increasing proficiency levels.

. . . and Students: O Confer with the Teacher

to achieve learning targets.

O Use assessment results to advance their learning.

Strive to: Close gaps between

their level of performance and their learning objectives.



32



Element c: Teachers demonstrate a rich knowledge of current research on effective instructional practices to meet the developmental and academic needs of their students. There is inadequate evidence that the Teacher: O Understands how to

match instructional practices to student academic needs.

O Incorporates evidence-based strategies into lessons.

O Adapts instructional practices to changing student needs.

The Teacher: O Makes lesson objectives

clear to the students.


O Provides instruction that

requires critical thinking, problem solving, and performance skills.

O Checks for student

understanding of content.

. . . and The Teacher: O Facilitates learning by

supporting students as they learn new material.


. . . and Students: O Recognize effective

instruction and support the teacher in providing it.

O Articulate the ways in

which they learn most effectively.

Element d: Teachers thoughtfully integrate and utilize appropriate available technology in their instruction to maximize student learning. There is inadequate evidence that the Teacher: O Uses available technology

to facilitate classroom instruction.

O Monitors the use of technology in the classroom.

The Teacher: O Employs strategies and

procedures to ensure that all students have equal and appropriate access to available technology.

. . . and The Teacher: Researches effectiveness

of instructional technology approaches and activities.

Uses available technology to: O Enhance student learning. O Develop students’

knowledge and skills. O Enhance creative and

innovative skills. O Provide engaging and

motivating learning experiences.

. . . and

Students: use available technology to: O Accelerate their

learning. O Apply team building

and networking skills. O Deepen critical

thinking skills. O Communicate

effectively.



33



Element e: Teachers establish and communicate high expectations for all students and plan instruction that helps students develop critical-thinking and problem solving skills. There is inadequate evidence that the Teacher: O Understands that students

need to employ critical thinking and problem-solving skills.

O Incorporates practical application of higher order thinking and/or problem-solving skills into lessons.

The Teacher: O Sets student expectations

at a level that challenges students.

O Incorporates higher order thinking, critical thinking and/or problem-solving skills into lessons.

. . . and

The Teacher: O Clearly communicates

high expectations for all students.

O Challenges all students to learn to their greatest ability.

O Systematically and explicitly teaches higher-order thinking and problem-solving skills.

O Allows time for responses and discussion.

. . . and Students:

O Apply higher-order

thinking and problem-solving skills to address challenging issues.

O Have a clear understanding of the Teacher’s expectations for their learning

O Monitor their progress toward achieving Teacher’s high expectations.

Element f: Teachers provide students with opportunities to work in teams and develop leadership qualities. There is inadequate evidence that the Teacher: O Groups students to

maximize learning.

O Includes all students in individual and group activities.

The Teacher plans lessons that: O Require students to work

individually and in groups.

O Provide opportunities for students to participate using various roles and modes of communication.

. . . and The Teacher: O Provide students with

opportunities to work in teams.

O Adjusts team composition based on lesson objectives and student needs.

O Varies group size,

composition, and tasks to create opportunities for students to interact and learn from each other.

. . . and Students:

O Utilize group processes

to build trust and promote effective interactions among team members.



34



Element g: Teachers communicate effectively, making learning objectives clear and providing appropriate models of language. There is inadequate evidence that the Teacher: O Communicates effectively

with students.

The Teacher: O Models effective

communication skills.


. . . and The Teacher: O Models and teaches



. . . and Students:

O Demonstrate effective

written and oral communication skills with fellow students and the Teacher.

O Model formal communication in academic settings.

Element h: Teachers use appropriate methods to assess what each student has learned, including formal and informal assessments, and use results to plan further instruction. There is inadequate evidence that the Teacher: Provides adequate

feedback to students, families, and significant adults.

O Involves students in monitoring their learning.

O Understands the expected outcomes of learning experiences in order to assess them appropriately.

The Teacher: Establishes consistent and

appropriate strategies for assigning grades.

Bases grades on multiple measures that provide a comprehensive and consistent picture of student skills and knowledge.

Includes goal setting and

documentation of student progress toward mastery of state content standards in assessment plans.

. . . and The Teacher: O Requires students to

complete assessment tasks similar to those on state (e.g., CSAP) and national (e.g., SAT, NAEP) assessments.

O Uses a variety of

assessment methods. O Provides frequent, timely,

specific and individualized feedback about the quality of student work.

O Teaches students to use feedback in their learning.

. . . and Students: O Apply Teacher

feedback to improve performance and accelerate their learning.

O Recognize the value of assessments to inform their learning.

O Articulate their

personal strengths and needs based on self-assessment.

O Regularly evaluate and monitor their progress.



35

Rubric for Evaluating Colorado’s Teachers Five Point

Effective Teachers in the state of Colorado have the knowledge, skills, and commitments needed to provide excellent and equitable learning opportunities and growth for all students. They strive to support growth and development, close achievement gaps and to prepare diverse student populations for postsecondary and workforce success (See Appendix A). Effective Teachers facilitate mastery of content and skill development, and employ and adjust evidence-based strategies and approaches for students who are not achieving mastery and students who need acceleration. They also develop in students the skills, interests and abilities necessary to be lifelong learners, as well as for democratic and civic participation. Effective Teachers communicate high expectations to students and their families and utilize diverse strategies to engage them in a mutually supportive teaching and learning environment. Because effective Teachers understand that the work of ensuring meaningful learning opportunities for all students cannot happen in isolation, they engage in collaboration, continuous reflection, on-going learning and leadership within the profession. The Teacher Quality Standards outline the knowledge and skills required of an effective Teacher and will be used to evaluate Teachers in the state of Colorado. All School Districts and BOCES shall base their evaluations of licensed classroom Teachers on the full set of Teacher Quality Standards and associated detailed Elements included below, or shall adopt their own locally developed standards that meet or exceed the Teacher Quality Standards and Elements.


Not Evident Partially Proficient Proficient

(Meets State Standard)

Accomplished Exemplary

Element a: Teachers provide instruction that is aligned with the Colorado Academic Standards; their District's organized plan of instruction; and the individual needs of their students.

There is inadequate evidence that the Teacher: O Plans

instruction on a daily basis.



The Teacher: Develops lesson plans based on: O Colorado

Academic Standards.

O District’s plan of instruction.

O Student needs.



instruction. O Colorado

Academic Standards.

Collaborates with


. . . and Students: Advance to the

next level within the curriculum or next higher course in sequence.

O Interact with the rigorous and challenging content in meaningful ways.

. . . and

Students: Discuss gaps in their learning with: O Teacher. Families and

significant adults.



36


Not Evident Partially Proficient Proficient (Meets State Standard) Accomplished Exemplary

Element c: Teachers demonstrate knowledge of mathematics and understand how to promote student development in numbers and operations, algebra, geometry and measurement, and data analysis and probability.

This section describes professional practices that should be demonstrated by ALL TEACHERS.

There is inadequate evidence that the Teacher: O Includes math topics in

discussions that do not have math as the primary focus.

O Promotes and encourages students to make explicit math connections across content.

The Teacher: O Emphasizes to

students why they need to learn math content and skills.

O Uses instructional strategies that require students to apply and transfer mathematical knowledge to different content areas.

. . . and The Teacher: O Emphasizes

interdisciplinary connections to math.

. . . and Students: O Share ideas and

solutions to challenging problems.

Strive to achieve

the high standards set for them.

. . . and Students: O Use the

language of math to talk about what they are doing.

O Interpret mathematical information in ways that make it relevant to their learning.






Element b: Teachers demonstrate knowledge of student literacy development in reading, writing, speaking and listening. This section describes professional practices that should be demonstrated by

ALL TEACHERS, regardless of grade level or subject taught. There is inadequate evidence that the Teacher: O Emphasizes literacy

connections while teaching content other than reading, English, and/or language arts.

O Has knowledge of how to integrate literacy across content areas.

The Teacher: Makes complex reading accessible to students by: O Making necessary

adjustments to content.

O Integrating literacy skills and knowledge into lessons.

O Demonstrates a deep understanding of literacy content and skills.

. . . and The Teacher: Provides literacy instruction that enhances: O Critical thinking

and reasoning. O Information

literacy. O Collaboration. O Self-direction. O Innovation.

O Focuses lessons

on the reading of complex materials.

. . . and Students: O Communicate

orally and in writing at levels that meet or exceed expectations for their age, grade, and ability level.

. . . and Students: Exceed expectations in: O Critical thinking. O Problem solving

skills. O Literacy skills.


37





Element d: Teachers demonstrate knowledge of the content, central concepts, tools of inquiry, appropriate evidence-based instructional practices and specialized character of the disciplines being taught. There is inadequate evidence that the Teacher: O Breaks down concepts

and teaches each part using appropriate, effective strategies and/or tools.

O Uses appropriate instructional resources.

O Employs a variety of

instructional strategies to address student need.

The Teacher: Provides explanations of content that are: O Accurate. O Clear. O Concise. O Comprehensive.

O Uses instructional

materials that are accurate and appropriate for the lesson being taught.

O Maximizes learning opportunities.

. . . and The Teacher: O Designs lessons

to assure that student learning objectives are addressed.

O Engages students in a variety of explanations and multiple representations of concepts and ideas.

O Uses a variety of inquiry methods to explore new ideas and theories.

. . . and Students: O Develop a variety of

explanations and multiple representations of concepts.

O Build on the skills and knowledge learned in the classroom to engage in more complex concepts, ideas, and theories.

Use a variety of inquiry tools and strategies to: O Learn content. O Understand central

concepts. O Answer complex

questions.

. . . and Students routinely: O Choose challenging

tasks and instructional materials.

O Apply newly learned content skills to unique situations and different disciplines.

O Initiate discussions of intellectually challenging ideas and content.

Element e: Teachers develop lessons that reflect the interconnectedness of content areas/disciplines.

There is inadequate evidence that the Teacher: O Monitors learning

during instruction.

O Highlights key concepts and connects them to other powerful ideas.

O Implements

instruction that communicates a purpose for learning.

The Teacher: Establishes an environment and uses instructional strategies to assure that instruction: O Addresses the full

spectrum of learning needs, skill levels, and learning styles.

O Articulates content and interdisciplinary connections.

. . . and The Teacher: O Carefully and

clearly builds interdisciplinary connections for students.

O Provides instructional strategies that include literacy, numeracy, and language development across content areas.

. . . and Students: O Reflect on their

learning.

O Help set their learning objectives.

O Make connections between prior learning and the current lesson.

. . . and Students: O Use current lesson to

accelerate their learning, and advance to the next performance level.



38



Element f: Teachers make instruction and content relevant to students and take actions to connect students’ background and contextual knowledge with new information being taught. There is inadequate evidence that the Teacher: O Motivates students to

make connections to their learning.

O Selects instructional materials and strategies with regard to relevance, central contexts, or foundational evidence base.

O Consistently and appropriately links content and prior knowledge.

The Teacher: Designs lessons and units and uses instructional strategies that: O Helps students

connect to their learning by linking curriculum with prior knowledge, experiences, and /or cultural contexts.

O Employs appropriate services, resources, and materials to facilitate student engagement.

O Is developmentally appropriate.

. . . and The Teacher: O Motivates students to

make connections to prior learning.

O Designs lessons and materials to assure that student learning objectives are addressed in ways that are meaningful for diverse learners.

. . . and

Students connect to their learning by: O Interacting with

materials that are relevant to them.

O Asking questions and solving problems that are meaningful to them.

O Making

connections to prior learning in order to facilitate understanding of current content.

. . . and Students: O Are actively

engaged in learning.

O Choose tasks that challenge and expand their skills and knowledge.

O Transfer

knowledge to other theories, ideas, and/or content.



39



Element a: Teachers foster a predictable learning environment in the classroom in which each student has a positive, nurturing relationship with caring adults and peers. There is inadequate evidence that the Teacher creates a classroom environment in which: O Diversity is

acknowledged and used to further student learning.

O The importance of student and family background is considered in developing lesson plans.

O Students build positive relationships with each other.

The Teacher creates a classroom environment that: O Emphasizes mutual

respect for and understanding of all students.

O Encourages positive relationships between and among students.

O Is conducive for all students to learn.

. . and The Teacher: O Creates a classroom

environment which values diverse perspectives.

O Models empathy and respect for diversity.

O Sets common goals

for all students in order to build unity.

. . and Students: O Demonstrate

respect for classmates and their Teacher.

. . and Students: O Engage in

respectful and open dialogue with each other and their Teacher.

Element b: Teachers demonstrate a commitment to and respect for diversity, while working toward common goals as a community and as a country. There is inadequate evidence that the Teacher creates a classroom environment in which: O Student diversity is

valued. O Student and family

background characteristics are considered in developing lessons.

The Teacher: O Uses instructional

approaches and materials that reflect students’ backgrounds.

O Acknowledges the value of each student’s contributions to the quality of lessons.

. . . and The Teacher establishes routine processes that result in: O A strong sense of

community among students.

O Effective interactions among students.

O Respect for individual differences.

O Positive social relationships.

. . . and Students: O Respect the

backgrounds of fellow students.

. . . and Students: O Actively seek a

variety of perspectives to complete group assignments.



40



Element c: Teachers engage students as individuals with unique interests and strengths. There is inadequate evidence that the Teacher: O Has high

expectations for all students.

O Uses data for instructional decision making.

O Holds students

accountable for their learning.

O Considers student

interests in planning lessons.

The Teacher: O Monitors students

for level of participation.

O Encourages students

to share their interests.

O Challenges students

to expand and enhance their learning.

O Acknowledges

students for their accomplishments.

. . . and The Teacher: O Asks appropriately

challenging questions of all students.

O Scaffolds questions.

O Gives wait time

equitably. O Flexibly groups

students. O Ensures that all

students participate with a high level of frequency.

. . . and Students: O Actively participate

in classroom activities.

O Seek opportunities to respond to difficult questions.

. . . and Students: O Select challenging

content and activities when given the choice in order to stretch their skills and abilities.

O Encourage fellow students to participate and challenge themselves.

O Participate in

collaborative learning and appropriate group processes.

Element d: Teachers adapt their teaching for the benefit of all students, including those with special needs, across a range of ability levels. There is inadequate evidence that the Teacher: O Adapts lesson

plan to address individual student needs.

Uses recommendations made by specialists and colleagues to understand student needs.

The Teacher: Designs instruction

to address specific learning needs of all students.

O Monitors the quality

of student participation and performance.

. . . and The Teacher: Solicits input from

colleagues and specialists to understand students’ learning needs.

O Uses multiple strategies to teach and assess students.

O Adapts instructional strategies to meet student needs.

O Challenges and supports all students to learn to their greatest ability.

. . . and Students:

O Articulate an

awareness of their learning needs.

O Advocate for

themselves. O Reflect about their

learning.

. . . and Students: O Seek ways to cope

with learning differences.

O Apply coping skills to classroom situations.

O Share coping strategies with fellow students.

O Support fellow

classmates by implementing peer supports.



41



Element e: Teachers provide proactive, clear and constructive feedback to families about student progress and work collaboratively with the families and significant adults in the lives of their students. There is inadequate evidence that the Teacher: O Establishes a

classroom environment that is inviting to families and significant adults.

The Teacher: O Maintains

appropriate and respectful relationships with students, their families, and significant adults.

Uses a variety of methods to initiate communication with families and significant adults.

Is sensitive to the

diverse family structures.

. . . and The Teacher: Partners with families

and significant adults to help students meet education goals.

Coordinates

information from families and significant adults with colleagues who provide student services.

Seeks services and

resources to meet the diverse needs of students.

. . . and Students: O Communicate

freely and openly with Teachers.

Families and Significant Adults: Initiate

communication with Teachers to discuss student needs.

Participate in a variety of school-based activities.

Willingly share

information that may impact student learning.

. . . and Families and Significant Adults: Seek the

Teacher’s assistance to find resources and services to support student needs.

Partner with the Teacher and the school for the benefit of their students.

Element f: Teachers create a learning environment characterized by acceptable student behavior, efficient use of time, and appropriate intervention strategies. There is inadequate evidence that the Teacher: O Has rules to guide

students to behave appropriately in the classroom.

O Holds students accountable for school and/or class rules.

O Provides structures or

transitions at the beginning of each class.

The Teacher: O Puts procedures in

place to avoid interruption to instructional time.

O Posts class rules

where they are readily available to all students.

. . . and The Teacher: O Makes maximum use

of instructional time.

O Holds students accountable for adherence to school and class rules.

O Maintains a safe and

orderly environment.

. . . and Students: O Stay on task

during class periods.

O Avoid interruptions to their work.

O Abide by school

and class rules.

. . . and Students: O Help other

students stay on task.

O Accept responsibility for their behavior and use of time.



42





Element a: Teachers demonstrate knowledge of current developmental science, the ways in which learning takes place, and the appropriate levels of intellectual, social, and emotional development of their students. There is inadequate evidence that the Teacher: O Understands how to

differentiate instruction.

O Modifies content to

assure that students are able to work at their ability levels.

O Understands the

interrelatedness of students’ intellectual, social, and emotional development.

The Teacher: O Provides

instruction that is developmentally appropriate for all students.

Studies emerging research to expand personal knowledge of how students learn.

. . . and The Teacher: O Adapts lessons to

address students’ strengths and weaknesses.

O Applies knowledge of current developmental science to address student needs.

Collaborates with

colleagues with experience in developmental science to improve the quality of lessons.

. . . and Students: O Articulate their

learning needs.

O Seek materials and resources appropriate for their learning styles.

. . . and Students: O Offer suggestions to

the Teacher regarding ways to adapt lessons to make them more engaging, challenging, and relevant.

Seek to understand: How they learn. O Where their time

and efforts are best used.

Element b: Teachers plan and consistently deliver instruction that draws on results of student assessments, is aligned to academic standards, and advances students’ level of content knowledge and skills. There is inadequate evidence that the Teacher: Uses assessment

feedback to guide adjustments to instruction.

O Has explicit student outcomes in mind for each lesson.

The Teacher: O Instructs and

assesses required skills.

O Advances students’ content knowledge and skills.

O Aligns

instruction with academic standards and student assessment results.

. . . and The Teacher: O Monitors

instruction against student performance and makes real-time adjustments.

O Encourages students to take academic risks.

O Makes sure

students meet learning objectives while increasing proficiency levels.

. . . and Students: O Monitor their

level of engagement.

O Confer with the Teacher to achieve learning targets.

. . . and Students: Strive to: O Address their

learning needs. Close gaps between

their level of performance and that of other students.

O Take academic risks.



43



Element c: Teachers demonstrate a rich knowledge of current research on effective instructional practices to meet the developmental and academic needs of their students. There is inadequate evidence that the Teacher: O Understands

how to match instructional practices to student academic needs.

O Incorporates evidence-based strategies into lessons.

O Adapts instructional practices to changing student needs.

The Teacher: O Makes lesson

objectives clear to the students.


O Provides instruction

that requires critical thinking, problem solving, and performance skills.

O Checks for student

understanding of content.

. . . and The Teacher: O Facilitates learning

by supporting students as they learn new material.


. . . and Students: O Articulate the

importance of the lesson objective.

O Connect lesson objective to prior knowledge in a significant and meaningful way.

O Describe their level of performance in relation to lesson objectives.

. . . and Students: O Apply skills and

knowledge learned in the classroom.

O Articulate the ways

in which they learn most effectively.

Element d: Teachers thoughtfully integrate and utilize appropriate available technology in their instruction to maximize student learning. There is inadequate evidence that the Teacher: O Uses available

technology to facilitate classroom instruction.

O Monitors the use of technology in the classroom.

The Teacher: O Employs strategies

and procedures to ensure that all students have equal and appropriate access to available technology.

. . . and The Teacher: Researches

effectiveness of instructional technology approaches and activities.

Uses available technology to: O Enhance student

learning. O Develop students’

knowledge and skills. O Enhance creative and

innovative skills. O Provide engaging and

motivating learning experiences.

. . . and

Students: O Engage in virtual

or face-to-face learning activities enhanced by appropriate use of available technology.

. . . and Students use available technology to: O Accelerate their

learning. O Apply team building

and networking skills.

O Deepen critical thinking skills.

O Communicate effectively.



44



Element e: Teachers establish and communicate high expectations for all students and plan instruction that helps students develop critical-thinking and problem solving skills. There is inadequate evidence that the Teacher: O Understands

that students need to employ critical thinking and problem-solving skills.

O Incorporates practical application of higher order thinking and/or problem-solving skills into lessons.

The Teacher: O Sets student

expectations at a level that challenges students.

O Incorporates higher order thinking, critical thinking and/or problem-solving skills into lessons.

. . . and

The Teacher: O Clearly

communicates high expectations for all students.

O Challenges all students to learn to their greatest ability.

O Systematically and explicitly teaches higher-order thinking and problem-solving skills.

O Allows time for responses and discussion.

. . . and Students: O Strive to achieve

expectations set by the Teacher.

O Apply higher-order thinking and problem-solving skills to address challenging issues.

. . . and Students: O Monitor their

progress toward achieving Teacher’s high expectations.

O Perform at levels exceeding expectations.

O Seek opportunities

to test their problem-solving and higher-order skills.

Element f: Teachers provide students with opportunities to work in teams and develop leadership qualities. There is inadequate evidence that the Teacher: O Groups students

to maximize learning.

O Includes all students in individual and group activities.

The Teacher plans lessons that: O Require students to

work individually and in groups.

O Provide opportunities for students to participate using various roles and modes of communication.

. . . and The Teacher: O Provide students

with opportunities to work in teams.

O Adjusts team composition based on lesson objectives and student needs.

O Varies group size,

composition, and tasks to create opportunities for students to interact and learn from each other.

. . . and Students: O Assume

leadership roles in their teams whenever possible.

O Accept and fulfill their assigned roles within the team.

. . . and

Students: O Utilize group

processes to build trust and promote effective interactions among team members.



45



Element g: Teachers communicate effectively, making learning objectives clear and providing appropriate models of language. There is inadequate evidence that the Teacher: O Communicates

effectively with students.

The Teacher: O Models effective

communication skills.


. . . and The Teacher: O Models and teaches



. . . and Students: O Apply effective

written and oral communication skills in their work.

O Demonstrate a respectful and sensitive approach toward fellow students and Teachers.

. . . and Students: O Participate in teams

in ways that build trust and ownership of ideas among team members.

O Model formal communications in academic settings.

Element h: Teachers use appropriate methods to assess what each student has learned, including formal and informal assessments, and use results to plan further instruction. There is inadequate evidence that the Teacher: Provides

adequate feedback to students, families, and significant adults.

O Involves students in monitoring their learning.

O Understands the expected outcomes of learning experiences in order to assess them appropriately.

The Teacher: Establishes

consistent and appropriate strategies for assigning grades.

Bases grades on multiple measures that provide a comprehensive and consistent picture of student skills and knowledge.

Includes goal

setting and documentation of student progress toward mastery of state content standards in assessment plans.

. . . and The Teacher: O Requires students to

complete assessment tasks similar to those on state (e.g., CSAP) and national (e.g., SAT, NAEP) assessments.

O Uses a variety of

assessment methods. O Provides frequent,

timely, specific and individualized feedback about the quality of student work.

O Teaches students to use feedback in their learning.

. . . and Students: O Self-assess on a

variety of skills and concepts.

O Articulate their personal strengths and needs based on self-assessment.

O Effectively use formal and informal feedback to monitor their learning.

. . . and Students assume ownership for : O Evaluating and

monitoring their progress.

O Setting learning goals.

O Compiling portfolios of their work.

O Applying Teacher feedback to improve performance and accelerate their learning.



46

Appendix B: Teacher Quality Standards I, II, and III and Elements Quality Standard I: Teachers demonstrate mastery of and pedagogical expertise in the content they teach. The elementary Teacher is an expert in literacy and mathematics and is knowledgeable in all other content that he or she teaches (e.g., science, social studies, arts, physical education, or world languages). The secondary Teacher has knowledge of literacy and mathematics and is an expert in his or her content endorsement area(s). Element a: Teachers provide instruction that is aligned with the Colorado Academic Standards; their District’s organized plan of instruction; and the individual needs of their students. Element b: Teachers demonstrate knowledge of student literacy development in reading, writing, speaking and listening. Element c: Teachers demonstrate knowledge of mathematics and understand how to promote student development in numbers and operations, algebra, geometry and measurement, and data analysis and probability. Element d: Teachers demonstrate knowledge of the content, central concepts, tools of inquiry, appropriate evidence-based instructional practices and specialized character of the disciplines being taught. Element e: Teachers develop lessons that reflect the interconnectedness of content areas/disciplines. Element f: Teachers make instruction and content relevant to students and take actions to connect students’ background and contextual knowledge with new information being taught.

Quality Standard II: Teachers establish a safe, inclusive and respectful learning environment for a diverse population of students. Element a: Teachers foster a predictable learning environment in the classroom in which each student has a positive, nurturing relationship with caring adults and peers. Element b: Teachers demonstrate a commitment to and respect for diversity, while working toward common goals as a community and as a country. Element c: Teachers engage students as individuals with unique interests and strengths. Element d: Teachers adapt their teaching for the benefit of all students, including those with special needs across a range of ability levels. Element e: Teachers provide proactive, clear and constructive feedback to families about student progress and work collaboratively with the families and significant adults in the lives of their students. Element f: Teachers create a learning environment characterized by acceptable student behavior, efficient use of time, and appropriate intervention strategies.

Quality Standard III: Teachers plan and deliver effective instruction and create an environment that facilitates learning for their students. Element a: Teachers demonstrate knowledge of current developmental science, the ways in which learning takes place, and the appropriate levels of intellectual, social, and emotional development of their students. Element b: Teachers plan and consistently deliver instruction that draws on results of student assessments, is aligned to academic standards, and advances students’ level of content knowledge and skills. Element c: Teachers demonstrate a rich knowledge of current research on effective instructional practices to meet the developmental and academic needs of their students. Element d: Teachers thoughtfully integrate and utilize appropriate available technology in their instruction to maximize student learning. Element e: Teachers establish and communicate high expectations for all students and plan instruction that helps students develop critical-thinking and problem solving skills. Element f: Teachers provide students with opportunities to work in teams and develop leadership qualities. Element g: Teachers communicate effectively, making learning objectives clear and providing appropriate models of language. Element h: Teachers use appropriate methods to assess what each student has learned, including formal and informal assessments, and use results to plan further instruction.


47

Appendix C: Participant Surveys

Evaluator Post Feedback Session Survey [ONLINE]

CDE Rubric Comparison Study

Please take 15 minutes to reflect on your conduct of the teacher feedback sessions using the four point and five point versions of the CDE teacher evaluation rubric. How long did your teacher feedback sessions last, on average? ____hrs ____mins For how many feedback sessions were the school evaluators present?______ sessions Please rate the extent to which you agree with the following statements for each rubric. Five Point Four Point SA A D SD SA A D SD The rubric allowed me to provide clear feedback to teachers on professional practices.

The rubric was easy to use in providing teacher feedback.

The scale of the rubric allowed me to rate professional practices accurately.

The scale of the rubric allowed me to provide actionable feedback for growth.

The scale of the rubric limited my ability to rate a continuum of professional practices.


Which rubric allowed you to provide clearer and more actionable feedback to teachers for elements that were Not Evident 4 point 5 point Neither Both Partially Proficient or Proficient 4 point 5 point Neither Both Exemplary 4 point 5 point Neither Both What do you see as the main strengths of using a 4 point scale for the rubric? What do you see as the limitations of using a 4 point scale for the rubric? What do you see as the main strengths of using a 5 point scale for the rubric? What do you see as the limitations of using a 5 point scale for the rubric? What else would you like to share with us about your experiences using the 4 point and 5 point rubrics to provide actionable feedback to teachers?


48

Evaluator Post Observation Feedback Survey [ONLINE] CDE Rubric Comparison Study

Please take 10 minutes to reflect on your conduct of teacher observations using the four point and five point versions of the CDE teacher evaluation rubric. Observation Round 1 (October) Observation Round 2 (November)

1. Was the information you collected through the pre-observation protocol useful? Always Usually Seldom Comments:

2. Did you collect artifacts from teachers or school administrators? Always Usually Seldom

a. What types of artifacts did you collect?

3. Were there non-observable professional practices for which you had insufficient evidence to rate? Always Usually Seldom

a. Would having the evidence influence your ratings? Yes No

b. Was there a difference in this between the 5 pt and 4 pt rubrics?

Yes No

Four Point Rubric Five Point Rubric a. Were you comfortable using the

rubric? Yes No Yes No

b. Were you adequately prepared to apply the rubric? Yes No Yes No

c. Did any teachers receive an element rating of “Not Evident?” Yes No Yes No

d. Did any teachers receive an element rating of “Exemplary?” Yes No Yes No

4. Which rubric did you prefer? 4 point 5 point Neither a. Why did you prefer this rubric?

What else would you like to share with us about your experiences using the 4 point and 5 point rubrics?


49

Teacher Survey [ONLINE] CDE Rubric Comparison Study

Thank you for volunteering to participate in the Colorado Department of Education Rubric Comparison Study. This survey asks you about your experiences with and perceptions of the 4 point and 5 point rubrics of teacher professional practices. Please take a few minutes to reflect on the feedback session with your evaluator. Your responses are anonymous and will be reported in aggregated form across all teachers in the study. Thank you so much for your time and participation!

Which category best describes your position?

___ Regular classroom teacher _____Instructional specialist ____ Other (please describe)__________ What level do you teach? _____ Elementary ______Middle _______High School How many years teaching experience do you have? ______ years

Did you complete a self-assessment using the 5 point rubric? ___ Yes ___ No

If “yes,” Did you select aspects of your teaching for the focus of your evaluator? __ Yes __ No

If “yes,” Did your evaluator address these aspects during your feedback session? __ Yes __ No

Did your evaluator observe your teaching twice? ___ Yes ___ No

Did you have sufficient time for the feedback session ___Yes ____ No

For each rubric please rate the extent to which you agree with the following statements: Five Point Four Point SA A D SD SA A D SD I received clear feedback on professional practices based on the rubric.

I received actionable feedback for growth based on the rubric.

My ratings reflected my professional practices accurately.


Which rubric provided you with clearer and more actionable feedback across elements? 4 point 5 point Neither

Which rubric would you prefer to be used as part of your teacher performance evaluation? 4 point 5 point Neither Either

What else would you like to share with us about your experiences receiving feedback based on the 4 point and 5 point rubrics?


50

School Evaluator Survey [ONLINE] CDE Rubric Comparison Study

Thank you for supporting the Colorado Department of Education Rubric Comparison Study. You are receiving this survey because you’ve been identified as someone who conducts teacher evaluations and you are familiar with the rubrics used in this study. This survey asks you about your perceptions of the 4 point and 5 point rubrics of teacher professional practices. Please take a few minutes to reflect on the observation rubrics that evaluators used to observe teachers. Your responses are anonymous. Thank you so much for your time and participation! Which category best describes your position? ____ Principal ____ Assistant Principal ____Teacher

_____ Other (please describe)__________ How many years have you conducted teacher evaluations? _____years

Did your involvement in the study include sitting in on observations for your own school or a neighboring school? ____ My own school ____ Neighboring school

Please select the option that best describes your involvement in the observations in that school during the study. I watched the evaluator observe all teachers in the study twice. I watched the evaluator observe all teachers in the study at least once. I watched the evaluator observe some teachers in the study at least once. I did not watch the evaluator observe teachers in the study.

Please select the option that best describes your involvement during the final teacher feedback sessions. I observed all teacher feedback sessions with the evaluator. I observed some teacher feedback sessions with the evaluator. I did not observe any teacher feedback sessions with the evaluator.

For each rubric please rate the extent to which you agree with the following statements: Five Point Four Point SA A D SD Not enough

information to rate

SA A D SD Not enough information to rate

Overall, teachers received clear feedback on professional practices based on the rubric.

Overall, teachers received actionable feedback for growth based on the rubric.


Overall, which rubric provided clearer and more actionable feedback to teachers? 4 point 5 point Neither I don’t have enough information to choose

Overall, which rubric would you prefer to use in teacher performance evaluations? 4 point 5 point Neither Either I don’t have enough information to choose [if 4 point or 5 point:] Why would you prefer this rubric? What else would you like to share with us about your perceptions of the 4 point and 5 point rubrics?


51

Appendix D: Evaluator Focus Group Protocol

FOCUS GROUP PROTOCOL – EVALUATORS (Abbreviated)

Rubric Comparison Study Welcome Recording: We’re recording the session both via the phone system and separately – no names will be

included in the report. Introductions: Background What is a focus group, the research questions, ground rules – usual and special with

recording, state name when speak, speak up, be careful to not talk over others.

AGENDA:

CONTEXT/PROCESS Hear from each evaluator – Thinking about the observation process - What was for you a plus? A minus? Observations:

The pre-or post- observation protocol The rubrics The non-observables - Obtaining the artifacts you needed? Did your first observation process differ from your second? Anyone? How?

On the Feedback Sessions: What further reflections have you had on the feedback sessions? Any surprises?

o In the discussion with the teacher using the 5 or 4 point rubrics? (In element selection? In the actionable steps?)

COMPARISONS - Now to the 4 pt 5 pt rubric comparisons:

• Is there a difference between the 2 rubrics in making the observations? (Yes/No All respond) Discuss: To what do you attribute the difference? Or lack of difference?

• In rating an element? (Yes/No All respond) Discuss: • In the overall rating of a teacher? (Yes/No All respond) Discuss: • Did you have adequate evidence to make the ratings? (Yes/No All respond) Discuss. • Was there a difference in the feedback between the 4 and 5 pt? If so, for how many of your

teachers? What was the source of the difference? Discuss. Anything we haven’t said: Were there limitations in assessing professional practices using one rubric compared to the other? Were there strengths in assessing professional practices using one rubric compared to the other?

OVERALL What have we learned?

• Which rubric would you (we) recommend to CDE? (All respond) o And if one or the other, why that one?

• What message would you send with your recommendation?

colorado department of education teacher rubric … · colorado department of education teacher...

Documents