ase june statewide conference june 10, 2013 ron noble educator evaluation project co-lead

The New MA Educator Evaluation Framework: District-Determined Measures andStudent and Staff Feedback

ASE June Statewide ConferenceJune 10, 2013

Ron NobleEducator Evaluation Project Co-Lead

Massachusetts Department of Elementary & Secondary Education

22


Agenda Setting the Stage 2012-2013 Implementation

Lessons Learned On the Horizon

District-Determined Measures Student and Staff Feedback

Q&A


3

Setting the StageWhen policy and practice must move faster than

research and development, where do you begin?

ESE Philosophy: Don’t let perfection be the enemy of good: the

work is too important to delay. Understand this is just the beginning: we will be

able to do this work with increasing sophistication each year

Phased-in implementation: take advantage of emerging research, resources, and feedback from the field.


Questions for Policy Makers:

• Attribution: “When crediting teachers for student learning, how should the individual contributions of teachers acting in a coteaching or consultant role be determined?

• Assessments: “How can the contributions to student achievement be accurately measured for teachers instructing special populations for which alternative standards and/or assessments are used?”

• Educator differentiation: “Are the key features of teacher effectiveness for specialized personnel, such as special education teachers different… and should those unique features lead to additional or different content on observation protocols, student growth assessments, or alternative instruments?”

• Evaluator training: “When rating special education teachers…using an observation protocol or alternative instrument, what special training, if any, do evaluators need?”

Holdheide, L.R., Goe, L., & Reschly, D.J.. (2010) Challenges in Evaluating Special Education Teachers and English Language Learner Specialists. National Comprehensive Center for Teacher Quality.

4

Who IS the evaluator?

Are variations in contributions measurable? How should we

use the MCAS Alternate

Assessment?How do we differentiate

without creating “two systems”?


5

Implementation TimelineJune 2011 MA Board of Education passed new educator evaluation regulations

September 2011 Implementation began in 34 “Level 4” schools, 11 “Early Adopter” districts and 4 Special Education Collaboratives

January 2012 MA Department of Elementary and Secondary Education (ESE) published the MA Model System for Educator Evaluation

September 2012 RTTT districts began implementation with at least 50% of educators.

September 2013 •RTTT districts begin implementation with remaining educators.•Non-RTTT districts begin implementation with at least 50% of educators.

2013-2014 school year

•All districts pilot District-Determined Measures.•Selected districts pilot student and staff surveys

2014-2015 school year

•All districts implement District-Determined Measures.•All districts implement student and staff surveys

June 2016 •District determine Student Impact Ratings for all educators.


6

2012-2013 Implementation 234 Race to the Top Districts

At least 50% of educators Summative Performance Rating only

5-Step Evaluation Cycle June data reporting (EPIMS)

6 data elements:1. Rating on Standard I2. Rating on Standard II3. Rating on Standard III4. Rating on Standard IV5. Overall Summative Performance Rating6. Professional Teacher Status (Y/N)


7

5 Step Evaluation Cycle Every educator is

an active participant in an evaluation

Process promotes collaboration and continuous learning

Process applies to all educators


8

Summative Performance Rating

Summativ

e Ratin

g Unsatisfactory Improvement Plan

Needs Improvement Directed Growth Plan

Exemplary

Self-Directed Growth Plan

Proficient


9

Educator Evaluation Spring Convening: Connecting Policy, Practice, and Practitioners

May 29, 2013 Over 700 participants from district teams

(RTTT and non-RTTT) and educator preparation programs

Key messages: Integrate with other key district initiatives Opportunity to strengthen labor-

management relations Albeit difficult, it’s the right work


10

On the Horizon District-Determined Measures

Student and Staff Feedback


11

District-Determined Measures: Key Terms Student Impact Rating – a rating of high,

moderate, or low for an educator’s impact on student learning

District-Determined Measures – measures of student learning, growth, and achievement that will inform an educator’s Student Impact Rating


12

Student Impact Rating Regulations Evaluators must assign a rating based on trends (at

least 2 years) and patterns (at least 2 measures)

Options – 603 CMR 35.07(1)(a)(3-5) Statewide growth measure(s)* District-determined Measure(s) of student

learning comparable across grade or subject district-wide.

For educators whose primary role is not as a classroom teacher, the appropriate measures of the educator's contribution to student learning, growth, and achievement set by the district.

http://www.doe.mass.edu/lawsregs/603cmr35.html?section=07


13

Two Ratings

Summativ

e Ratin

g Unsatisfatory Improvement Plan

Needs Improvement Directed Growth Plan

Exemplary

Self-Directed Growth Plan

Proficient

1-yr Self-Directed Growth Plan 2-yr Self-Directed Growth Plan

Low Moderate High

Rating of Impact on Student Learning


14

Student Impact Rating Regulations Why focus on growth?

Level playing field Fairness

Achievement measures may be acceptable when the district judges them to be the most appropriate/feasible measure for certain educators


15

Revised Implementation Timeline Commissioner’s Memo - 4/12/13

2013-2014 – districts pilot and identify DDMs 2014-2015 – districts implement DDMs and collect the

first year of trend data 2015-2016 – districts collect the second year of trend

data and issue Student Impact Ratings for all educators Districts positioned to accelerate the timeline should

proceed as planned. Guidance and resources to support districts with the

identification of DDMs are available here: http://www.doe.mass.edu/edeval/ddm/

http://www.doe.mass.edu/edeval/ddm/


16

Revised Implementation Timeline Minimum Piloting Requirements

Early grade (K-3) literacy Early (K-3) grade math Middle grade (5-8) math High school writing to text Traditionally non-tested grades and subjects (e.g., fine arts,

music, physical education)

If a district is unable to identify a DDM in the grades and subjects listed above, the district must pilot one of ESE’s exemplar DDMs to be released in summer 2013.


17

Recommended Steps for Districts Identify a team of administrators, teachers and

specialists to focus and plan the district’s work on District-Determined Measures.

Complete an inventory of existing assessments used in the district’s schools.

Identify and coordinate with partners that have capacity to assist in the work of identifying and evaluating assessments that may serve as District-Determined Measures. 17

Quick Reference Guide: District-Determined Measures

http://www.doe.mass.edu/edeval/resources/QRG-Measures.pdf

http://www.doe.mass.edu/edeval/resources/QRG-Measures.pdf


18

WestEd is supporting ESE with next steps in implementing the Commonwealth’s Model System for Educator Evaluation

Two broad categories of work Support development of anchor standards in

almost 100 separate grades/subjects or courses Identification and evaluation of promising

measures, tools, tests, rubrics Work to be completed by mid-August

ESE Supports


19

ESE Supports Supplemental guidance on the selection of

DDMs and the process of determining an Impact Rating DDM and Assessment Literacy Webinar Series

(March – December) Technical Guide A (released in May 2013) focuses

on selecting high quality assessments Includes Assessment Quality Checklist and Tracking Tool

Technical Guide B (expected in August 2013) will focus on measuring growth.


20

ESE SupportsAssessment Quality Checklist Tool

General Information

Grade and Subject or Course Potential DDM Name Potential DDM Source

Type of Assessment Item Types

Step #1: Evaluate Content Alignment Describe the Process Used to Determine RatingsAlignment Alignment to Curriculum 0

Rigor Alignment to Intended Rigor 0 Total Score 0

% of Possible Score 0%

Step #2: Evaluate Remaining Evidence of Assessment Quality Describe the Process Used to Determine Ratings

Utility & FeasibilityUtility 0

Feasibility 0

Assessment Components

Table of Test Specifications 0 Administration Protocol 0

Instrument 0 Scoring Method 0

Technical Documentation 0

ReliabilityReliability Evidence Collection

Approach 0

Reliability Evidence Quality 0

ValidityValidity Evidence Collection

Approach 0

Validity Evidence Quality 0

Non-BiasGathered Evidence of Non-Bias

0

Item QualityRange of Item Difficulties 0

Positively Discriminating Items 0 No Floor/Ceiling Effects 0

Total Score 0

% of Possible Score 0%


21

DDMs: Request for Feedback Attribution: How can ESE best support districts in developing

attribution policies related to the determination of Student Impact Ratings, particularly for coteachers, consulting teachers, and other scenarios where more than one teacher contributes to student learning, growth, and achievement?

Movement of Students: Due to highly specialized and often changing needs, the population of children identified as needing special education services fluctuates annually, sometimes in significant amounts, and mostly in the elementary grades. This fluctuation means students move in and out of special education classes and may not receive special education instruction for an entire year. How should ESE recommend districts take student movement into account when determining special educators’ Student Impact Ratings?

Selecting Assessments: What are some considerations ESE should be

aware of when providing guidance on the selection of measures of student growth to be used in the determination of special educators’ Student Impact Ratings? Please include specific examples of measures that would or would not be appropriate and why.

21


22

Student and Staff Feedback Revised Implementation Timeline:

Beginning in the 2014-2015 school year, districts will include student feedback in the evaluation of all educators and staff feedback in the evaluation of all administrators.

During the 2013-2014 school year, ESE will work with districts to pilot/field test model survey instruments.


23

Multiple sources of evidence inform the summative rating


Idaho Kentucky Maine Massachuset

ts Michigan Missouri

24

Alaska Arizona Colorado Delaware Georgia Hawaii

National Overview A growing number of states are currently using or preparing to use student surveys in educator evaluations

Mississippi New Jersey New York North

Carolina Rhode Island Washington


Perception surveys round out a multiple measure evaluation system

Research also finds student surveys are correlated with student achievement The Measures of Effective Teaching Project found students’

perceptions are reliable, stable, valid, and predictive Surveys may be the best gauge of student engagement

When asked which measures are good or excellent at assessing teacher effectiveness, teachers reported

District standardized tests (56 percent) Principal feedback (71 percent) Students’ level of engagement (92 percent)

25

Why Use Student Surveys in Educator Evaluations?


26

MA’s State Student Advisory Council and six regional student advisory councils provide a unique feedback loop for students

MA Student Advisory Council focus groups were overwhelmingly positive toward soliciting their input through student surveys MA students want to help teachers improve MA students are excited about the prospect of being surveyed for

this purpose MA students offered thoughtful precautions about survey use:

Use surveys for teacher goal-setting Consider making survey feedback visible only to teachers Provide 3rd party screeners of any open-ended questions

What Students Say…


27

Benefits of Surveys of Classroom/School Experiences Offers valuable insight

from those with first-hand experience

Empowers and engages survey recipients, sending a signal that their input is valued

Comparatively inexpensive

Surveys as a Form of Feedback

Considerations When Using Surveys of Classroom/School Experiences Students may lack

cognitive ability or maturity

Could become a popularity contest or “rate-your-teacher.com”

Survey results could be misused by evaluators


28

The more immediate the feedback the better The more flexibility for teachers to administer surveys

when they wish the better Surveys for early grades and special populations require

special attention To the extent that surveys are used for high stakes

decisions at all, this should not happen until after they have been used effectively and reliably, and educators have grown comfortable with them, in a low stakes setting

When used for formative purposes, surveys are generally seen as a good thing

National Perspective – Lessons Learned


29

Key areas for state or district consideration: 1. Determining survey samples 2. Timing of survey administration 3. Reporting of survey results 4. Using survey results in evaluations 5. Considerations for pre-readers, special education,

and English Learners

Perspectives & Considerations


30

Student and Staff Feedback: Request for Feedback Source of Evidence: In what way or ways should ESE recommend

student and staff feedback be used as a source of additional evidence relevant to one or more Performance Standards?

Accommodations: What types of arrangements are most appropriate for the special populations, i.e., pre-readers, students with limited English proficiency, and students with disabilities, so that their feedback can be taken into account as well?

Data Collection Tools: In addition to perception surveys, what other

types of data collection tools for capturing student feedback should ESE recommend and for what populations would these tools be most useful?


31

Additional Questions? Ron Noble –

[email protected] or 781.338.3243

ase june statewide conference june 10, 2013 ron noble educator evaluation project co-lead

Documents