concurrent session ii: combining evaluation measures evaluation... · metric! indiv. score! weight!...
TRANSCRIPT
Concurrent Session II: Combining Evaluation Measures
TQ Center Annual Conference Systems that Last: Great Teachers and Leaders for
America’s Schools Day 2 – September 6, 2012
2
Welcome and Introductions
Moderator: Lisa Lachlan-Haché Senior Research and Policy Associate, American Institutes for Research Panelists: Mary Ann Snider Chief of Educator Excellence and Instructional Effectiveness, Rhode Island Department of Education Michaela Miller Teacher-Principal Evaluation Project Manager, Office of the Superintendent of Public Instruction (WA) Dale Chu Assistant Superintendent for Innovation and Improvement, Indiana Department of Education
3
Session Overview
Objectives: 1. Describe three models for combining multiple measures 2. Highlight examples from the field and key decisions 3. Provide structured time for participants to discuss and
apply ideas
Dale Chu (IN) QUESTION HERE . . .
Questions? Record them on the index cards provided. Include the intended respondent(s) at the top of the card.
4
Common Approaches to Combining Evaluation Measures for Rating Purposes Lisa Lachlan-Haché Senior Research and Policy Associate American Institutes for Research Resource: Creating Summative Educator Effectiveness Scores: Approaches to Combining Measures, by Sheri Leo and Lisa Lachlan-Haché
5
Numerical Approach
• Identify weight associated with each measure.
• Assign points to each measure and add or average together.
• Create and apply score ranges for each summative rating.
Metric Indiv. Score
Weight Final Ra6ng
Classroom Observa-ons 88% 25% 0.22 Professional Goal Se:ng 90% 10% 0.09 Professionalism 76% 15% 0.11 Student Growth 84% 50% 0.42 Summa6ve Teacher Effec6veness Score 0.84
Classroom Observa-ons
Professional Goal Se:ng
Professionalism
Student Growth
Does Not Meet Standards
Par6ally Meets Standards
Meets Standards
Exceeds Standards
0.0 – 0.19 0.20 – 0.54 0.55 – 0.89 0.90 – 1.0
6
Profile Approach
• Gather and maintain evidence for multiple measures and rate educators separately on each measure.
• Combine results from disparate measures using a matrix, lookup table, or series of decision rules.
Summa6ve Professional Prac6ce and Responsibility Ra6ng Dis6nguished Accomplished Proficient Emerging Unsa6sfactory
Summa6ve Student Growth Ra6ng
4 Highly effec-ve
Highly effec-ve
Effec-ve Effec-ve Minimally effec-ve
3 Highly effec-ve
Effec-ve Effec-ve Minimally effec-ve
Ineffec-ve
2 Effec-ve Effec-ve Minimally effec-ve
Minimally effec-ve
Ineffec-ve
1 Minimally effec-ve
Minimally effec-ve
Minimally effec-ve
Ineffec-ve Ineffec-ve
7
Holistic Rating Approach
• Review the body of collected evidence and interpret it using the performance rubric to issue a single holistic rating for the educator.
Evidence and other factors • Teacher’s background and
experience • Evaluation evidence • Local context, district priorities
Evaluator judgment Teacher effectiveness score or rating
Discussion with teacher
8
Most Systems Use a Hybrid Approach
• Balances strengths and weaknesses of each “pure” approach
• Incorporates stakeholder input and local context
• Acknowledges the multiple levels of decision-making in rating performance
• Breaks down the system into more easily communicated components
9
Optional Implementation “Rules”
Minimum Competence Thresholds • Create decision rules around minimum standards for some or all
performance criteria that supersede other rules. • Apply these rules to all or some educators (e.g. veteran, those
nearing tenure).
Proficiency Progression • Choose the performance criteria that are most critical for proficiency
in the first year/phase. • Increase minimum requirements year by year until desired
proficiency standards are met.
10
Mary Ann Snider Chief of Educator Excellence and Instructional Effectiveness Rhode Island Department of Education
11
Components of a Final Effectiveness Rating
Professional Prac6ce Ra6ng
Professional Founda6ons
Ra6ng
PP and PF
Score
Student Learning Objec6ve Ra6ng
RI Growth Model Ra6ng
(When Available)
Student Learning Score
FINAL RATING
12
Evidence Informing Summative Ratings: Finding the “Best Fit”
• Observations of Professional Responsibilities
• Observations of Classroom Practice
• Artifact Review
• Data from Student Learning Objectives
• Student Growth Scores
• Matrices
13
Professional Practice Rubric
Component Observa6on 1 Observa6on 2 Observa6on 3 Average
There are 8 components
1-‐4 1-‐4 1-‐4 Each component averaged to the nearest tenth.
TOTAL Max. of 32
Exemplary = 29–32 Proficient = 22–28 Emerging = 15–21 Unsa6sfactory = 8-‐14
14
Professional Foundations Rubric
There are 8 components (e.g. understands and participates in district or school-based activities and initiatives) Each component is rated as a 3, 2, or 1 The ratings across all eight components are summed
Exceeds Expecta6ons= 21-‐24 Meets Expecta6ons= 16-‐20 Does Not Meet Expecta6ons= 8-‐15
15
Matrix Used for All Educators Professional Prac-ce
Exemplary 29-‐32
Proficient 22-‐28
Emerging 15-‐21
Unsa-sfactory 8-‐14
Professio
nal Fou
nda-
ons Exceeds
Expecta-ons 21-‐24
4 4 2 2
Meets Expecta-ons
16-‐20 4 3 2 1
Does Not Meet Expecta-ons
8-‐15 2 2 1 1
Matrix for: Professional Practice and Professional Foundations
16
Student Learning Objectives
Each SLO is scored as Exceeded, Met, Nearly Met, or Did Not Meet Scores are entered into the Educator Performance and Support
System. Combinations of scores results in a final rating.
Excep6onal A[ainment Full A[ainment Par6al A[ainment Minimal A[ainment
17
Matrix Used for All Educators Student Learning Objec-ves
Excep-onal A[ainment
Full A[ainment
Par-al A[ainment
Minimal A[ainment
Grow
th Score
High Growth 4 4 3 2
Typical Growth 4 3 2 1
Low Growth 2 2 1 1
Matrix for: Student Growth and SLO Scores
18
Student Learning
4 3 2 1
Professio
nal Fou
nda-
ons a
nd
Professio
nal Prac-ce 4 HE E D D*
3 HE E D D
2 E E D I
1 D* D I I
Final Effectiveness Matrix
19
Michaela Miller Teacher-Principal Evaluation Project Manager, Office of the Superintendent of Public Instruction (WA)
20
Influences on TPEP Development
2012 2012-‐
ESSB 5895
ESEA Flexibility Waiver
TPEP Pilot Sites & Steering Cmte
Instruc-onal and
Leadership Framework Authors
Research and Best Prac-ce
E2SSB 6696 & Race to the Top
Washington State
Evalua-on and Professional
Growth System
2010-‐12
21
Summative Rating Process Overview
• ESSB 5895 requires OSPI to determine a summative scoring methodology by Dec. 1, 2012
• Summative Rating is determined through a “Raw Score” Model
• Generated from the TPEP Pilot Sites and approved by the TPEP Steering Committee
• Used for both the teacher and principal evaluation systems.
• Determination of overall criterion score based on both: § Instructional framework rubrics § Student growth rubrics
22
The RAW Score Model
Teaching Criteria * Indicate Criterion embedded with student growth rubrics
Overall Criterion Scores
Criterion 1: Centering instruc6on on high expecta6ons for student achievement 3 Criterion 2: Demonstra6ng effec6ve teaching prac6ces 4 *Criterion 3: Recognizing individual student learning needs and developing strategies to address those needs 3 Criterion 4: Providing clear and inten6onal focus on subject ma[er content and curriculum 2 Criterion 5: Fostering and managing a safe, posi6ve learning environment 3 *Criterion 6: Using mul6ple student data elements to modify instruc6on and improve student learning 2 Criterion 7: Communica6ng and collabora6ng with parents and school community 3 *Criterion 8: Exhibi6ng collabora6ve and collegial prac6ces focused on improving instruc6onal prac6ce and student learning 2
Total Summa6ve Score 22
Evaluators place teachers into preliminary summative rating categories based on score bands. As illustrated above, this teacher would receive a preliminary overall summative rating of Proficient.
8-‐14 15-‐21 22-‐28 29-‐32 1
Unsa6sfactory 2
Basic 3
Proficient 4
Dis6nguished
23
Evaluation Summative Scoring Process
Criteria 2
Criteria 1
Criteria 3
Criteria 4
Criteria 5
Criteria 6
Criteria 7
Criteria 8
Frameworks +
Student Growth Rubrics
Observa-on Ar-facts Other evidence relevant to the frameworks
Student Growth Measures
(From 3 specific criteria)
State determined process Dis-nguished Proficient Basic Unsa-sfactory
Student Growth Impact Ra-ngs: Low, Average, High
District determined process Dis-nguished Proficient Basic Unsa-sfactory
24
Student Growth Rubric and Rating (Teachers Only)
Student Growth Goal-‐Sebng Score Based on Rubric
Student Growth* Score Based on Rubric
Overall Student Growth Criterion Score
Criterion 3 3 2** 5 Criterion 6 2 2** 4 Criterion 8 2 N/A 2 Student Growth Score 7 4 11
*Must include a minimum of two student growth measures (i.e., state-, district-, school-, and classroom-based measures). ** A student growth score of “1” in any of the student growth rubrics will result in a Low growth rating.
Evaluators place teachers into summative rating categories based on score bands. As illustrated below, this teacher would receive a low student growth rating
5-‐12 13-‐17 18-‐20 Low Average High
25
Summative Rating & Impact on Student Learning Matrix
26
Dale Chu Assistant Superintendent for Innovation and Improvement, Indiana Department of Education
27
Bellwether ranking of state teacher evaluation laws
Rank State Overall Ra6ng
Rank State Overall Ra6ng
1 Indiana 11.75 12 Rhode Island 6.75
2 Louisiana 10.00 13 Delaware 6.25
3 Florida 9.75 14 Connec-cut 5.75
4 Colorado 9.00 15 New York 5.75
5 Michigan 8.00 16 Arkansas 5.50
6 Oklahoma 8.00 17 Ohio 5.50
7 Illinois 7.50 18 New Jersey 5.25
8 Arizona 7.25 19 Washington 5.25
9 Nevada 7.25 20 Maryland 4.25
10 Idaho 7.00 21 Minnesota 3.00
11 Tennessee 7.00 http://bellwethereducation.org/recent-state-action-on-teacher-effectiveness/
28
There are two major components of the RISE evaluation system
Professional Practice
Student Learning
Summative Evaluation
Rating
Ineffec6ve
Improvement Necessary
Effec6ve Highly Effec6ve
29
Professional Practice: Assessment of instructional knowledge and skills
Guiding Principles 1.) Nothing matters more for students
than effective instruction. Therefore, the professional practice evaluation must focus on instruction above all else.
2.) Teachers need regular and actionable feedback on their performance in order to continually improve their practice.
3.) In order to build an effective teaching staff, administrators must be able to differentiate teachers based on performance in the classroom, and therefore, must spend more time in classrooms.
Professional Practice
Student Learning
Summative Evaluation
Rating
30
Measure for Professional Practice in RISE: The Teacher Effectiveness Rubric
Domain 1: Planning 1.1 Utilize Assessment Data to Plan 1.2 Set Ambitious and Measurable Achievement Goals 1.3 Develop Standards-Based Unit Plans and Assessments 1.4 Create Objective-Driven Lesson Plans and Assessments 1.5 Track Student Data and Analyze Progress
Domain 3: Leadership 3.1 Contribute to School Culture 3.2 Collaborate with Peers 3.3 Seek Professional Skills and Knowledge 3.4 Advocate for Student Success 3.5 Engage Families in Student Learning
Domain 2: Instruction 2.1 Develop Student Understanding and Mastery of Lesson Objectives 2.2 Demonstrate and Clearly Communicate Content Knowledge to Students 2.3 Engage Students in Academic Content 2.4 Check for Understanding 2.5 Modify Instruction as Needed 2.6 Develop Higher Level Understanding Through Rigorous Instruction and Work 2.7 Maximize Instructional Time 2.8 Create Classroom Culture of Respect and Collaboration 2.9 Set High Expectations for Academic Success
10% 75%
15%
Domain 4: Core Professionalism 1. Attendance 2. On-Time Arrival 3. Policies and Procedures 4. Respect
-1 If did not meet:
31
Student Learning: Contribution to student academic progress
Guiding Principles
1.) RISE is student-centered. Ultimately, we care whether or not students are learning. Where we can measure this directly, we should.
2.) Evaluation systems should take into account multiple measures of performance to ensure that ratings are fair and accurate to teachers.
3.) Focusing on student data helps teachers and evaluators to implement specific instructional interventions that increase student learning.
Professional Practice
Student Learning
Summative Evaluation
Rating
32
Measures for Student Learning in RISE: IGM, SWL, and SLOs
IGM: Only for teachers in grades 4-8 ELA/Math. Corporations link students to teachers. Score of 1-4 reported by state.
SWL: School grades A-F reported by state based on student learning and achievement measures. Counts equally for all teachers.
SLOs: Targets of student growth and achievement set at the start of the course that teachers and students work towards throughout year. All teachers will have student learning objectives.
Measures of Student Learning
Individual Growth
Model Data
School-wide
Learning Measure
Student Learning
Objectives
33
Teachers fall into one of three groups for the purpose of calculating a summative rating
Most 4th-8th Grade Teachers Some Elementary/ Middle Teachers
High School and PK-3rd Teachers
34
Questions from the Audience
35
Team Time and Discussion
36
Guiding Worksheet
Step 1. Identify summative rating priorities for your evaluation system.
Step 2. Review the approach(es) that are most appropriate for each aim.
Step 3. Consider whether any laws, policies, regulations, and/or past practices favor or preclude a particular approach.
Step 4. Determine a recommended approach to combining information/data/measures at each step of the evaluation process.
37
Session Wrap-Up
• Presenter contact information § Lisa Lachlan-Haché: [email protected] § Mary Ann Snider: [email protected] § Michaela Miller: [email protected] § Dale Chu: [email protected]
• For the session PowerPoint and related resources, go to www.tqsource.org