EXPLORING INTER-RATER RELIABILITY

Professional Learning Team Conference

June 2014

Desired Outcomes:Each participant will leave the session

knowledgeable about inter-rater reliability

What it is Why it’s important, and How to use evidence to consistently

and accurately rate teachers’ performance

Goals for Inter-Rater Reliability 2014-2015

Demonstrate familiarity with district rubric

Describe proficient practices based on district rubric descriptors

Provide feedback to educators based on evidence collection

Calibration of evaluators

PARTICIPANTS WILL:

Establish a common vocabulary around inter-rater reliability

Be able to explain levels of performance with look-fors

Gain a deeper understanding of proficient level of performance aligned to standards

GOALS:

CHINESE PROVERB:What I hear, I forget 5% retention after 24 hoursWhat I see, I remember

30% retention rate if use of audio-visual and demonstration

What I do, I understand 80% retention rate with

discussion and practice by engagement

NORMSShare experiences to enrich othersAsk questionsLearn by doingSet aside any preconceived notions about evaluating educatorsApply your own work

LEARNING TARGET:Participants will become familiar with look fors of effective practice and be able to use that knowledge to assess performance in three focused areas:

Student Engagement

Questions, Prompts & Discussion

Assessment

NATIONAL FOCUS ON TEACHER EFFECTIVENESS AND EVALUATION

EFFECTIVENESS:

Teacher effectiveness is the single most important school-related factor affecting student learning.

EVALUATION:

However, most schools fail to evaluate teachers in a meaningful way, with 94% of teachers receiving the highest ratings.

Teacher evaluations in five urban school districts, based on data taken from a report by The New Teacher Project: http://widgeteffect.org/downloads/TheWidgetEffect.pdf.

Teacher evaluations in five urban school districts, based on data taken from a report by The New Teacher Project: http://widgeteffect.org/downloads/TheWidgetEffect.pdf

a systemic approach to calibrate observations for consistent and fair professional practice ratings

INTER-RATER RELIABILITY:

Learning Experience:

At your table define and describe

Student Engagement

Capture your collective thoughts on chart paper

Educational author and former teacher, Michael Schmoker shares in his book, 

Results Now, a study that found of 1,500 classrooms

visited, 85 percent of them had engaged less than

50 percent of the students

In other words, only 15 percent of the classrooms had more than half of the class at

least paying attention to the lesson

You will see students... Paying attention (alert, tracking with their eyes) Taking notes (particularly Cornell) Listening (as opposed to chatting, or sleeping) Asking questions (content related, or in a game) Responding to questions (whole group, small group, four

corners, Socratic Seminar) Following requests (participating, Total Physical

Reponses (TPR), storytelling, Simon Says) Reacting (laughing, crying, shouting, etc.) Reading critically (with pen in hand Interacting with other students

Michael Schmoker

WHAT DO "ENGAGED" STUDENTS LOOK LIKE?

Activities aligned with the goals of the lesson Student enthusiasm, interest, thinking, problem-solving… Learning tasks that require high-level student thinking

and are aligned with lesson objectives Students are highly motivated to work on all tasks and

are persistent even when the tasks are challenging Students actively "working," rather than watching while

the teacher "works“ Suitable pacing of the lesson: neither dragging nor

rushed, with time for closure and student reflection

ENGAGING STUDENTS IN LEARNING

When determining the level of performance for Engaging Students in Learning,

consider the quality of activities, assignments, and resources the instructor

uses to pull students into the lesson

It is also important to understand how the lesson uses student grouping and pacing

when determining the level of performance for

this component

STUDENT ENGAGEMENT

Learning Experience:

At your table define and describe

Assessment

Capture your collective thoughts on chart paper

is to support student learning and

to communicate that learning to others

THE PURPOSE OF  CLASSROOM ASSESSMENT

In order to support student learning classroom assessment needs to:

involve students deeply in the assessment process

provide specific, descriptive feedback during the learning, and

include evaluative feedback as required to communicate and report progress over time

ASSESSMENT:

To ensure success of all:1) students need to know what they

already know2) know what needs to be learned and 3) know what success looks like

Students also need to learn how to guide their own learning through being involved in setting and

using criteria, giving themselves feedback for learning (self-assessment), setting goals, collecting

evidence and communicating that evidence of learning to others

ASSESSMENT:

The teacher pays close attention to evidence of student understanding

The teacher poses questions specifically created to elicit evidence of student understanding

The teacher circulates to monitor student learning and to offer feedback

Students assesses their own work against established criteria

The teacher adjusts instruction in response to evidence of student understanding (or lack of it)

INDICATORS THAT MAY BE USED AS A GUIDE FOR LOOKING FOR EVIDENCE OF ASSESSMENT

BREAK

Work with your District partners to identify the elements in your rubric that focus on

Student EngagementQuestion, Prompts & Discussion

Assessment

Learning Experience:

While observing the video, collect evidence that you will later use to rate the teacher’s

performance using your district rubric

Learning Experience:

You will use your district’s rubric to rate the teacher’s performance

in the following areas:Student EngagementQuestion, Prompts & DiscussionAssessment

Remember…

VideoMath Lesson: CC Standard 7.G.6

https://www.teachingchannel.org/videos/surface-area-lesson#

LUNCH

In just a moment you will be asked to review the data

you’ve collected, cluster and code it for easy interpretation, and rate the

teacher’s performance using your district’s rubric

DATA

DATA

DATA

Evidence

Interpretation

Judgment

WHAT IT LOOKS LIKE TOCLUSTER AND CODE

ANOTHER METHOD OF EXAMINING YOUR EVIDENCE

Student: Um, how he has the equation. CM: Exactly, how he has the equation so every calculation starts with his equation. I like that too, good. Kay-Wen, keep it moving. Student: He does it step by step. CM: What do you mean? Student: Well, for the circumference, see how he switched the d – 4:11 CM: Into a…? Student: Where is it? CM: Here. Into the diameter of 60, so that’s one step, that step’s called substitution. Very good, I like that, Kay-Wen. What else do we like, Corrine? Student: I like how it’s organized and that there’s no mistakes and he’s not skipping steps. CM: Absolutely, did any of you guys have to do a double take, like look at it like ugh, what does that say? No, it’s very neat and organized, it’s clearly stated, very good. Taylor, what else? Student: Um, his answers are circled so that we know where to look for the actual answer.

Feedback that guides performance

Select student at randon to participate

Prompts

Feedback to reinforce behavior rather than instruction

Fade 10:44 Student: We’re using a basic strategy, but I think it might work. CM: Let me, let me see. First of all, start your strategy off, what was your strategy? Student: Our strategy was um, height by circumference would equal the surface area of the cylinder and the reason is because – CM: The whole cylinder? Which part? Student: This part. CM: Which part? Student: The part around. CM: Okay, is that it?

Teacher circulates, checking understanding and asking clarifying question to deepen understanding Assessing learning

ANOTHER METHOD OF EXAMINING YOUR EVIDENCE

With an elbow partner, use the verbatim transcript of the teacher’s observation to identify and organize

evidence to accurately and fairly rate the teacher’s performance in the

areas of

Student engagementQuestions, Prompts and Discussion

Assessment

Look at your evidence for each component.  Where does the preponderance of evidence align with levels of performance?

Match your data with the rubric performance level description

Assign a performance level

Use your clicker to rate this teacher’s performance using the numbers

Unsatisfactory

Basic Proficient

Distinguish

TESTIDENTIFY YOUR FAVORITE

CANDY BAR

1. Snickers2. Milky Way3. Kit Kat4. Trix

Vikings

Beavers

Boxers

Ducks

64%

0%9%

27%

1. Unsatisfactory2. Basic3. Proficient4. Distinguish

Unsatisfacto

ryBasic

Proficient

Distinguish

0%

27%

64%

9%

STUDENT ENGAGEMENT

DISCUSS

1. Unsatisfactory2. Basic3. Proficient4. Distinguish

Unsatisfacto

ryBasic

Proficient

Distinguish

0% 0%

88%

13%

QUESTIONS, PROMPTS & DISCUSSION

DISCUSS

1. Unsatisfactory2. Basic3. Proficient4. Distinguish

Unsatisfacto

ryBasic

Proficient

Distinguish

0%

18%

82%

0%

ASSESSMENT

DISCUSS

ONE – TWO PUNCH

One thing you heard today that you want to

remember

Two things you heard that you want to act on

before the next PLT Conference

John Doe1234 Paradise LaneAnywhere, OR 97890

QUESTIONS

Thank you for your active

participation and cooperation