INTER-RATER AGREEMENT IN KANSASSummer Principals AcademyJuly 22-24, 2014Abilene, KS

2

What is Inter-Rater Agreement?• Consistent

understanding• Consistent

agreement• Consistent

process• Consistent

credibility

The degree 2 raters using the same scale give the same rating in identical situations.

3

Where is Inter-rater Agreement Useful??

Inter-Rater Agreement is useful in all areas of administrative practice including:

• Kansas Educator Evaluations• Kansas Accreditation • Personnel Decisions• Family Engagement Programs• Budget Management

4

What are the Goals of Inter-rater Agreement?

• The Inter-Rater Agreement Statistical Goal is 75% - agreement with no rating more than one level apart.

• The Goal is to have an administrative Common Frame of Reference for your district such as:• Agreement with no rating more than one level apart. • Consistent understanding• Consistent agreement• Consistent process• Consistent credibility

5

How is Inter-rater Agreement Helpful??

• What is a Common Understanding?

• How does Inter-rater Agreement help administrators??

• Facilitates Quality• Provide Consistent Messaging• Impacts Increased Student Growth• Enhances Trust • Facilitates Quality Feedback• Legally Defensible • Assists Accurate Data Collection• Allows for Increase Cooperation

over Time • Passes Public Scrutiny

6

District-wide Agreement about External Criteria

• Agreed Upon External Criteria for:• Observations• Artifacts• Units of Study• Processes• Program Development• Time Limits • Walk-Throughs• Protocols

7

Pragmatics/ConsiderationsPotential Landmines

• Discuss with an Elbow Partner Possible Issues that may cause Rater Error• Leniency: Benefit of the

Doubt• Central Tendency: Safety in

the Middle• Comparing One with

Another: Stick to the Rubric• Relying on Gut too Much:

Stick to the Rubric• Evaluating Absent Evidence:

Stick to What is Observed

8

How to Train Administrators in using Inter-rater Agreement

• Inter-Rater Agreement Training – Things to Consider: • Synchronous and Asynchronous • Experience of the Participants• How to Ensure Fidelity • Use of Video Examples• Use of Case Studies• Bias Exists• Single versus Multiple Observations• Professional Judgment• Degree of Familiarity

9

Getting Started in YOUR District• Getting Your Bearings• Developing a Guidebook• Developing a Training Protocol

10

I. Getting Your BearingsPilot with Administrative Team1. Pilot Approach: Start with Safe Example:

• Use a video from YouTube• Assess Specific Area of Teaching

• Student Engagement• Teacher Activity• Classroom Environment

2. Review current issue/goal/vision/needs3. Review current practices/instrumentation

• Rubrics• Protocol• Training

11

II. Developing a Guidebook• Paper and Electronic: Synchronous and Asynchronous• EXCELLENT Administrative Council Endeavor –

Ultimately to be used during Professional Learning Times with the Staffs

• Pieces and Parts• Terms/Definitions: Frame of Reference/Common Understanding

• “Demonstrates”• “Frequently”• “Usually”

12

II. Developing a Guidebook• Pieces and Parts (continued)

• Examples and Rubrics• Areas to Observe

• Establishes/maintains a culture of safety, respect and rapport

• Knows how to make content accessible to all learners

• Plans standards-based instruction and formative assessments

• Uses a variety of instructional strategies to meet needs

• Deepens understanding through critical thinking, problem-solving and student reflection

• Analyzes student performance and provides feedback

• Collaborates to support student learning

13

II. Developing a Guidebook• Pieces and Parts (continued)

• Exemplars: Video and Artifacts• Master-Coded Videos: Leads to high level of

agreement• Example Artifacts

• Classroom Rules• Lesson Plans• Parent Note Home

• Lesson Plan for Adults: Andragogy

14

III. Developing a Training Protocol• Step-by-Step Procedure1. Initial Training: Goals/Terms

• Practice Interpreting Rubrics• Differentiate between Different Performance Levels with Examples

2. Calibration of the Players• Video and Artifacts

3. Review of the System4. Adjustments (If needed)5. Implementation6. Evaluate/Reflect/Adjust7. Re-Calibrate…

15

• Start Slowly• Start with

“Engagement” with a Small Group

• Get Agreement• Gain Confidence

Over Time• Write and Refine in

Teams

Take Aways – To Do List

16

• Start with the End in Mind• Develop Common

Understandings• Use Groups to Ensure

Buy-In• Build Measurable Steps to

Reach Each Goal

Final Thoughts

17

Contact InformationTeacher Licensure and Accreditation Team DirectorScott Myers785-296-8010smyers@ksde.org