inter-rater reliability on observation practices “good observations provide good feedback which...
TRANSCRIPT
Inter-Rater Reliability on Observation Practices
“Good observations provide good feedback which leads to support of instruction resulting in student growth”
(Certification Council Teacher Evaluation, 2011).
2
Teachers are immune to feedback from a coach or administrator when they have different definitions of quality.
The single most important thing that a school leader can do is reach agreement with the staff about quality.
Enhancing RTI (ASCD, 2010) Fisher and Frey, 2010
3
Is there a shared definition of good teaching in your school/district?
1. We all agree on what excellent, good, mediocre, poor teaching looks like.
2. We agree on what good teaching looks like.
3. There are some disparities within the school or across the district.
4. There are many different opinions on what good teaching is.
A Shared Definition?
1 2 3 4
0% 0%0%0%
1. We all agree on what excellent, proficient, basic or does not meet looks like.
2. We agree on what good teaching looks like.
3. There are some disparities within the school or across district.
4. There are many different opinions on what good teaching is.
Goals for Inter-Rater Reliability3 Day Training 2011-2012
• To know how inter-rater reliability improves instruction
definitionlevels of performance
• To apply consistent, fair ratings across classrooms and schools using standards based outcomes with criteria.– Look fors/critical attributes– Rate proficiency
• To understand how using rubrics can be a systemic approach to support professional development and a culture for learning.– Professional conversations/conferences
AM AGENDA
• What is Inter-rater Reliability?– Why do we need it?
• What is effective teaching?• Why is evidence important to inter-rater
reliability?• What are the look-fors/attributes for one
component of instruction?• How would we rate this teacher: Practicing!• Putting it all together: Video Observation• Next steps…
PM AgendaAdministrative Evaluation Process
and Teacher/Principal Inter-rater Reliability
Research on Improving the Instructional ProcessBackground information on the rubrics Components of the supervision/evaluation
processChallenges and supportInter-rater reliability for supervisors of
administrators
Quotes
Pick a quote from the center of the table.
Read the quote to yourself and be prepared to share its implications for our work as instructional leaders.
When your table group is ready, start with the person with a birthday closest to today.
Read your quote and share its implications.
McKinsey & Co. (2007)
“And the evidence is indisputable: you can’t
improve student learning without improving instruction” (p.11).
Marzano (2007), p. 1
Among elements such as a well-articulated curriculum and a safe and orderly environment, the one factor that surfaced as the single most influential component of an effective school is the individual teachers within that school.
Danielson (2009) ..When we consider the importance of ongoing
teacher learning for the success of schools, then it is essential to consider how best to promote that learning (p. 3).
Professional competence also, at times, requires maintaining the focus of all educators on the important principles of teaching and learning, and not permitting them to be distracted by the shifting winds of fashion (p.19)
• Louis and Miles (1990) found that no other type of classroom change causes more conflict than attempting to change the instructional practices of teachers.
• Researching and valuing instruction is one thing, implementing and collectively sustaining it systemically over time is another.
Leithwood, Day, Sammons, Harris, and Hopkins (2007)
In a ten year study of leadership in three countries, the researchers found that the best predictor of student achievement is improving the instructional practices of teachers. The second most powerful factor is the leadership practice of the principal in facilitating student and teacher learning.
“..different approaches proven to be effective all begin with what it
takes to improve the quality of a single teacher, and then develop
the systems to create these conditions for all teachers” (p. 32).
McKinsey & Co. (2007)
What’s happening in your school?
How are teachers improving their practice?
What are you doing to facilitate improvement of practice?
Get the right people to become teachers
Developing them into effective instructors
Ensuring that the system delivers the best possible instruction for every child
1. Strong processes for selection and training
2. Pay good starting compensation.
3. Manage the status of profession.
1. Coaching classroom practice.
2. Moving teacher training into the classroom.
3. Develop strong school leaders.
4. Enable teachers to learn from each other.
1. Set high expectations for all students.
2. Monitor and intervene at school level.
3. Monitor and intervene at the student level.
Things matter most:
McKinsey & Co. Report (2007)
Inter-Rater Reliability
Systemic approach to calibrate observations for consistent and fair professional practice
ratings.
Establish common
vocabulary
RESEARCH FINDINGSCINCINNATI’S USE OF A RUBRIC (2010)
• Teachers have substantial effect on student achievement
• Correlation between evaluation using Danielson’s rubric and student achievement
• Evaluation using rubric found– Unsatisfactory and basic: students had lower gains than
expected (based on test scores from previous years)– Proficient: students made expected gains– Distinguished/Exemplary: students made positive gains– Discriminating between basic and proficient is most difficult
Findings continued…Training was needed:1. On collecting evidence from classroom observations and
artifacts.2. A deeper understanding of the knowledge and skills in
each rubric3. On conducting reflective conferences for growth.
A study of 6000 teachers (MTA) found only 34% of teachers had feedback suggesting strategies or professional development for feedback.
A study with New Teacher Project found 73 % of most recent observations/evaluations did not identify any areas for growth.
What do Effective Teachers DO!!!!!
A Memorable Teacher
Imagine you are in a classroom of a highly effective teacher:
What would you see the teacher doing, hear the teacher saying?
What would you see the students saying or doing?
Effective teachers….• Put a greater emphasis on meaning vs. memorization• Had coherent content and clarity of objectives• Constantly engaged the students• Assessed students frequently• Had well managed and 0rganized classrooms• Used allocated time for instruction• Used a variety of materials• Had higher expectations for students• Were enthusiastic• Exhibited support, fairness, and respect
Adapted from Good (2010)
Effective teachers….• Put a greater emphasis on meaning vs. memorization• Had coherent content and clarity of objectives• Constantly engaged the students• Assessed students frequently• Had well managed and organized classrooms• Used allocated time for instruction• Used a variety of materials• Had higher expectations for students• Were enthusiastic• Exhibited support, fairness, and respect
Adapted from Good (2010)
Danielson-based:Redmond, Crook Co., Oregon City, Sisters
Domains/Standards Planning and PreparationClassroom EnvironmentInstructionProfessional Responsibilities
Components 22 Components in 4 Domains
Elements Additional descriptors that define component
Levels of Performance 4
Levels of Performance
Exemplary Proficient Basic Does Not Meet Standard
Instruction Component of Rubric
Danielson-based:Cook County, Redmond, Sisters, Oregon City
Domain 3 Instruction
Components 3a. Communicating with students3b. Using Questioning and Discussion Techniques3c. Engaging students in learning3d.Using assessment in Instruction3e. Demonstrating flexibility and responsiveness
Watch Video
Collect data you could use to rate this teacher’s performance in 3C: Engaging Students.
Compare your data with an elbow partner. Confirm that you have data (see/hear/count) and not opinion.
Compare your data to the rubric.
Rating Effective Instruction
Using your clicker, rate this teachers’ performance based on your rubric using the numbers 4, 3, 2, or 1.
Rate this teacher’s performance based on your rubric 1-4.
1 2 3 4
0% 0%0%0%
1. does not meet standard
2. basic 3. proficient4. exemplary
Effective Rubrics are…..
• Definitions of effective teaching• Based on research
– Human Development Theories– Principles of Learning– Learning Theories and Brain Research
• Research based– Using rubrics improves teacher practice and
student achievement
Evidence vs.. OpinionE or O?
1. Some students have difficulty staying on task.
2. The teacher asked five yes/no questions in rapid succession.
3. The teacher said, “The Civil War was a tragedy for the U.S. civilization.”
4. The seating arrangement should be flexible because it is a kindergarten class.
5. “I assure you that today’s lesson will be quite interesting.”
6. The last activity, ‘discussion of the key scene,’ was rushed.
7. The teacher has clearly planned and organized for maximum effect.
8. As the activity progressed, students started calling out, “What should we do next?”9. The new table arrangement encourages concentration and interaction with one’s neighbor.
10. The pacing of the lesson was slow, allowing for student restlessness, disengagement, and disruptive behavior.
EvidenceData
Data
Data
Judgment
Interpretation
EXAMPLES of TYPES of EVIDENCE!• Verbatim scripting of teacher or student comments:
“Bring your white boards, markers and erasers to the carpet and sit on your square.”
• Non-evaluative statements of observed teacher or student behavior:Teacher presented the content from the front of
room.• Numeric information about time, student participation,
resource use, etc.:Two groups started on the assigned project immediately, one group talked for five minutes before starting.
• An observed aspect of the environment:Desks were arranged in groups of four with room to walk between each
group.
Evidence Reminder
• Factual Information– What the teacher says/does– What the student says/does– Numbers– Time– Artifacts
Reviewing Evidence
Review your notes with elbow partner.
Read samples of evidence collected.
Does it meet criteria for evidence?
Review Original Rating
Look at specific statements for each level of performance for the component “engagement”.
Review evidence collected on what teacher/student said or did
Match your data with the best description for level of performance.
Rate the teacher
Teaching….cannot be considered proficient or distinguished if students are not thinking or doing the learning themselves.
How Does a Teacher Engage?Becoming More Specific
• Developing a common definition of ‘engaging students’.
• Why engage students?• How often should we engage students?
Centerpiece of Each Rubric
• Cognitive Engagement
• Constructivist Learning
Discuss a definition of cognitive engagement.
What would it look like in the classroom at the proficient level?Share an example.
Centerpiece: Engaging Students• Teaching is a behavior that increases the probability of learning:
actively and mindfully process and practice what they are learning
• Constructivist Learning– Learning is done by the learner– The active nature of learning and how to promote it– Proficient practice has evidence of learning experiences designed to facilitate
students’ construction of knowledge.
• Cognitive Engagement– Proficient practice has evidence of students engaged cognitively.
“To meet the needs of our citizens of our future, nothing less will do.” (Danielson, 2007, p. 17)
Engagement at Proficient LevelREDMOND, SISTERS, OREGON CITY, LEBANON, CROOK CO.
3 c: Engaging students in learning:
Activities and assignments, materials, and groupings of students are fully appropriate for the instructional outcomes and students’ cultures and levels of understanding All students are engaged in work of a high level of rigor. The lesson’s structure is coherent, with appropriate pace.
Element: Activities and Assignments
Most activities and assignments are appropriate to students and almost all students are cognitively engaged in exploring content.
Beyond Definitions: Critical Attributes
All
Processing the learning (exploring content )
Beyond Definitions: Critical Attributes
All Simultaneously
Throughout the lesson
Processing the learning (exploring content )
More ExamplesDo these meet the criteria?
• Write one question you might find on a test about today’s lesson. Trade questions with your neighbor and write a response. Then trade again and ‘grade’ the answer.
• Here are some questions about the main character in the story. I’ll read one and call on volunteers to answer them.
• Using your response cards, agree or disagree, show your response to this statement: Free trade will have a long term benefit for Canada, US, and Mexico. Be prepared to share your answer.
• Who can tell me how to factor an equation?
Engaging Students/Activities and Assignments
A1. Discretionary
2. One by one
B1. Mandatory/all
2. Simultaneous
Engaging Students/Activities and Assignments
Not engaging
1. Discretionary
2. One by one
3. Long stretches of telling without processing.More than 7 minutes
Engaging1. Mandatory/all
2. Simultaneous3. Throughout the lesson
2-3 min. for primary5-7 min inter./ 7- 10 hs/adult
4. Processing the learningDoes the engagement activity focus students on the learning?
Observing Classroom practice
Watch the video and collect data on examples of engagement using these critical attributes:
Were all students engaged simultaneously?
Was the engagement t/o the lesson? (based on level)
Were the engagement strategies focused on the learning?
Using your evidence….
Compare examples with elbow partners…Reach consensus on your examples…………….
Rate this teacher’s level of proficiency using the attributes and your rubric as a 4, 3, 2, or 1.
Rate this teacher’s performance based on your rubric 1-4.
1 2 3 4
0% 0%0%0%
1. does not meet standard
2. basic 3. proficient4. exemplary
Putting it all Together: Another Video
Watch videoCollect data on what teacher/student says or
doesAnalyze data using criteria/attributes/look-forsRate the teacher
EVALUATION
Self-assessmentGoalsPGP
Key Components of Professional Learning
• Reflection on Practice
• Collaboration and Professional Conversations
• Self-assessment
• Self-directed Inquiry
• A Community of Learners
Key Components of Professional Learning
• Reflection on Practice
• Collaboration and Professional Conversations
• Self-assessment
• Self-directed Inquiry
• A Community of Learners
Professional Conversations to Promote Growth
• Growth oriented: Rubrics and levels of performance provide targets for growth and conversations include teacher reflection and feedback for growth.
• Collaborative and reflective• Evidence based• Timely• Developmental: reinforcement and refinement
Key Components of Effective Conversations/Conferences
1. Setting the tone, building relationships, and overview of conference.
2. Teacher Reflection – what was effective? What would you differently or refine next time?
3. Reinforcement – Area of strength to reinforce continued use
4. Refinement/Growth – Area that needs refinement or growth
5. Next Steps
Summary
Name 3 attributes or look-fors for engaging students.
Name 2 reasons for collecting evidence.
Name 1 component of effective supervision.
Next StepsDay 2 of Training: Review observation
homework Study another component from instruction domain for
inter-rater reliability Conferencing for teacher reflection and reinforcement
Day 3 of Training: Another component of instruction Conferencing for professional growth
PM AgendaAdministrative Evaluation Process
and Teacher/Principal Inter-rater Reliability
Research on Improving the Instructional Process
Where did the rubrics come from?Components of the supervision/evaluation
processChallenges and supportInter-rater reliability for supervisors of
administrators
Inter-Rater Reliability
Systemic approach to calibrate observations for
consistent and fair professional practice ratings within a district and across
TIF districts.
Establish common
vocabulary
Principal Inter-rater Reliability
TIF RequirementsStandards and Criteria for
Effective Leaders (Rubric)Two ObservationsCalibration of at least one
observationAdministrator Proficiency
Best PracticesStandards and Criteria defining
effective leaders (rubric)Formative Activities
Self-assessmentGoalsObservations of multiple settingsSelf-reflection
Summative Evaluation
RUBRIC BACKGROUND
Who developed it?Why was it selected it?What voice will administrators have in the
development or revision of handbook?Who is my supervisor?Who will be observing me?
Administrative Evaluation Handbook
What knowledge/skills would you expect to be evaluated on?
What activities would you like to see as part of your supervision and evaluation process?
Focus 2011-12: Self-Assessment
IV. Learning and Teaching
Leader implements consistent quality classroom routines and instructional strategies.
Read each level of performance for this component self-assess on this component.
Components of the ProcessFORMATIVE:
Self-assessmentGoal settingObservationsArtifacts (portfolio of evidence)
SUMMATIVE: Admin:
Self-reflection on growth Goals for next year
Superintendent: Summative evaluation
Observations for 2011-12
Winter-Spring:
Supervisor observes a faculty meeting or professional development session when the administrator will be modeling effective instructional practices.
Spring:
Supervisor observes with administrator in classroom, observes post conference with teacher and provides to administrator.
ArtifactsEach school’s improvement plan will be
reviewed by central office administrative team to administrator’s plan for:
• Monitoring of classroom routines and instructional practices.
• Measuring instructional practices and prioritizing strategies for improvement.
• Engaging grade level content area teams in planning processes and monitoring processes.
• Implementing follow-up and support structures for instructional strategies and routines.
Implementation
What challenges do you see?Time….
What support do you need to be successful?
Professional development on…
Feedback on Day 1Advice for August 29
Significant Learnings?
Suggestions?
Questions?