measuring teaching with teachers - fadss

Measuring Teaching with Teachers

Lessons from the Measures of Effective Teaching Project

ECET2 Florida May 3, 2013

2

Why do we focus on teaching?

Why start with measurement?

Multiple measures of effectiveness

Useful and actionable feedback

Targeted support & PD

Meaningful tenure

Differentiated roles

Strategic placement of

teachers

Elevated teaching profession

Improved student outcomes

3

Data Collection

•Two school years: 2009-10 and 2010-11

•>100,000 students

•Grades 4-8: ELA and Math

•High School: ELA I, Algebra I and Biology 4

MET Project Study Design

5

6

Key lesson from the MET project results:

Observers can provide reliable, objective feedback on

specific elements of instruction.

Caveats:

• Clear expectations based on a validated rubric.

• Raters should be certified, demonstrating accuracy

before beginning observations.

• Multiple observations.

Ensuring Reliable & Trustworthy

Observations

• Calibrate • Refine

• Test • Train

Video examples for anchor points

Rater certification: Don’t pass, don’t rate

Periodic tuning: Out of

tune, don’t rate

Adjustments to observation

framework based on data

7

Actual scores

for 7500

lessons.

Framework for Teaching (Danielson)

Un

sa

tis

facto

ry

Yes/no questions; posed

in rapid succession;

teacher asks all

questions; same few

students participate.

Ba

sic

Some questions ask for

explanations; uneven

attempts to engage all

students.

Pro

fic

ien

t Most questions ask for

explanation; discussion

develops, teacher steps

aside; all students

participate.

Ad

va

nc

ed

All questions high quality;

students initiate some

questions; students

engage other students.

© 2013 Bill & Melinda Gates Foundation | 8

Many roads to reliability

One more

lesson +.07

One more observer

+.16

© 2013 Bill & Melinda Gates Foundation | 9

10

Key lesson from the MET project results:

Students can provide feedback on specific aspects

of their classroom experience, which is both reliable

and predictive of student achievement.

Caveats:

• Questions should focus on specific aspects of practice

(not a popularity contest).

• Students should fill out surveys confidentially to lessen

pressure from peers and others.

Tripod Student Perception survey

Survey Statement Category Rank

1

2

3

5

Students in this class treat the teacher with respect

My classmates behave the way my teacher wants them to

Control

Control

Control Our class stays busy and doesn’t waste time

Challenge

• In this class, we learn a lot every day Challenge In this class, we learn to correct our mistakes

33 I have learned a lot this year about [the state test] Test Prep

34 Getting ready for [the state test] takes a lot of time in our class Test Prep

4 In this class, we learn a lot every day

11

12

Key lesson from the MET project results: Any

single measure is open to manipulation. • If test scores alone, risk “teaching the test.”

• If student surveys alone, risk pandering.

• If observations alone, teachers using unconventional

teaching methods would be forced to conform.

13

14

15

How much does practice differ?

16

Role of teachers in the MET project

What do you think?

What should the foundation do to ensure findings from the MET project are used to advance high-quality teacher evaluation?

How should the foundation engage teachers

in future research projects?

17

Thank You

© 2013 Bill & Melinda Gates Foundation. All Rights Reserved. Bill & Melinda Gates Foundation is a registered trademark in the United States and other countries.

@szbuhayar

info@metproject.org

Improvement-focused evaluation systems

19

Monitor validity

Ensure reliability

Assure accuracy

Make meaningful distinctions

Prioritize support and feedback

Use data for decisions at all levels

Set expectations

Use multiple measures

Balance weights

Closing the effectiveness gap

20

21

Preliminary Results

Multiple Measures are more strongly related to student performance gains that Master’s Degrees or Experience

22

-.1

-.0

5

0

.05

.1

Actu

al A

ch

ievem

ent a

fter

Ra

nd

om

Assig

nm

ent

-.1 -.05 0 .05 .1Predicted achievement using teacher's past measures of teaching.

Note: Teachers were sorted into 20 groups by their predicted student achievement relative to the randomization group mean.

Means are reported for each of the 20. Predictions are adjusted for non-compliance.

of Randomized Classrooms (Math)

Figure 1. Actual and Predicted Achievement

Actual = Predicted

Final Results

23

Students with Most Effective Teachers Learn More in School

Do students of higher-scoring teachers learn more?

24

25

1

2

3

4

5

6

7

0 0.25 0.5 0.75 1

CLASS Score Distribution

26

1

2

3

4

5

6

7

0 0.25 0.5 0.75 1

CLASS Score Distribution

27

1

2

3

4

5

6

7

0 0.25 0.5 0.75 1

CLASS Score Distribution

0.68 points

28 Invest in Improvement

Prioritize support & feedback

29

Domain Expectation Indicator

Providing

Feedback

on

Strengths

and Areas

of Growth

Communicating

Strengths S.1

The teacher’s strength is exposed in a way that encourages

reflection on how that strength impacts student learning.

Communicating

Area of Growth

G.1

The teacher’s area of growth and related evidence from the

lesson is exposed in a way that encourages reflection on how

that area impacts learning and specific next steps.

Invest in Improvement

Use data for decisions at all levels