1

Studying Teacher Professional Development: Challenges and

Possible Solutions

Nicole KerstingResearch Scientist

LessonLab Research Institute, Santa Monica, CA

2

Overview of Presentation

• PD Program under study

• Challenges to implementation & possible solutions

• Challenges to evaluation & possible solutions

3

Study Background and Goals

• Builds on key findings of TIMSS 1999 Video Study.

• Aims at helping 6th-grade mathematics teachers to maintain the cognitive level of rich mathematics problems.

• Key topic areas: fractions, ratio/proportion, and variables/expressions

4

PD ProgramCan we get teachers to effectively engage students

in thinking about math concepts?

• Three video-based modules, each focusing on:• Mathematics content knowledge• Analysis of videotaped lessons (strategies

for maintaining problems’ richness and student lens)

• Teaching of videotaped lesson• Sharing meetings

• Total of 45 face-to-face hours– 7 pull-out days (42 hours) distributed during

the year– 3 sharing meetings (3 hours)

5

Research Questions

• Does the PD program impact teacher knowledge?

• Does the PD program impact teaching practices?

• Does the PD program impact student learning?

6

Sample

• Predominantly low-income district in Los Angeles

• 5 Title-1 middle schools• All 6th-grade teachers (=71); PD program

made mandatory by the district• All 6th-grade students (approx 4400 in

137 classes)• Student population: 73.6% Latino and

25.9% African American; 40% are English learners

• 6.2% of students are mathematics proficient (as measured by state tests in 2004)

7

Design

Year 1

Random assignment within schools and tracks: Treatment and no-treatment group

Year 2

All teachers participated in PD program (1 year vs. 2 years of PD)

8

Key Measures - Teachers

• Content and pedagogical content knowledge – Multiple-choice survey: Items from Study of Instructional Improvement (Ball & Hill, University of Michigan); California Professional Development Institutes (MPDI - UC office of the president); and developed in-house– Administered at 3 points in time; 2 equivalent forms

• Video Analysis AssessmentAdministered prior and at the end of Year 2

• Compliance Ratings

9

Key Measures – Teaching Practices

• Videotapes of lessons including a rich problem

• Fieldnotes from classroom observations

10

Key Measures - Students

• State standardized tests (most distal)

• Subset of district quarterly assessment items targeting key content areas

• In-house student performance assessment items (most proximal)

11

Key Features of PD Program

• District mandated teacher participation

• Multi-site randomized field trial

• Title 1 schools

• Multi-track schools and calendars

12

Challenges to Faithful Program Implementation

Possible Solutions

• Full participation of all teachers• Opposing instructional goals

Involve every level at the school and district

• Complications of multiple track calendarsConstant contact with the person at

the school “Try to get on their calendar and try to stay on it”.

13

Challenges to Faithful Program Implementation

Possible Solutions

• Role of math coachesDon’t make assumptions about district

and school personnel position responsibilities.

• Changes in school and district personnelDon’t presume personnel stability.

14

Challenges to Experimental Design

Possible Solutions• Group contamination: students are

moved across classrooms obtain class rosters to assess percent

of students moved.• Teacher attrition (especially for programs

with more than one year implementation). be conservative to ensure sufficiently

large sample size for inferential statistics

15

Challenges to Experimental Design

Possible Solutions

• Compliance: 6/28 teachers in Year 1 and 5/38 teachers in Year 2 were rated as fully compliant Create compliance scale to be

able to evaluate differential program effects for compliant and non-compliant teachers.

16

Challenges to Experimental Design

Possible Solutions• Threats to the validity of measures

– Low motivation (PCK survey is time consuming)

– Resistance to being assessed (exposes teachers’ lack of knowledge)

embedded measures of teacher knowledge

• Lack of control over collecting student measuresUse own resources to collect data

17

Conclusions

• Conducting experimental research in schools (particularly in disadvantaged schools) is time consuming and difficult

• Partnerships between researchers and schools are extremely valuable

• Disadvantaged schools deserve our attention