science teachers learning from lesson analysis (stella) · science teachers learning from lesson...

Science Teachers Learning from Lesson Analysis (STeLLA)

Chris Wilson , Joe Taylor, Kathleen Roth

NARST Annual Conference, Pittsburgh PA, April, 2014

STeLLA is:

STeLLA Professional

Development Program

Student Science Content

Knowledge

Teacher Science Content

Knowledge

Teacher Pedagogical

Content Knowledge

Teaching Practice

In the context of videocase-based analysis-of-practice

A one-year professional development program for elementary teachers:

STeLLA is also: a Scale-Up Study

Population: 153 4th and 5th grade teachers from 80 schools, ~4500 students

Cohort 1 2011-2012 45 Schools

STeLLA 22 Schools

37 Teachers

Comp 23 Schools

47 Teachers

Cohort 2 2012-2013 35 Schools

STeLLA 23 Schools

42 Teachers

Comp 12 Schools

27 Teachers

STeLLA I

Lines of Research

Overview of STeLLA PD

Content Deepening Program STeLLA PD Program

Summer Institute, 2 weeks

Full-day science content sessions with university science faculty

• 4th grade: food webs, Earth’s changing surface

• 5th grade: water cycle, Sun’s effect on seasons

Half-day science content sessions with university science faculty

4th grade: food webs, Earth’s

changing surface

5th grade: water cycle, Sun’s effect

on seasons

Half-day lesson analysis study groups with BSCS PD Leaders

4th grade: Earth’s changing surface 5th grade: Sun’s effect on seasons

Academic Year, 30 hours

Content sessions with university science faculty

Lesson analysis study groups Fall: Content Area 1 Winter/Spring: Content Area 2

Key STeLLA Components Video analysis tasks: Teachers analyze videocases from other teachers’ and each other’s classrooms

Videocases include:

sequence of lesson videos

student and teacher interview videos

student written work

student pre- and posttests

Key STeLLA Components Analysis of Practice Cycles and Tools

FOCUS ON STUDENT THINKING AND

SCIENCE CONTENT STORYLINE

Key STeLLA Components STeLLA lesson plans and analysis tools

Key STeLLA Components Student pre-post tests and analysis tools

Theory of Change and Focus Today

STeLLA Professional

Development Program

Student Science Content

Knowledge

Teacher Science Content

Knowledge

Teacher Pedagogical

Content Knowledge

Teaching Practice

STeLLA Intervention Effects

Level 1 (student)

POSTij = π0j + π1jk (PRE)ij + eij

Level 2 (school)

π0j = β00 + β01 (TREAT)j + r0j

Participants Grade

Level

Effect

Coefficient (β01)

Standard

Error t-ratio d.f. p-value

Effect Size

(Hedges g)

Students 4th & 5th 2.96 0.61 4.88 77 <.001 0.40

MEASURE PERSON - MAP - ITEM

<more>|<rare>

66 .####### T+

65 .# + PO23WC

64 .# +

63 .### + PO1SEC PO3FW

62 .## + PO24WC

61 .#### +T

60 .#### + PO14WC PO7ECS

59 .### S+

58 .##### +

57 .####### + PO16ECS PO23SEC

56 .###### + PO5FW PO9ECS

55 .########## +S PO12SEC PO14FW PO19ECS PO4WC PO7SEC

54 .######## + PO11SEC PO21ECS PO6FW

53 .######## + PO11ECS PO12ECS PO15ECS PO15SEC PO16FW

52 ####### M+ PO15FW PO24ECS PO24SEC PO2SEC PO6SEC

51 .############ + PO22ECS PO22SEC PO2WC PO4FW PO5WC PO6ECS PO8ECS

50 .######### + PO17SEC PO21WC PO7WC PO9SEC

49 .###### + PO20ECS PO20FW PO2ECS PO3SEC PO5SEC

48 .############ +M PO13SEC PO13WC PO14ECS PO17WC PO1ECS PO23ECS PO4SEC

47 .####### + PO15WC PO16WC PO19SEC PO19WC PO1FW PO20WC PO21FW PO21SEC PO3ECS

46 .#### + PO16SEC PO18SEC PO18WC PO8FW

45 .####### + PO10FW PO12WC PO14SEC

44 ####### S+ PO10ECS PO13ECS PO20SEC PO4ECS

43 .### + PO23FW PO24FW PO8SEC PO9WC

42 .#### + PO12FW PO13FW PO5ECS

41 .### +S PO10SEC PO10WC PO11FW PO18ECS PO18FW

40 .### + PO17ECS PO22FW PO22WC PO2FW

39 .# +

38 .# + PO19FW PO6WC PO8WC

37 .# T+ PO1WC PO9FW

36 .# + PO3WC

35 . +T PO17FW

34 . + PO7FW

33 . +

32 +

31 . +

30 + PO11WC

29 . +

<less>|<frequent>

EACH "#" IS 20: EACH "." IS 1 TO 19


<more>|<rare>

66 .####### T+

65 .# + PO23WC

64 .# +


62 .## + PO24WC

61 .#### +T

60 .#### + PO14WC PO7ECS

59 .### S+

58 .##### +

57 .####### + PO16ECS PO23SEC

56 .###### + PO5FW PO9ECS


54 .######## + PO11SEC PO21ECS PO6FW









45 .####### + PO10FW PO12WC PO14SEC






39 .# +



36 .# + PO3WC

35 . +T PO17FW

34 . + PO7FW

33 . +

32 +

31 . +

30 + PO11WC

29 . +

<less>|<frequent>


Pre Mean = 43

Post Mean = 52


<more>|<rare>

66 .####### T+

65 .# + PO23WC

64 .# +


62 .## + PO24WC

61 .#### +T

60 .#### + PO14WC PO7ECS

59 .### S+

58 .##### +

57 .####### + PO16ECS PO23SEC

56 .###### + PO5FW PO9ECS


54 .######## + PO11SEC PO21ECS PO6FW









45 .####### + PO10FW PO12WC PO14SEC






39 .# +



36 .# + PO3WC

35 . +T PO17FW

34 . + PO7FW

33 . +

32 +

31 . +

30 + PO11WC

29 . +

<less>|<frequent>


BaU Post = 49.73

STeLLA Post = 53.22


<more>|<rare>

66 .####### T+

65 .# + PO23WC

64 .# +


62 .## + PO24WC

61 .#### +T

60 .#### + PO14WC PO7ECS

59 .### S+

58 .##### +

57 .####### + PO16ECS PO23SEC

56 .###### + PO5FW PO9ECS


54 .######## + PO11SEC PO21ECS PO6FW









45 .####### + PO10FW PO12WC PO14SEC






39 .# +



36 .# + PO3WC

35 . +T PO17FW

34 . + PO7FW

33 . +

32 +

31 . +

30 + PO11WC

29 . +

<less>|<frequent>


BaU Post = 49.73

STeLLA Post = 53.22

The primary source of energy for a food web comes from:

Where does water in a lake get most of the energy it needs to evaporate?

What is the connection between clouds and rain?

Why is the equator hotter than the poles?


<more>|<rare>

66 .####### T+

65 .# + PO23WC

64 .# +


62 .## + PO24WC

61 .#### +T

60 .#### + PO14WC PO7ECS

59 .### S+

58 .##### +

57 .####### + PO16ECS PO23SEC

56 .###### + PO5FW PO9ECS


54 .######## + PO11SEC PO21ECS PO6FW









45 .####### + PO10FW PO12WC PO14SEC






39 .# +



36 .# + PO3WC

35 . +T PO17FW

34 . + PO7FW

33 . +

32 +

31 . +

30 + PO11WC

29 . +

<less>|<frequent>


STeLLA Post = 53.22

BaU Post = 49.73

Some Conclusions

This study is yielding strong evidence that the STeLLA Lesson Analysis approach is efficacious for students, even when non-developers deliver the intervention.

Effects are statistically and practically significant.

Next Steps

• Fold in Colorado state science assessment data and student demographics.

• Add in teacher content knowledge, teacher PCK, and teacher practice measures.

• Examine how teacher learning predicts student learning.

• Look at which STeLLA strategies were most predictive of student learning.

• Qualitative analyses.

An Issue

“Education reform usually arrives with fanfare, great expectations, and overconfidence. Truth be known, typical education-reform effects tend to be small. Evaluations, if done at all, burst the reform balloon, having difficulty finding effects.”

Ruiz-Primo et al., 2002

Ab

ility

Expert Novice

The Measurement Challenge: 1. An Instrument used to Assign Grades

Pretest Posttest

Growth

The Measurement Challenge: 2. An Instrument to Measure Change over Time

Pretest Posttest

*Fidelity of Implementation and Achieved Relative Strength

Control for Pretest

Pretest Posttest

The Measurement Challenge: 3. An Instrument to Differentiate between the Effects of Two Treatments over Time

Treatment Comparison

The Measurement Challenge: 4. An Instrument to Differentiate between the Effects of Two Treatments over Time of a Treatment at the Teacher Level on Learning at the Student Level.

Pretest Posttest


Trickle Down

Teacher Level Student Level

PD Program Student

Outcomes Teacher Learning

Pretest Posttest Pretest Posttest Pretest Posttest


We clearly need very precise measurement instruments…

But due to the scale of many efficacy studies, we are often limited to more crude measures.

(e.g. 200 teachers, pre and post, 4 topics, 30 student per class = ~48,000 student tests)

Instructional Sensitivity

• Learning goal specification and monitoring

• Developing construct models

• Item selection and development

• Pilot testing and item/instrument analyses

Learning Goals Learning Goals Curriculum Materials

Backward Design (Wiggins and McTighe, 1998)

Curriculum Mapping (Ruiz-Primo et al., 2010)

1

2

Item Number

Difficulty Pre

Difficulty Post

Item Discrimination

Item-Total Correlation

Pre-Post Difference

Instructional Sensitivity

Flag

7 .32 .65 .77 .51 .33 .49 8 .12 .53 .33 .52 .41 .47 9 .21 .29 .21 .19 .08 .10 X

NOVICE EXPERT

Increase in Sensitivity Example

To download this session please go to bscs.org/sessions

science teachers learning from lesson analysis (stella) · science teachers learning from lesson...

Documents