User vs Nonuser :A Multi-State, Multi-District Study of the Impact of Participation in PD 360

on Student Performance

Prepared by Steven H. Shaha, PhD, DBAMarch 2011

1

Overarching Research Question:

Does teacher engagement in PD 360 and Observation 360,

tools within the Educator Effectiveness System,

significantly affect student success?

Does teacher engagement in PD 360 and Observation 360,

tools within the Educator Effectiveness System,

significantly affect student success?

2

Methods

• Design: Quasi-experimental, retrospective, pre-post, normalized treatment-control / participation vs. non-participation (2009-10, 2010-11)

• Goal: Multi-State, large n with comparable student populations (matched, controlled)

• Student Change:* Metric was percent students classified as Proficient or Advanced in respective States.

3

SampleParticipation

– Systematic sample of 176 schools, in 59 districts, in 25 States• N determined by a priori Power analysis

– Schools eligible for inclusion in the sample as participating Schools met the following criteria:

• More than 10 teachers total• 80% or more of teachers viewed materials• Minimum average of 90.0 minutes of viewing per teacher for the school

– Districts included were only those for which eligible schools were included• Normalizing for difference in socio-economic and demographic factors between participating Schools and their

Districts cumulatively as the statistical comparison group

Data– Participation data were extracted from the Internet-based professional development

application as surveilled– Student performance data were captured from publically available, Internet-accessed

sources (school as unit of measure, percent Proficient or Advanced as metric)

4

Demographics of the Sample

5

State

State

WV

WI

WA

VA

TX

TN

OK

OH

NV

NM

NJ

NC

MO

MN

MD

LA

KY

KS

IN

IL

HI

GA

FL

CA

AZ

Fre

qu

en

cy

30

20

10

0

StatesAZCAFLGAHIILINKSKYLAMDMNMONCNJNMNVOHOKTNTXVAWAWIWV

PD 360 Impact AssessmentPD 360 Impact AssessmentInterpretation:Interpretation:

• Schools that leverage PD 360 experience:– Significant and meaningful gains in student performance for

Math and Reading– Bring their district performance averages up

• The biggest predictor of gains in student performance are:– Minutes per User – Teacher time using PD 360– Programs viewed and re-viewed – Teacher study within PD 360

6* Statistical significance establishes genuine differences between groups and verifies that impacts were “real” and not merely due to chance and, in this case, due to any pre-existing biases in group differences. The appropriate p-values are included with all differences explained herein.

Math

7

Gains in Proficiency Rates

8

Schools DistrictsPre Year 48.23 51.66Post Year 56.46 53.25Improvement 8.24 1.59Pct Change 17.1% 3.1%Statistical Significance p<.001 p<.001Contrasting Schools with Districts p<.001

44.00

46.00

48.00

50.00

52.00

54.00

56.00

58.00

Pre Year Post Year

Profi

cien

cy R

ate

Improvement in Math Proficiency

Schools

Districts

44.00

46.00

48.00

50.00

52.00

54.00

56.00

58.00

Pre Year Post Year

Profi

cien

cy R

ate

Improvement in Math Proficiency

Schools

Districts

NOTE: Statistical significance is affected by sample size, wherein significance is achieved with smaller increases given larger samples or numbers of students.

Consistent pattern:Despite schools beginning at a disadvantage, by year 2 they not only close the gap, but surpass their districts.

Consistent pattern:Despite schools beginning at a disadvantage, by year 2 they not only close the gap, but surpass their districts.

Gains in Advanced Rates

9NOTE: Statistical significance is affected by sample size, wherein significance is achieved with smaller increases given larger samples or numbers of students.

Schools DistrictsPre Year 30.05 33.13Post Year 36.48 33.89Improvement 6.43 0.76Pct Change 21.4% 2.3%Statistical Significance p<.001 p<.001Contrasting Schools with Districts p<.001

28.00

29.00

30.00

31.00

32.00

33.00

34.00

35.00

36.00

37.00

Pre Year Post Year

Adv

ance

d Ra

te

Improvement in Math Advanced

Schools

Districts

28.00

29.00

30.00

31.00

32.00

33.00

34.00

35.00

36.00

37.00

Pre Year Post Year

Adv

ance

d Ra

te

Improvement in Math Advanced

Schools

Districts

Consistent pattern:Despite schools beginning at a disadvantage, by year 2 they not only close the gap, but surpass their districts.

Consistent pattern:Despite schools beginning at a disadvantage, by year 2 they not only close the gap, but surpass their districts.

10NOTE: Factors or variables listed are those isolated through Regression Analyses. These are those statistically most predictive or correlated with the increases identified..

Factors/Variables Most Affecting Increases in Math Achievement Rates

Priority of Predictive Significance

Math School Prof Pct Change

Math School Advanced Pct

Change

Math District Prof Pct Change

Math District Advanced Pct

Change

1 Avg Min/UserPct Users

RegisteredAvg Min/User

Pct Users Registered

2Programs Viewed and Re-viewed

Pct Users w/Community

Programs Viewed and Re-viewed

Pct Users w/Community

3 Forums PostedLinks

UploadedForums Posted

4 Forums Viewed Forums Viewed5 Avg Min/User

Significant Predictors of Improvements in Student Achievement Rates

Factors/Variables Most Affecting Increases in Math Achievement Rates

11NOTE: Factors or variables listed are those isolated through Regression Analyses. These are those statistically most predictive or correlated with the increases identified..

Significant Predictors of Improvements in Student Achievement Rates

Priority of Predictive Significance

Math School

Prof Pct Change

Math School

Advanced Pct

Change

Math District Prof Pct Change

Math District

Advanced Pct

Change

Times Ranked as Significant Predictor

Times Ranked as

#1 PredictorAvg Min/User 1 5 1 3 2Programs Viewed and Re-viewed

2 4 2 3

Pct Users Registered 1 1 2 2Pct Users w/Community 2 2 2Forums Posted 3 3 2Links Uploaded 3 1Forums Viewed 4 1

Impact is predicted primarily by Teacher use and depth of personal investmentImpact is predicted primarily by Teacher use and depth of personal investment

Greatest Predictors of Improvement in Math

Schools:• Average Minutes per User• Numbers of Pro Viewed

Districts:• Percent of Users Registered• Percent of Users with Community• Forums Posted

12

Reading

13

Note to Reader: To better dramatize the magnitude of the consistently favorable impact of PD 360, graphics included hereafter represent a variety of perspectives and a sampling of different interpretive insights, and not an exhaustive nor uniformly arrayed set of results.

Gains in Proficiency Rates

14NOTE: Statistical significance is affected by sample size, wherein significance is achieved with smaller increases given larger samples or numbers of students.

Schools DistrictsPre Year 46.98 51.67Post Year 56.12 55.12Improvement 9.15 3.45Pct Change 19.5% 6.7%Statistical Significance p<.001 p<.001Contrasting Schools with Districts p<.001

45.00

47.00

49.00

51.00

53.00

55.00

57.00

Pre Year Post Year

Profi

cien

cy R

ate

Improvement in Reading Proficiency

Schools

Districts

45.00

47.00

49.00

51.00

53.00

55.00

57.00

Pre Year Post Year

Profi

cien

cy R

ate

Improvement in Reading Proficiency

Schools

Districts

Consistent pattern:Despite schools beginning at a disadvantage, by year 2 they not only close the gap, but surpass their districts.

Consistent pattern:Despite schools beginning at a disadvantage, by year 2 they not only close the gap, but surpass their districts.

27.00

28.00

29.00

30.00

31.00

32.00

33.00

34.00

Pre Year Post Year

Adv

ance

d Ra

te

Improvement in Reading Advanced

Schools

Districts

27.00

28.00

29.00

30.00

31.00

32.00

33.00

34.00

Pre Year Post Year

Adv

ance

d Ra

te

Improvement in Reading Advanced

Schools

Districts

Gains in Advanced Rates

15NOTE: Statistical significance is affected by sample size, wherein significance is achieved with smaller increases given larger samples or numbers of students.

Schools DistrictsPre Year 28.12 32.16Post Year 33.47 30.95Improvement 5.35 -1.22Pct Change 19.0% -3.8%Statistical Significance p<.001 p<.001Contrasting Schools with Districts p<.001

Consistent pattern:Despite schools beginning at a disadvantage, by year 2 they not only close the gap, but surpass their districts.

Consistent pattern:Despite schools beginning at a disadvantage, by year 2 they not only close the gap, but surpass their districts.

Factors/Variables Most Affecting Increases in

Reading Achievement Rates

16NOTE: Factors or variables listed are those isolated through Regression Analyses. These are those statistically most predictive or correlated with the increases identified..

Priority of Predictive Significance

Reading School Prof Pct Change

Reading School Advanced Pct

Change

Reading District Prof Pct Change

Reading District Advanced Pct

Change

1Links

UploadedPct Users

RegisteredAvg Min/User Links Viewed

2 Links ViewedPrograms Viewed and Re-viewed

Pct Users Registered

Significant Predictors of Improvements in Student Achievement Rates

Significant Predictors of Improvements in Student Achievement Rates

Priority of Predictive Significance

Reading School

Prof Pct Change

Reading School

Advanced Pct

Change

Reading District Prof Pct Change

Reading District

Advanced Pct

Change

Times Ranked as Significant Predictor

Times Ranked as

#1 PredictorPct Users Registered 1 2 2 1Links Viewed 2 1 2 1Avg Min/User 1 1 1Links Uploaded 1 1 1Programs Viewed and Re-viewed

21

17NOTE: Factors or variables listed are those isolated through Regression Analyses. These are those statistically most predictive or correlated with the increases identified..

Factors/Variables Most Affecting Increases in

Reading Achievement Rates

AND by Teacher use and depth of personal investmentAND by Teacher use and depth of personal investment

Impact is predicted by Teachers engaged and depth of engagementImpact is predicted by Teachers engaged and depth of engagement

Greatest Predictors of Improvement in Reading

Schools:• Percent of Users Registered• Links Uploaded• Link Viewed

Districts:• Average Minutes per User• Percent of Users Registered• Links Viewed• Pro Viewed

18

PD 360 Impact AssessmentPD 360 Impact AssessmentExecutive SummaryExecutive Summary• Statistically significant* advantages were verified favoring

schools with PD 360 versus District Benchmarks.• Math (p<.001)

• Reading (p<.001)

Average Gains:• Schools:

– Math Proficient Rate: 17.1% (p<.001)

– Math Advanced Rate: 21.4% (p<.001)

– Reading Proficiency Rate: 19.5% (p<.001)

– Reading Advanced Rate: 19.0% (p<.001)

• Districts as Ripple Effect:– Math Proficient Rate: 3.1% (p<.001)

– Math Advanced Rate: 2.3% (p<.001)

– Reading Proficiency Rate: 6.7% (p<.001)

– Reading Advanced Rate: -3.8% (unknown how much this was boosted by schools)

19* Statistical significance establishes genuine differences between groups and verifies that impacts were “real” and not merely due to chance and, in this case, due to any pre-existing biases in group differences. The appropriate p-values are included with all differences explained herein.

Greatest Predictors of Improvement

The most significant predictors of success were identified, which maximizes the generalizability of successes documented

• Average Minutes per User• Percent of Users Registered• Links and Pro Viewed• Percent of Users with Community

20