1

Key SINet Impacts:Immediate Impacts

Predictors of Success2-Year Sustained Impacts

7-Year Longevity

Prepared by: Steven H. Shaha, PhD, DBA

Prof. Steven H. Shaha, PhD, DBA

Professor, Center for Public Policy & AdministrationIndependent Evaluator

100+ peer-reviewed publications, 200+ peer-reviewed presentations, 2 books

Advisory and consulting work for 11 governments on four continents

Educational work in 16 States

Professor or Lectured or at 14 universities and graduate schools e.g. UCLA, Harvard, Cambridge (UK), King’s College, London University, South Manchester, Yale, Princeton, Columbia, Cornell, Zayed Univ (UAE)

Advisory and consulting to 50+ non-education-focused organizations, for example:Disney, Ritz-Carlton, Coca-Cola, New Line Cinema, IBM, AT&T, Time Warner, Johnson & Johnson, Marriott, 3 Depts. of Defense, and Pharma: Sanofi, Aventis, Novartis

Education: PhD, Research Methods & Applied StatisticsDBA, Business Administration (PhD)MA, MEd, BS

Disclosures & Bio

4

Executive Summary

• Next Study – Aligned Seminars• Next Study – Aligned

Implementation• Next Study – Aligned Observation

4. Advantages with Longevity into 6th and 7th Years3. Advantages Sustained into Second Year of PD 3602. Prdictors of Success

High vs. Lower Engagement1. Advantage with PD 360

versus none, Year-over-Year

Immediate ImpactsIntent: Define year-to-year changes associated with PD 360Participation

– Systematic sample of 169 elementary schools, in 73 districts, in 19 States– Schools eligible for inclusion in the sample as participating Schools met the

following criteria:• More than 10 teachers total• 80% or more of teachers viewed materials• Minimum average of 90.0 minutes of viewing per teacher for the school

– Districts included were only those for which eligible schools were included• Normalized for socio-economic and demographic factors

Data– Unit of measure was percent Proficient or Advanced as metric– Student performance data were captured from publically available, Internet-accessed

sources

Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

Immediate Impacts

58.4

69.5

62.8

65.5

52.054.056.058.060.062.064.066.068.070.072.0

Year 1 Year 2

Comparative Change in Math Performance

Schools

Districts

Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

56.9

67.2

61.262.7

50.052.054.056.058.060.062.064.066.068.0

Year 1 Year 2

Comparative Change in Reading Performance

Schools

Districts

MATH:11.1 net PD 360 gain in Pct Adv+Prof (p<0.001)

Vs. 2.6 net gain for Districts

That equals:18.9% improved year-over-yearVs. 4.2% improvement for DistrictsThat’s 4.5 times MORE Improvement than their Districts

Reading:10.3 net PD 360 gain in Pct Adv+Prof (p<0.001)

Vs. 1.5 net gain for Districts

That equals:18.1% improved year-over-yearVs. 2.5% improvementThat’s 7.4 times MORE improvement than their Districts

7

4.9% gain for lower engagement schools (p<.01)

18.0% gain for higher engagement schools (p<.001)

Closed the Gap:267% advantage in gains for higher engagement schools (p<.001)

Nearly 4 times the impact

Predictors of Success

0.5% gain for lower engagement schools (p=ns)

18.9% gain for higher engagement schools (p<.001)

Surpassed the Gap:3,520% advantage in gains for higher engagement schools (p<.001)

36 times greater impact

Shaha SH, Ellsworth H (2014). Predictors of Success for Professional Development: Linking Student Achievement to School and Educator Successes through On-Demand, Online Professional Learning. Journal of Instructional Psychology (JIP), Vol. 40, No. 1

Predictors of Success

8

Higher Engagement

Schools

Lower Engagement

Schools Difference p-va lue

Percent Difference

Educator UtilizationSegments Viewed (school cumulative) 2298.0 528.9 1769.1 0.000 334.5% 4.3 Times

Programs Viewed 588.7 223.5 365.3 0.000 163.5% 2.6 Times

Minutes Viewed 359.9 180.6 179.3 0.000 99.3% 2.0 Times

Registered Users 87.7% 49.0% 38.7% 0.000 78.9% 1.8 Times

Registered Users Viewing at 90 min 79.1% 48.7% 30.4% 0.000 62.4% 1.6 Times

Forums Viewed 138.7 87.0 51.7 0.000 59.4% 1.6 Times

Total Users Licensed 93.9% 61.4% 32.5% 0.000 52.9% 1.5 Times

Educator EngagementFollow-up Questions Answered 359.6 167.0 192.6 0.000 115.3% 2.2 Times

Reflection Questions Answered 588.3 340.4 247.9 0.000 72.8% 1.7 Times

Files Uploaded 46.3 29.3 17.0 0.000 58.0% 1.6 Times

Focus Objectives Set Up 3120.7 2149.0 971.7 0.000 45.2% 1.5 Times

Files Downloaded 45.2 35.4 9.8 0.011 27.7% 1.3 Times

Forums Posted 28.5 23.4 5.1 0.031 21.8% 1.2 Times

Pct of Users in Communities 43.0% 36.5% 6.5% 0.045 17.8% 1.2 Times

Leadership, Implementation, AccountabilityObservations Performed 65.2 29.2 36.0 0.000 123.2% 2.2 Times

Focus Objectives Set Up (school cumulative) 3120.7 2149.0 971.7 0.000 45.2% 1.5 Times

Magnitude

Shaha SH, Ellsworth H (2014). Predictors of Success for Professional Development: Linking Student Achievement to School and Educator Successes through On-Demand, Online Professional Learning. Journal of Instructional Psychology (JIP), Vol. 40, No. 1

Intent: Quantify sustained impacts of PD360Participation

– Second Year follow-on study– Systematic sample of 169 elementary schools, in 73 districts, in 19 States– Same eligibility criteria for School versus District inclusion – Districts included were only those for which eligible schools were included

• Normalized for socio-economic and demographic factors

Data– Unit of measure was percent Proficient or Advanced as metric– Student performance data were captured from publically available, Internet-accessed

sources

Glassett, K. & Shaha, S. (2014). Lessons learned from a two year implementation project: Sustaining Student Gains with Online On-demand Professional Development. In Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2014 (pp. 660-667). Chesapeake, VA: AACE.Glassett K, Shaha S & Ellsworth H. (2014). Are Student Gains Sustainable when using Online On-demand Professional Development? Association of Teacher Educators 2014 Summer Conference. Niagara Falls, NY: The Association of Teacher Educators. August 2014.

2-Year Sustained Impacts

2-Year Sustained ImpactsMATH vs. baseline:16.4 net PD 360 gain in Pct Adv+Prof (p<0.001)

13.5 more net gain than respective Districts

That equals:28.1% improved from baseline (p<0.001)

That’s 4.6 times MORE improvement than their Districts

Reading vs. baseline:17.2 net PD 360 gain in Pct Adv+Prof (p<0.001)

13.5 more net respective Districts (again)

That equals:30.2% improved from baseline (p<0.001)

That’s 5.0 times MORE improvement than their Districts

Glassett, K. & Shaha, S. (2014). Lessons learned from a two year implementation project: Sustaining Student Gains with Online On-demand Professional Development. In Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2014 (pp. 660-667). Chesapeake, VA: AACE.Glassett K, Shaha S & Ellsworth H. (2014). Are Student Gains Sustainable when using Online On-demand Professional Development? Association of Teacher Educators 2014 Summer Conference. Niagara Falls, NY: The Association of Teacher Educators. August 2014.

7-Year Longevity

Intent: What is the long-term impact of PD 360 participation

Design– Quasi-experimental, longitudinal study– 25 states and 78 school districts– Each district is included only in the longevity duration class to which it

belongs

Shaha SH, Glassett K, Ellsworth H (2015). Long-term Impact of On-demand Professional Development on Student Performance: A Longitudinal Multi-State Study. Journal of International Education Research. Vol 11(1).Glassett, K. & Shaha, S. (2014). Long-term Impact of On-demand PD on Student Performance: A Longitudinal Multi-State Study. In Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2014 (pp. 668-671). Chesapeake, VA: AACE.Shaha, S., Glassett, K., & Ellsworth, H. (2014). Long-term Impact of On-demand PD on Student Performance: A Longitudinal Multi-State Study. 2014 International Education Conference. San Francisco, CA: The Clute Institute. August 2014.

Change in Reading Perf Change in Math Perf0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

Longitudinal Effect on Change in Student Performance Versus Pre-PD Baselines

2-3 years4-5 years6-7 years

Net C

hang

e in

Pct

Adv

+Pro

f

7-Year Longevity

Reading gains from Baseline:2-3 years: 5.58 net gain in pct Adv+Prof

4-5 years: 7.34 net gain in pct Adv+Prof (p=0.042)

6-7 years: 12.77 net gain in pct Adv+Prof (p=0.037)

MATH gains from Baseline:2-3 years: 2.90 net gain in pct Adv+Prof

4-5 years: 6.92 net gain in pct Adv+Prof (p=0.031)

6-7 years: 12.15 net gain in pct Adv+Prof (p=0.009)

Shaha SH, Glassett K, Ellsworth H (2015). Long-term Impact of On-demand Professional Development on Student Performance: A Longitudinal Multi-State Study. Journal of International Education Research. Vol 11(1).

13

Executive SummaryPD360 Schools versus their Districts, Year-over-Year

– Math:• 11.1 Net gain in Math vs. 2.6 for District peers• That’s 18.9% better than Pre vs. 4.2% better for District peers• That’s 4.5 times more improvement than their peer Districts

– Reading:• 10.3 Net gain in Math vs. 1.5 for District peers• That’s 18.1% better than Pre vs. 2.5% better for District peers• That’s 7.4 times more improvement than their peer Districts

Predictors of Success– Math :

• 13.16 Net gain in for High Observation schools versus 2.22 net gain for Lower Observation Schools• That’s 24.9% improvement with High Observation & PD360 baseline versus 3.9% improvement for Lower Obs w/PD360• That’s 6.4 times more improvement for Higher Observation versus Lower

– Reading:• 12.47 Net gain in for High Observation schools versus 2.21 net gain for Lower Observation Schools• That’s 24.6% improvement with High Observation & PD360 baseline versus 3.5% improvement for Lower Obs w/PD360• That’s 7.0 times more improvement for Higher Observation versus Lower

2-Year Sustained Advantage versus their Districts– Math :

• 16.4 Net gain in Math from baseline – that’s 13.5 more net gain than District peers• That’s 28.1% better than baseline, which is 4.6 times more improvement than their peer Districts

– Reading:• 17.2 Net gain in Math from baseline – that’s 13.5 more net gain than District peers (matches Math)• That’s 30.2% better than baseline, which is 5.0 times more improvement than their peer Districts

7-Year Longevity: into 6th and 7th Years– Math net gains:

• 5.58 for educational organizations with 2-3 years of PD• 7.34 for orgs with 4-5 PD years • 12.77 for orgs with 6-7 PD years

– Reading net gains:• 2.90 for educational organizations with 2-3 years of PD• 6.92 for orgs with 4-5 PD years • 12.15 for orgs with 6-7 PD years

14

Executive Summary

Next Studies• Aligned Observation• Aligned Seminars• Aligned Implementation

4. Advantages with Longevity into 6th and 7th Years3. Advantages Sustained into Second Year of PD 3602. Prdictors of Success

High vs. Lower Engagement1. Advantage with PD 360

versus none, Year-over-Year

15

Soon to be Published

Observation Integrated– Math :

• 13.16 Net gain in for High Observation schools – versus 2.22 net gain for Lower Observation Schools

• 24.9% improvement with High Observation & PD360 baseline – versus 3.9% improvement for Lower Obs w/PD360

• That’s 6.4 times more improvement for Higher Observation versus Lower– Reading:

• 12.47 Net gain in for High Observation schools – versus 2.21 net gain for Lower Observation Schools

• 24.6% improvement with High Observation & PD360 baseline – versus 3.5% improvement for Lower Obs w/PD360

• That’s 7.0 times more improvement for Higher Observation versus Lower

16

Key Impacts:Immediate Impacts

2-Year Sustained Impacts7-Year Longevity

Predictors of Success

Prepared by: Steven H. Shaha, PhD, DBA