wsu greg lobdell september 2008 data and decision making
DESCRIPTION
TRANSCRIPT
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
The Use of Data to Inform Instruction, Building, & District Decisions
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Greg LobdellCo-founder & Research Director
19 September, 2008
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
CEE & Data: Supporting a Cycle of Continuous Improvement
• Expand Capacity• Provide Value-add services• Expertise in core areas• Partnership is critical
CEE Services
OSPI & Summit Partners
ESDs
Districts
Schools & Classrooms
Demographics & Community CharacteristicsDemographics & Community Characteristics
PerceptualAcademic
AchievementContextual-Program and
Process
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Who we serve . . .
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Who We ServeWASHINGTON SCHOOLS
April 2006 WASL Data
0
0.5
1
1.5
2
2.5
3
3.5
4
0 10 20 30 40 50 60 70 80 90 100
Percent Poverty
Co
mb
ined
Rea
din
g /
Mat
h L
evel
In
dex
All Schools CEE Clients
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Today’s Outcomes
• Knowledge and skills: Models and strategies for using data to inform our practice
• Ideas and application: how do we get from here to there…
• Time to reflect and share
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
How Will We Get There?
• Models of Interpretation
• Application, investigation, and understanding the data use in the Schools of Distinction
• Taxonomy of Assessment Responsiveness
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
So Much Data, So Little Time…By itself, Data has no value. When data
is put into a form that is easily understandable, it becomes information.
When information is used to guide decisions that are in the best interest of the students and families we serve, it
becomes applied knowledge.Stan Beckelman, former President of Boeing Information Services
and Board Member for the Center for Educational Effectiveness
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
3 Models of Assessment Interpretation
• Status
• Improvement
• Growth
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
StudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
StudentLearning
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Status
• Where are we?– Typically viewed relative to a target or desired
state– Can be viewed system wide or with any unit of
analysis – down to student-by-student– AYP is a 1-year “Status” model
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Improvement (Change)
• Are we getting better?
• Requires historical data– 1 year does NOT make a trend– Common tools (tests, surveys, etc)
• Requires leadership to define “Better”
• “Safe Harbor” is a simple “Improvement” model
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Growth
• Most often applied to student achievement when viewing student-by-student achievement
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Measuring Growth and Improvement
• Systems for measuring and acting-on require:– Summative (evaluative) Assessments
• E.g. WASL
– Formative: Guiding (predictive) assessments• E.g. NWEA MAP assessment
– Diagnostic: indicate student-level strengths and challenges which can be used by staff to assist each student
• E.g. Pearson Benchmark
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Conclusive Validity
• Are the conclusions we draw from the data the right ones?
• Are the conversations and actions taken as a result of the conclusions– Supported by the data?– Constructive- leading to positive change?
• Due to the volume of data- Summary is a required technique
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Be Careful with Cause & Effect(Jumping to Conclusions)
• Establishing Cause (3 required conditions) :– The cause is related to the effect– No plausible alternative explanation for
the effect exists other than the cause– The cause precedes the effect
Robsinson, D. (et al.). The Incidence of “Causal” Statements in Teaching-and-Learning Research Journals. AERA Research Journal. Vol. 44 No. 2. June 2007
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Status, Improvement, and Growth Data Must Share Certain Attributes
• Educationally Significant• Aligned with Appropriate ‘Standards’• Longitudinal• Comparative / relevant• Community Sensitive / Culturally Responsive• Appropriate to target• Accurate / Valid
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
The Three Critical Questions
• Where are we?– Comparison points provide context
• Where do we want to be?– The movement from Good to Great
• Are we “improving” and “growing”? Are we on a path to get were we want to be?
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Application Across Domains of Data
• Viewing data across Status, Improvement, and Growth should not be limited to student outcome data
• But let’s start with a simple achievement example…
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Comparison PointWASL 10 Reading: Percentage of Students Meeting Standard
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2002 2003 2004 2005 2006 2007 2008 2009 2010
State Uniform BarCopyright © The Center for Educational Effectiveness, Inc., 2004
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Comparison PointsWASL 10 Reading: Percentage of Students Meeting Standard
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2002 2003 2004 2005 2006 2007 2008 2009 2010
State State Uniform BarCopyright © The Center for Educational Effectiveness, Inc., 2004
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Relevant SSD DataWASL 10 Reading: Percentage of Students Meeting Standard
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2002 2003 2004 2005 2006 2007 2008 2009 2010
Sunnyside SD State State Uniform BarCopyright © The Center for Educational Effectiveness, Inc., 2004
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Additional Comparison PointsWASL 10 Reading: Percentage of Students Meeting Standard
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2002 2003 2004 2005 2006 2007 2008 2009 2010
Sunnyside SD State State Uniform BarCopyright © The Center for Educational Effectiveness, Inc., 2004
68.7%
%= Results from 17 Latino-Majority Districts
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
What does your eye say about improvement?
WASL 10 Reading: Percentage of Students Meeting Standard
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2002 2003 2004 2005 2006 2007 2008 2009 2010
Sunnyside SD State State Uniform BarCopyright © The Center for Educational Effectiveness, Inc., 2004
68.7%
%= Results from 17 Latino-Majority Districts
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
WASL 10 Reading: Percentage of Students Meeting Standard
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2002 2003 2004 2005 2006 2007 2008 2009 2010
Sunnyside SD State State Uniform Bar Linear (State)Copyright © The Center for Educational Effectiveness, Inc., 2004
68.7%
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
WASL 10 Reading: Percentage of Students Meeting Standard
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2002 2003 2004 2005 2006 2007 2008 2009 2010
Sunnyside SD State State Uniform Bar
Linear (State) Linear (Sunnyside SD)Copyright © The Center for Educational Effectiveness, Inc., 2004
68.7%
Improvement?
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
3 Models of Assessment Interpretation
• Status
• Improvement
• Growth
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
StudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
StudentLearning
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Growth Models
• Models of accountability which measure progress by tracking the achievement of the same students from year to year with the intent of determining whether the students made progress (growth)
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Two Flavors of Growth Models
• Growth Models• Value-Add Models
Growth Modeling is NOT a “silver-bullet”– rather, another data analysis technique to use in improvement conversations.
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Interpreting Growth Rates
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Interpreting Growth
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Interpreting Growth
Highest (or fastest) growth “Rate”
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Source: Policymaker’s Guide to Growth Models for School Accountability: How Do Accountability Models Differ? Council of Chief School Officers (CCSSO) Web site.
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Source: Policymaker’s Guide to Growth Models for School Accountability: How Do Accountability Models Differ? Council of Chief School Officers (CCSSO) Web site.
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Which is Better?• Depends on the audience and purpose
– Parents: interested in year to year growth for their child
– Communities: interested in which school or district shows the greatest growth
• Policy-makers: tend to be interested in improvement based on input-factors (Value-add)
• District and Building Leadership: growth and value-add
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Myth of Mobility?
District7th
Total7th
match%
Match 10 Total10
match%
MatchTotal
Enrollment
Brewster 77 57 74.0% 69 55 79.7% 922
Bridgeport 43 31 72.1% 60 48 80.0% 727
Clover Park 785 374 47.6% 783 395 50.4% 11757
Grandview 269 214 79.6% 248 183 73.8% 3229
Granger 135 97 71.9% 94 70 74.5% 1462
Highline 1236 817 66.1% 1323 720 54.4% 17359
Lake Washington 1824 1404 77.0% 1830 1469 80.3% 23799
Longview 554 415 74.9% 546 437 80.0% 7405
Mabton 84 57 67.9% 88 57 64.8% 940
Marysville 884 612 69.2% 939 720 76.7% 11896
Mount Adams 85 66 77.6% 66 53 80.3% 1015
Othello 265 223 84.2% 212 167 78.8% 3278
Pasco 947 639 67.5% 848 494 58.3% 12660
Port Angeles 335 262 78.2% 421 323 76.7% 4519
Quincy 185 130 70.3% 163 122 74.8% 2419
Renton 1010 657 65.0% 1004 680 67.7% 13565
Richland 752 521 69.3% 888 639 72.0% 10135
Sunnyside 415 352 84.8% 360 302 83.9% 5742
Toppenish 232 186 80.2% 239 172 72.0% 3317
Tumwater 473 332 70.2% 539 404 75.0% 6081
Vancouver 1631 1074 65.8% 1768 1205 68.2% 22306
Wahluke 147 134 91.2% 135 110 81.5% 1917
Wapato 264 214 81.1% 215 171 79.5% 3373
Warden 75 61 81.3% 75 54 72.0% 970
Yakima 1032 821 79.6% 1144 669 58.5% 14247
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
MATH- 4th to 7th Grade Growth
y = 0.9974x + 0.06
y = 0.9718x + 0.1673
300
320
340
360
380
400
420
440
460
480
500
300 320 340 360 380 400 420 440 460 480 500
Math Gr-4
Ma
th G
r-7
Latino Majority:Current 8th Gr. ('03-'06) YSD:Current 8th Gr. ('03-'06)
Linear (YSD:Current 8th Gr. ('03-'06)) Linear (Latino Majority:Current 8th Gr. ('03-'06))
Met
Sta
ndar
d
Met Standard
Copyright © Center for Educational Effectiveness, 2004. All Rights Reserved
Met Std. BOTH Improved Declined
Did NOT meet Either
Total Students
398 115 254 1214 198120.1% 5.8% 12.8% 61.3%
Latino Majority:Current 8th Gr. ('03-'06)Met Std. BOTH Improved Declined
Did NOT meet Either
Total Students
136 89 62 381 66820.4% 13.3% 9.3% 57.0%
YSD:Current 8th Gr. ('03-'06)
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
MATH- 4th to 7th Grade Growth
y = 1.0089x + 0.026y = 0.9791x + 0.0729
300
320
340
360
380
400
420
440
460
480
500
300 320 340 360 380 400 420 440 460 480 500
Math Gr-4
Ma
th G
r-7
Lat. Majority- Struggling:Current 8th Gr. ('03-'06) YSD-Struggling:Current 8th Gr. ('03-'06)
Linear (YSD-Struggling:Current 8th Gr. ('03-'06)) Linear (Lat. Majority- Struggling:Current 8th Gr. ('03-'06))
Met
Sta
ndar
d
Met Standard
Copyright © Center for Educational Effectiveness, 2004. All Rights Reserved
Met Std. BOTH Improved Declined
Did NOT meet Either
Total Students
0 115 0 1214 13290.0% 8.7% 0.0% 91.3%
Lat. Majority- Struggling:Current 8th Gr. ('03-'06)Met Std. BOTH Improved Declined
Did NOT meet Either
Total Students
0 89 0 380 4690.0% 19.0% 0.0% 81.0%
YSD-Struggling:Current 8th Gr. ('03-'06)
What about students at levels 1 & 2?
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Predictive Assessment to Understand “expected” Growth
• Summative Predictive Diagnostic
• If we are to look at “expected” or “typical” growth we need:– Aligned assessments– Common for all students– Fit within overall assessment system
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Question 1: How Strong are the Correlations with MAP and WASL? (MVSD)
Grade READING MATH
3 0.782 0.817
4 0.826 0.879
5 0.795 0.876
6 0.802 0.892
7 0.78 0.872
8 0.725 0.905
• Correlations of .80 are generally accepted as “Strong” correlations
• I.e. “Strong Predictability”
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Interesting Findings
• Using “Highest” RIT – not necessarily the most recent – IMPROVES predictability in all cases (confirmed with several districts’ data what Highline published at Spring WERA Conference)
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Math: MAP as Predictor of WASL
260
280
300
320
340
360
380
400
420
440
460
480
500
520
540
160 170 180 190 200 210 220 230 240 250 260
M-Rit
WA
SL
Ma
th
Grade 5
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Math: MAP as Predictor of WASL
260
280
300
320
340
360
380
400
420
440
460
480
500
520
540
160 170 180 190 200 210 220 230 240 250 260
M-Rit
WA
SL
Ma
th
Grade 5
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
MVSD Reading
260
280
300
320
340
360
380
400
420
440
460
480
500
520
540
160 170 180 190 200 210 220 230 240 250 260
R-Rit
WA
SL
Re
ad
ing
Grade 6
How Can We Use This?
What Do We See Here in the PINK?
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
MVSD Reading
260
280
300
320
340
360
380
400
420
440
460
480
500
520
540
160 170 180 190 200 210 220 230 240 250 260
R-Rit
WA
SL
Re
ad
ing
Grade 6
How Can We Use This?
What Do We See Here in the Yellow?
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
MVSD Reading
260
280
300
320
340
360
380
400
420
440
460
480
500
520
540
160 170 180 190 200 210 220 230 240 250 260
R-Rit
WA
SL
Re
ad
ing
Grade 6
How Can We Use This?
What Do We See Here in the Green?
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
What, When, Where, and How?
• Diagnostic
• Formative
• Summative
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Most Data Can (or should be) Used in Multiple Ways
WASL 4 Math: Percentage of Students Meeting Standard
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2002 2003 2004 2005 2006 2007 2008 2009 2010
Pullman SD State State Uniform Bar
Copyright © The Center for Educational Effectiveness, Inc., 2004
WASL 5 Math: Percentage of Students Meeting Standard
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2006 2007 2008 2009 2010
Pullman SD State State Uniform Bar
Copyright © The Center for Educational Effectiveness, Inc., 2004
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Extending Beyond Achievement Data
• Using the assessment types, how might this apply beyond achievement?– Fiscal– Leadership– Organizational– Instructional
• Reflect on this and then pair and share
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
StudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
StudentLearning
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Schools of DistinctionWhat Makes Them Different?
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Schools of Distinction: Design Objectives
• Recognize improvement in performance over 5+ years.
• Meaningful – Used a Reading and Math Learning Index to determine improvement.
• Additional information for stakeholders—not a replacement for AYP determinations.
• Transparency and openness through the use of publicly available data.
• Must have at least “adequate performance” in both Math and Reading.
• Designed with the future in mind.
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Remembering the Award Winners
• 86 schools were identified as the “highest improving” schools– 51 elementary schools– 20 middle schools– 15 high schools
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
School Student Demographics
• Poverty ranges from 2.4% to 99%.– 31 have greater than 50% poverty
• Non-white enrollment ranges from 0% to 98%– 14 have a non-white majority multi-ethnic
student population
• ESL/ELL ranges from 0% to 50%– 17 have more than double the state average for
ESL students served (state average is 7%)
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
School and District Demographics
• 36 are from districts smaller than 3,000 students (23 < 1,000)
• 29 are from districts 3,000 to 15,000
• 21 are from districts greater than 15,000
• 36 east of the Cascades
• 50 west of the Cascades
• 48 are Title I, (38 are NOT)
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
ESD Representation101
105
112
113
114
121
123
171
189
101
105
112
113
114
121
123
171
189
• 101 = 6• 105 = 9• 112 = 8• 113 = 9• 114 = 5• 121 = 21• 123 = 10• 171 = 11• 189 = 7
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Elem RMLI 2001-02WA Grade 4: 2001-2 Reading and Math Learning Index
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 100.0
Percent Poverty
Rea
din
g &
Mat
h L
earn
ing
Ind
ex
represents the top 5% of the schools showing the greatest improvement over 7 years.
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Elem RMLI 2007WA Grade 4: 2007 Reading and Math Learning Index
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 100.0
Percent Poverty
Rea
din
g &
Mat
h L
earn
ing
Ind
ex
represents the top 5% of the schools showing the greatest improvement over 7 years.
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
What’s It Look Like in the Building?
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
WASL 4 Reading: Percentage of Students Meeting Standard
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2001 2002 2003 2004 2005 2006 2007 2008
Lincoln Elem Toppenish SD State State Uniform Bar
Copyright © The Center for Educational Effectiveness, Inc., 2004
Lower than the state 4 of 5 years
Above the state the last 2 years
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
WASL 4 Reading: Percent of Students by Level
33%15%
36%
61%
36% 44% 39%
-47% -55% -48%
-18%-29%
-5% -11%
-11%-14%
-10%
-2%
-6%
0% 0%9%
13%
7%
20%
20%
50%47%
2001 2002 2003 2004 2005 2006 2007 2008
20%
40%
60%
80%
MeetingStandard
100%
69% NOT meeting standard to86% MEETING standard
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Today’s Research Data Will Be:
Phase III– What can we learn about repeat
winners in ’08-’09?
Phase I Practices of Improving or
Turnaround Schools Oct 2007 – Jan
2008
Phase II EES-Staff Survey Characteristics of High Performing
Schools Dec 2007 – May
2008
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Phase II Who Responded?
• Number of schools = 31
• N = 811 School of Distinction employees
• 514 – Certificated teachers
• 276 – Other Staff
• 22 - Administrators
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Phase I Focus
"Nine Characteristics of High Performing
Schools" OSPI
Characteristics of Improved School Districts" OSPI,
Shannon & Bylsma, 2004
"Beat the Odds, 2006"
Morrison Institute for Public Policy,
"Knowing the Right Things
to Do", Elmore, 2004.
"School Turnarounds" Public Impact,
2007
What Works In Schools,
Marzano, 2003
Effective (Turnaround) Leadership
(Instructional Leadership)
student learning focus distributed leadership observes instruction
frequently
√ √ √ √ √ √
High Quality Teaching and Learning
High expectations use of student data
personalized interventions aligned C & I
√ √ √ √ √ √
Systems Support for Improvement
effective use of data policy support
parent involvement professional learning
community
I(2nd ed.) √ √ √ √ I
Collaboration
communication addressing conflict
organizational trust peer observation
√ √ √ √ √ √
Readiness for Improvement
belief in student learning openness to new ideas problem solving, conflict
management
√ I √ I I I
CEE Research Basis for Phase I & II - Schools of Distinction
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
"Nine Characteristics of High Performing
Schools" OSPI
Characteristics of Improved School Districts" OSPI,
Shannon & Bylsma, 2004
"Beat the Odds, 2006"
Morrison Institute for Public Policy,
"Knowing the Right Things
to Do", Elmore, 2004.
"School Turnarounds" Public Impact,
2007
What Works In Schools,
Marzano, 2003
Effective (Turnaround) Leadership
(Instructional Leadership)
student learning focus distributed leadership observes instruction
frequently
√ √ √ √ √ √
High Quality Teaching and Learning
High expectations use of student data
personalized interventions aligned C & I
√ √ √ √ √ √
Systems Support for Improvement
effective use of data policy support
parent involvement professional learning
community
I(2nd ed.) √ √ √ √ I
Collaboration
communication addressing conflict
organizational trust peer observation
√ √ √ √ √ √
Readiness for Improvement
belief in student learning openness to new ideas problem solving, conflict
management
√ I √ I I I
CEE Research Basis for Phase I & II - Schools of Distinction
Phase II Focus
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Let’s Get Into the Results
Comparison Set: CEE’s Educational Effectiveness Survey of the 9 Characteristics– WA Schools who Met AYP: 283 Schools
(actual responses: N=16,747 Staff)– WA School who Did NOT meet AYP: 194
Schools (actual responses: N=13,119 Staff)
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
What is The Leader’s Role?
Teachers and principals alike assess student and teacher achievement early and often – and use the information to drive improvement rather than assign blame.
The key, however, is not simply that the successful schools have data – it’s who is using the data and how they use the data.
Beat The Odds (2006)
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Experienced Teachers
Length of Service- In Education
0
50
100
150
200
250
300
350
400
< 1year
1 - 3years
4 - 5years
6 - 10years
11 - 15years
15+years
# of
Res
pond
ents
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
A Variety of Time In Building
Length of Service- This School
0
50
100
150
200
250
300
350
400
< 1year
1 - 3years
4 - 5years
6 - 10years
11 - 15years
15+years
# of
Res
pond
ents
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Does it Make a Difference?Comparison Perspective- 9 Characteristics of High Performing Schools
Percent Positive Responses
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%Readiness To Benefit
Clear & Shared Focus
High Standards & Expectations
Effective School Leadership
Collaboration, Communication &Community
Parent & Community Involvement
Supportive Learning Environment
Frequent Monitoring of Teaching &Learning
Focused Professional Development
Curriculum, Instruction, and Assessment
Did Not Meet AYP
Note: Further from the center implies more positive responses
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Does it Make a Difference?Comparison Perspective- 9 Characteristics of High Performing Schools
Percent Positive Responses
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%Readiness To Benefit
Clear & Shared Focus
High Standards & Expectations
Effective School Leadership
Collaboration, Communication &Community
Parent & Community Involvement
Supportive Learning Environment
Frequent Monitoring of Teaching &Learning
Focused Professional Development
Curriculum, Instruction, and Assessment
Met AYP Did Not Meet AYP
Note: Further from the center implies more positive responses
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Does it Make a Difference?Comparison Perspective- 9 Characteristics of High Performing Schools
Percent Positive Responses
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%Readiness To Benefit
Clear & Shared Focus
High Standards & Expectations
Effective School Leadership
Collaboration, Communication &Community
Parent & Community Involvement
Supportive Learning Environment
Frequent Monitoring of Teaching &Learning
Focused Professional Development
Curriculum, Instruction, and Assessment
Schools of Distinction Met AYP Did Not Meet AYP
Note: Further from the center implies more positive responses
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Schools of Distinction Research Review of Phase I - Research Findings• Very High Readiness for Improvement
– 75% belief that ALL students can meet state standards– 75% willingness to change, and openness to new ideas
• Culture of Collaboration– High trust across staff and with leadership– 75% willingness to address conflict
• Leadership– Stable – average of 4 yrs in building and 8 years as principal– Focus on instruction and student learning - 50% observe classrooms daily
• System Support for Improvement– 80% have release time monthly for professional development– 60% monitor school improvement plans at least monthly
• High Quality Instruction and Supportive Instructional Practice– 92% use assessment data to identify student needs and instructional
intervention– 84% use data to guide professional development– 80% use collaborative lesson design and analysis of student work
• High Level of Trust– 71% believe there is a high level of trust in their school
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Schools of Distinction Research Phase II – What Have We Learned?
Very High Readiness for Improvement75% belief that ALL students can meet state
standards75% willingness to change, and openness to new
ideasCulture of Collaboration
High trust across staff and with leadership75% willingness to address conflict
LeadershipStable – average of 4 yrs in building and 8 years as
principalFocus on instruction and student learning - 50%
observe classrooms dailySystem Support for Improvement
80% have release time monthly for professional development
60% monitor school improvement plans at least monthly
High Quality Instruction and Supportive Instructional Practice
92% use assessment data to identify student needs and instructional intervention
84% use data to guide professional development80% use collaborative lesson design and analysis of
student workHigh Level of Trust
71% believe there is a high level of trust in their school
Reading and Math Beliefs are more important – both in top 10!
Collaborative planning for integration of literacy and numeracy across the curric.
Leadership facilitate processes for improvement
Staff have frequent feedback about how they are doing
Teachers engage in PD learn and apply skills and strategies
Struggling students receive interventionCelebrating student successTeachers integrate literacy and numeracy
Strength in positive side of TrustLower “Trust Erosion” factors
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Readiness for Improvementor
”Culture Eats Strategy for Breakfast”
Values and culture stand out as one of the strongest and most consistent contrasts between beat-the-odds schools and the comparison schools.
Beat The Odds (2006)
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Application for Today
Given your personal theory of change: what in the following results
a) Supports where you are heading?
b) Causes you to adjust your thinking?
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
20%
13%
13%
42%
34%
33%
27%
29%
6%
10%
11%
3%
5%
5%
9%
11%
8%
20%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Schools of Distinction(N=811)
Met AYP (N=16,747)
Did Not Meet AYP(N=13,119)
Almost Always True Often True Sometimes True Seldom True Almost Never True Missing
Our staff believes that all students can meet state reading standards
Copyright © The Center for Educational Effectiveness, Inc., 2004
Beliefs- Reading
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Beliefs- Math
14%
10%
10%
40%
31%
29%
29%
31%
8%
11%
13%
4%
6%
6%
10%
13%
10%
24%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Schools of Distinction(N=811)
Met AYP (N=16,747)
Did Not Meet AYP(N=13,119)
Almost Always True Often True Sometimes True Seldom True Almost Never True Missing
Our staff believes that all students can meet state math standards
Copyright © The Center for Educational Effectiveness, Inc., 2004
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Welcoming New Ideas
28%
18%
14%
43%
42%
39%
31%
35%
3%
5%
7%
1%
3%
4%
2%
1%
1%
23%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Schools of Distinction(N=811)
Met AYP (N=16,747)
Did Not Meet AYP(N=13,119)
Almost Always True Often True Sometimes True Seldom True Almost Never True Missing
My colleagues welcome new and innovative ideas
Copyright © The Center for Educational Effectiveness, Inc., 2004
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
A Culture of Collaboration
Improved districts build a culture of commitment, collegiality, mutual respect, and stability.– Professional culture of high standards– Trust, mutual respect, and competence– Opportunities for peer support, collaboration, and develop
professional learning communitiesShannon & Bylsma (2004)
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Collaborative Problem Solving
31%
24%
20%
40%
36%
36%
26%
29%
4%
10%
11%
1%
3%
5%
4%
1%
0%
20%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Schools of Distinction(N=811)
Met AYP (N=16,747)
Did Not Meet AYP(N=13,119)
Almost Always True Often True Sometimes True Seldom True Almost Never True Missing
When there is a problem in my school, we talk about how to solve it
Copyright © The Center for Educational Effectiveness, Inc., 2004
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Addressing Conflict
32%
25%
20%
34%
35%
34%
26%
29%
7%
9%
11%
2%
3%
4%
2%
2%
3%
23%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Schools of Distinction(N=811)
Met AYP (N=16,747)
Did Not Meet AYP(N=13,119)
Almost Always True Often True Sometimes True Seldom True Almost Never True Missing
There is a willingness to address conflict in this school
Copyright © The Center for Educational Effectiveness, Inc., 2004
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Turnaround Instructional Leadership
Principals help schools succeed not when they are flashy superstars, but when they stay focused on the things that truly improve schools and keep pushing ahead, no matter what the roadblocks
Beat The Odds (2006)
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Stable Leadership Matters
• Principals average 5 years “At This School” and 6-10 years experience “As a Principal”
Principal- Years at This School
0
2
4
6
8
10
12
14
< 1year
1 - 3years
4 - 5years
6 - 10years
11 - 15years
15+years
# of
Res
pond
ents
Principal- Years as a Principal
0
2
4
6
8
10
12
14
< 1year
1 - 3years
4 - 5years
6 - 10years
11 - 15years
15+years
# of
Res
pond
ents
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Focused Leadership
Successful turnarounds are typically marked by vigorous analysis of data, identification of key problems, and selection of strategies to address the central challenges.
Two leader actions fall into this category:• Collecting and personally analyzing organization performance data• Making an action plan based on data
School Turnarounds (2007)
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Leadership
55%
36%
32%
25%
33%
33%
15%
17%
3%
8%
12%
2%
4%
3%
3%
4%
3%
12%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Schools of Distinction(N=811)
Met AYP (N=16,747)
Did Not Meet AYP(N=13,119)
Almost Always True Often True Sometimes True Seldom True Almost Never True Missing
My principal facilitates systems / processes to support school imporvement
Copyright © The Center for Educational Effectiveness, Inc., 2004
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Frequent Feedback
27%
18%
15%
39%
35%
33%
29%
32%
7%
12%
13%
2%
4%
5%
1%
2%
2%
24%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Schools of Distinction(N=811)
Met AYP (N=16,747)
Did Not Meet AYP(N=13,119)
Almost Always True Often True Sometimes True Seldom True Almost Never True Missing
We are frequently informed about how well we are doing
Copyright © The Center for Educational Effectiveness, Inc., 2004
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Focused Leadership (from Phase I)Leadership-Focus on Results
58% 26% 13% 2%2%1%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
The principal is goal-focused and results driven
Almost Always Often True Sometimes True Seldom True Almost Never True Missing
Focus on Data
48% 30% 15% 3%1%2%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
The principal analyzes datawith staff throughout the
school year
Almost Always Often True Sometimes True Seldom True Almost Never True Missing
Classroom Observation
22% 29% 30% 8% 8% 3%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
The principal observesclassroom instruction on a
daily basis
Almost Always Often True Sometimes True Seldom True Almost Never True Missing
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
System Support
Teachers and principals alike assess student and teacher achievement early and often – and use the information to drive improvement rather than assign blame.
The key, however, is not simply that the successful schools have data – it’s who is using the data and how they use the data.
Beat The Odds (2006)
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Systems Support (Phase I)
Regular Release Time
4% 26% 15% 34% 11% 10%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
The district provides regularrelease time for staff tofocus on professionaldevelopment and/or
collaborative planning
Daily Weekly Bi-Weekly Monthly Yearly N/A
Monitoring Improvement Plan
3% 5% 12% 41% 33% 6%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Our school meets tomonitor the implementationof our school improvement
plan
Daily Weekly Bi-Weekly Monthly Yearly N/A
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Measuring and Reporting
Successful turnarounds are typically marked by measuring and reporting data frequently and publically.
Multiple principals in successful turnarounds identified sharing data on a regular basis as a key means to identify practices that were working well, and alternatively, those that were not working.
School Turnarounds (2007)
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Data-driven Professional Development
32%
22%
18%
35%
34%
35%
23%
27%
4%
6%
9%
2%
2%
3%
11%
13%
8%
16%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Schools of Distinction(N=811)
Met AYP (N=16,747)
Did Not Meet AYP(N=13,119)
Almost Always True Often True Sometimes True Seldom True Almost Never True Missing
Appropriate data are used to guide building-directed professional development
Copyright © The Center for Educational Effectiveness, Inc., 2004
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
High Quality Teaching and LearningThe Instructional Work of the Organization
Reduce isolation and open practice up to direct observation, analysis, and feedback.– Make direct observation of practice, analysis, and
feedback a routine feature of work.Elmore (2000, 2002, and 2004)
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Data-driven Instruction
Beat-the-odds-schools are figuring out ways to customize instruction and intervention so it exactly suits each student’s needs.The beat-the-odds schools are putting in place a whole set of interlocking
practices and policies geared toward winning a marathon (instead of a sprint). It involves a vital cycle of instruction, assessment, and intervention, followed by more instruction, assessment and intervention.
Beat The Odds (2006)
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Personalized Instruction
36%
23%
16%
37%
32%
29%
25%
29%
2%
8%
13%
1%
3%
3%
5%
9%
9%
18%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Schools of Distinction(N=811)
Met AYP (N=16,747)
Did Not Meet AYP(N=13,119)
Almost Always True Often True Sometimes True Seldom True Almost Never True Missing
Struggling students receive early intervention and remediation to acquire skills
Copyright © The Center for Educational Effectiveness, Inc., 2004
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Data-driven Instruction
50%
32%
22%
29%
34%
35%
18%
24%
2%
5%
7%
0%
2%
3%
7%
9%
9%
11%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Schools of Distinction(N=811)
Met AYP (N=16,747)
Did Not Meet AYP(N=13,119)
Almost Always True Often True Sometimes True Seldom True Almost Never True Missing
Assessment data is used to identify student needs and appropriate instuctional intervention
Copyright © The Center for Educational Effectiveness, Inc., 2004
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Frequency of PracticeCollaborative Lesson Design
5% 30% 9% 24% 6% 24%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Teachers design lessonstogether on a regular basis
Daily Weekly Bi-Weekly Monthly Yearly N/A
Analysis of Student Work
7% 24% 16% 30% 11% 11%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Teacher teams meet toanalyze student work
Daily Weekly Bi-Weekly Monthly Yearly N/A
Assessments to Inform Instruction
30% 21% 13% 23% 6% 7%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
We use assessments toinform instruction
Daily Weekly Bi-Weekly Monthly Yearly N/A
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Collaboration for Instruction
22%
13%
10%
33%
28%
28%
27%
29%
9%
12%
15%
3%
4%
5%
11%
15%
12%
22%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Schools of Distinction(N=811)
Met AYP (N=16,747)
Did Not Meet AYP(N=13,119)
Almost Always True Often True Sometimes True Seldom True Almost Never True Missing
We collaboratively plan integration of literacy and numeracy concepts across the curriculum
Copyright © The Center for Educational Effectiveness, Inc., 2004
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Making it Happen
23%
16%
10%
36%
30%
27%
26%
31%
6%
9%
13%
2%
2%
4%
12%
17%
15%
21%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Schools of Distinction(N=811)
Met AYP (N=16,747)
Did Not Meet AYP(N=13,119)
Almost Always True Often True Sometimes True Seldom True Almost Never True Missing
All teachers integrate literacy and numeracy concepts into their teaching
Copyright © The Center for Educational Effectiveness, Inc., 2004
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Schools of Distinction Research Phase II – What Have We Learned?
Very High Readiness for Improvement• Belief that ALL students can meet state standards- Reading and
Math• 75% willingness to change, and openness to new ideas
Culture of Collaboration• High trust across staff and with leadership• Strong willingness to address conflict and openness to new ideas
Leadership• Stable – average of 4 yrs in building and 8 years as principal• Focus on instruction and student learning - 50% observe
classroom instruction daily with frequent feedback to staff• Facilitate systems and processes for improvement
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Schools of Distinction Research Phase II – What Have We Learned? (cont…)
System Support for Improvement• 80% have release time monthly for professional development• 60% monitor school improvement plans at least monthly• Teachers use collaborative time for instructional improvement
High Quality Instruction and Supportive Instructional Practice• 73% Struggling students receive intervention based on
assessment data• 80% use assessment data to identify student needs and
instructional intervention• 68% use data to guide professional development• 80% use collaborative lesson design and analysis of student
work
High Level of Trust• 71% believe there is a high level of trust in their school• Low trust erosion factors-- negativity and “I vs They” mindset
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
StudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
StudentLearning
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
3 Models of Assessment Interpretation
• Status
• Improvement
• Growth
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
StudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
StudentLearning
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
Taxonomy of Assessment Responsiveness
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
As leaders- How do you Build Toward Responsiveness?
• Compliance
• Awareness
• Sensitivity
• Competence
• Responsiveness
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
StudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
StudentLearning
C E N TE R F OR E D UC ATI ON A L EF F E CT I V E N E S S , I N C .
References You Can UsePrimary• Elmore, R. (2004). Knowing the Right Things to Do: School Improvement and Performance-Based
Accountability. Washington, D.C.: National Governors Association- Center for Best Practices.• Marzano, R. (2003). What Works in Schools: Translating Research Into Action. Alexandria, VA: ASCD.• Beat The Odds (2006). Morrison Institute for Public Policy (2006). Why Some Schools With Latino Children
Beat the Odds…and Others Don’t. Tempe, AZ.: Morrison Institute for Public Policy, Arizona State University, jointly with Center for the Future of Arizona. (aka: “Beat The Odds (2006) ).
• School Turnarounds (2007). Public Impact (2007). School Turnarounds: A review of the cross-sector evidence on dramatic organizational improvement. Public Impact, Academic Development Institute- prepared for the Center on Innovation and Improvement. Retrieved from: http://www.centerii.org/ (aka: School Turnarounds (2007)).
• Shannon, G.S. & Bylsma, P. (2004). Characteristics of Improved School Districts: Themes from Research. Olympia, WA. Office of Superintendent of Public Instruction.
• Shannon, G.S. & Bylsma, P. (2003). Nine Characteristics of High Performing Schools. A research-based resource for school leadership teams to assist with the School Improvement Process. Office of Superintendent of Public Instruction. Olympia, WA.
Secondary• Center for Educational Effectiveness (CEE) (2005). Longitudinal Change in Staff Perceptions of the 9
Characteristics of High Performing Schools in OSPI SIA Cohort-II and III Schools. Redmond, WA: Center for Educational Effectiveness.
• Elmore, R. (2000). Building a New Structure For School Leadership. Washington, D.C.: The Albert Shanker Institute.
• Elmore, R. (2002). Bridging the Gap Between Standards and Achievement. Washington, D.C.: The Albert Shanker Institute.
• Quinn, R. and Rohrbaugh, J. (1983). A spatial model of effectiveness criteria: Toward a competing values approach to organizational analysis. Management Science, 29(3), 363-377.
• Tschannen-Moran, (2004). Trust Matters, Leadership for Successful Schools. San Francisco, CA. Jossey-Bass.