cornelia taylor, eco at sri kathy hebbeler, eco at sri
DESCRIPTION
National Picture –Child Outcomes for Early Intervention and Preschool Special Education. Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI. October, 2013. On Today’s Call. Brief review of the national data Data quality Completeness of data State-to-state variation - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/1.jpg)
Cornelia Taylor, ECO at SRIKathy Hebbeler, ECO at SRI
National Picture –Child Outcomes for Early Intervention and Preschool Special Education
October, 2013
![Page 2: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/2.jpg)
On Today’s Call
• Brief review of the national data• Data quality
– Completeness of data– State-to-state variation– Pattern checking with other variables– Change over time
2
![Page 3: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/3.jpg)
3
Approach Part C
(N=56)
Preschool
(N=59)
COS 7 pt. scale 42/56 (75%) 37/59 (63%)
One tool statewide
8/56 (14%) 9/59 (15%)
Publishers’ online analysis
1/56 (2%) 6/59 (10%)
Other 5/56 (9%) 7/59 (12%)
State Approaches to Measuring Child Outcomes – 2011-12
![Page 4: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/4.jpg)
3 Methods Methods for Calculating National Estimates
1. All states averaged (each state weighted as 1)
2. All states weighted by child count
3. States with the highest quality data weighted by child count to represent all states*
4
*The data we will be presenting for the national picture.
![Page 5: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/5.jpg)
Identifying States with Quality Data
Criteria for high quality data:• Reporting data on enough children
– Part C – 28% or more of exiters– Preschool – 12% or more of child count
• Within expected patterns in the data– category ‘a’ not greater than 10% – category ‘e’ not greater than 65%
5
![Page 6: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/6.jpg)
Number of States that Met Criteria for Inclusion in the National Analysis
6
08-09 09-10 10-11 11-12
Part C 19 29 39 33
Preschool 15 33 36 39
![Page 7: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/7.jpg)
Part C – Reason States were Excluded from Analyses
7
Reason Part C state was excluded 2010-11 2011-12State is sampling 3 2No outcomes data reported 0 0Reported outcomes data on less than 28% of reported exiters
3 6
Had at least one outcome with category a greater than 10% or category e greater than 65%
4 5
Reported outcomes data on less than 28% of reported exiters
AND
Had at least one outcome with category a greater than 10% or category e greater than 65%
1 4
Questionable data quality based on review of SPP/APR and knowledge gained through TA
1 0
States included in the analysis 39 33
![Page 8: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/8.jpg)
Part B Preschool – Reason States were Excluded from Analyses
8
Reason Part B state was excluded 2010-11 2011-12State is sampling 4 2No progress category data reported 1 2
No outcomes data reported 1 0Reported outcomes data on less than 12% of child count 2 4
Had at least one outcome with category a greater than 10% or category e greater than 65%
4 3
Reported outcomes data on less than 12% of child countANDHad at least one outcome with category a greater than 10% or category e greater than 65%
0 0
Questionable data quality based on review of SPP/APR and knowledge gained through TA
2 0
No child count data available 1 0States included in the analysis 36 39
![Page 9: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/9.jpg)
9
social relationships knowledge and skills action to meet needs0
20
40
60
80
100
70
77 7671
74 76
6873 73
66
72 73
Part C: Greater than Expected GrowthTrends Across 4 Years
2008-09 (19 states) 2009-10 (29 states) 2010-11 (39 states) 2011-12 (33 states)
Av
era
ge
Pe
rce
nta
ge
Ac
ros
s S
tate
s
![Page 10: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/10.jpg)
10
social relationships knowledge and skills action to meet needs0
20
40
60
80
100
61
54
6162
54
6061
555960
52
59
Part C: Exited Within Age ExpectationsTrends Across 4 Years
2008-09 (19 states) 2009-10 (29 states) 2010-11 (39 states) 2011-12 (33 states)
Av
era
ge
Pe
rce
nta
ge
Ac
ros
s S
tate
s
![Page 11: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/11.jpg)
11
social relationships knowledge and skills action to meet needs0
20
40
60
80
100
83 83 8283 82 8281 81 8181 81 80
Part B Preschool: Greater Than Expected GrowthTrends Across 4 Years
2008-09 (15 states) 2009-10 (33 states) 2010-11 (36 states) 2011-12 (39 states)
Av
era
ge
Pe
rce
nta
ge
Ac
ros
s S
tate
s
![Page 12: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/12.jpg)
12
social relationships knowledge and skills action to meet needs0
20
40
60
80
100
59
51
67
59
52
67
60
53
66
5953
66
Part B Preschool: Exited Within Age Expecta-tions
Trends Across 4 Years
2008-09 (15 states) 2009-10 (33 states) 2010-11 (36 states) 2011-12 (39 states)
Av
era
ge
Pe
rce
nta
ge
Ac
ros
s S
tate
s
![Page 13: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/13.jpg)
Good News!
• Consistent data over time• Increasing number of children in the child
outcomes data• Increasing number of states in the ‘quality’
data for child outcomes
13
![Page 14: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/14.jpg)
Current Emphasis
• Data Quality– Increasing the number of children/families in the data– Pattern checking to identify data quality issues– Training, guidance, supervision, etc.
14
![Page 15: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/15.jpg)
Part C: Percent of States by completeness of child outcomes data*
2009-10 (N=49) 2010-11 (N-51) 2011-12 (N=51)0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
32.7
19.623.5
55.1
64.7 62.7
12.215.7 13.7
< 35%35% - <70%70% or more
* Completeness = (total with outcomes data/total exiters)
![Page 16: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/16.jpg)
Part B 619 :Percent of States by completeness of child
outcomes data*
09-10 (N=49) 10-11 (N=51) 11-12 (N=49)
20.4
11.814.3
53.149.0
44.9
26.5
39.2 40.8
<12%12 - <33%>33%
* Completeness = (total with outcomes data/child count)
![Page 17: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/17.jpg)
State Level Variation
• Lots of variation across states in summary statement values
• Variation is not a direct result of percent served
17
![Page 18: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/18.jpg)
Part C: Positive Social Emotional Skills Progress Category ‘b’
18
![Page 19: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/19.jpg)
Part B Preschool: Positive Social Emotional Skills Progress Category ‘b’
19
![Page 20: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/20.jpg)
Part C – Knowledge and Skills, State Percentages for Increased Rate of
Growth, 2011-12
![Page 21: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/21.jpg)
Part B Preschool – Meets Needs, State Percentages for Exited within Age
Expectations, 2011-12
![Page 22: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/22.jpg)
Pattern Checking
22
![Page 23: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/23.jpg)
Social Relationships Knowledge and Skills Action to Meet Needs0
10
20
30
40
50
60
70
80
90
100
55
45
52
61
52
61
66
58
66
Part C: Exited Within Age Expectations by State Percentages of Exiters Not Eligible for Part B
<20% (n=22)20-30% (n=21)30 or greater % (n=13)
Aver
age
Perc
enta
ge A
cros
s Sta
tes
23
![Page 24: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/24.jpg)
Part C, Average Percentage Who Exited Within Age Expectations by State Percent Served, 2011-12
24
Social Relationships Knowledge and Skills Action to Meet Needs0%
10%
20%
30%
40%
50%
60%
70%
62%
55%
61%
55%
44%
51%
56%
47%50%
63%
53%
66%
63%
55%
61%
<2% (N=8)2% to <2.5% (N=14)2.5% to <3% (N=8)3% to <3.9% (N=13)>3.9% (N=9)
![Page 25: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/25.jpg)
Social Relationships Knowledge and Skills Action to Meet Needs0
10
20
30
40
50
60
70
80
90
100
63
52
70
5652
6564
56
65
5248
5860
54
67
Part B Preschool: Average Percentage Who Exited Within Age Expectations by State Percent Served
<5.2% (n=9)5.2-5.7% (n=9)5.7-6.5% (n=8)6.5-7.5% (n=14)>7.5% (n=12)
Aver
age
Perc
enta
ge A
cros
s Sta
tes
25
![Page 26: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/26.jpg)
Part B 619 longitudinal patterns all statesOutcome 1 Summary Statement 1
![Page 27: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/27.jpg)
Part C longitudinal patterns all states
Outcome 1 Summary Statement 1
27
![Page 28: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/28.jpg)
Looking only at states that met the quality criteria for inclusion in the national analysis based on….
– Missing data criteria– Patterning criteria– APR/SPP reviews
28
![Page 29: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/29.jpg)
Part B 619 longitudinal patterns best states from last 3 years
Outcome 1 Summary Statement 1 (n=28)
![Page 30: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/30.jpg)
Part B 619 longitudinal patterns best states all 4 years
Outcome 1 Summary Statement 1 (12 states)
![Page 31: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/31.jpg)
Discussion
• How should we interpret differences between state values?
• What pieces of information already reported would predict summary statement values?
31
![Page 32: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/32.jpg)
Interpreting Change over time
32
![Page 33: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/33.jpg)
What types of change are important
• Small variations from year to year are expected• Large consistent increases are good news
particularly when linked to programmatic changes
• Large consistent decreases require explanation (e.g. changing population)
• Large up and down changes are an indicator of questionable data quality and require explanation
33
![Page 34: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/34.jpg)
Part C: Change between 2010-11 and 2011-12
34
Statistically Significant Change OC1-SS1 OC2-SS1 OC3-SS1 OC1-SS2 OC2-SS2 OC3-SS2
Negative 9 9 6 11 11 7
None 32 31 36 30 34 33
Positive 10 11 9 10 6 11Average Positive change 7.8 8.6 7.3 6.9 7.8 7.8Min and Max 2 – 15 2 – 15 3 – 13 2 – 18 3 – 15 3 – 22
![Page 35: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/35.jpg)
Part B Preschool: Change between 2010-11 and 2011-12
35
Statistically Significant Change OC1-SS1 OC2-SS1 OC3-SS1 OC1-SS2 OC2-SS2 OC3-SS2
Negative 8 6 7 13 10 10
None 30 32 27 30 22 33
Positive 11 11 15 6 17 6Average Positive change 5.4 4.5 8.3 4 5.6 5.2Min and Max 2 - 14 1 – 7 1 - 6 1 – 9 1 - 26 2 - 8
![Page 36: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/36.jpg)
Conclusions
• The data continue to be used by the federal government to justify funding
• Results Driven Accountability is shining a spotlight on each state’s child outcomes data.
• State’s can expect more scrutiny around data quality
36
![Page 37: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/37.jpg)
How we can help!
• Email Cornelia for a state profile of your data quality
[email protected]• Contact us for help with data quality analysis
and quality assurance activities• Contact us for help with program improvement
planning and data analysis.
37
![Page 38: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/38.jpg)
38
http://www.ectacenter.org/~pdfs/eco/OutcomesforChildren-FFY2011.pdf
http://www.ectacenter.org/~pdfs/eco/Family_fLYER_2013.pdf
![Page 39: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/39.jpg)
National Graphing Template
39
![Page 40: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/40.jpg)
Other resources
• National Graphing templates– http://www.ectacenter.org/eco/pages/summary.asp#su
mmarygraph
• Data quality TA resources– http://www.ectacenter.org/eco/pages/quality_assuranc
e.asp
• Data analysis for program improvement TA resources– http://www.ectacenter.org/eco/pages/usingdata.asp
40
![Page 41: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/41.jpg)
Upcoming family data webinar
• Stay tuned for an upcoming presentation of Family Data: Indicator C4 Highlights
41
![Page 42: Cornelia Taylor, ECO at SRI Kathy Hebbeler, ECO at SRI](https://reader030.vdocuments.net/reader030/viewer/2022032612/56813022550346895d95a4b2/html5/thumbnails/42.jpg)
42
Find more resources at:
http://www.the-eco-center.org