critical markers of high quality child outcomes data eco advisory board march, 2012
TRANSCRIPT
Critical Markers of High Quality Child Outcomes Data
ECO Advisory BoardMarch, 2012
Topics
• Why identify critical markers of high quality data?
• Anticipated use• 3 areas of focus• Draft critical markers• Where these might be
discussed in the APR• Reactions?
2Early Childhood Outcomes Center
Looking for…
Proposed markers that suggest a state’s EC outcomes systems may produce high quality data
Use markers to:
- systematically track over time - track internally (within states)
- to produce a national picture
3Early Childhood Outcomes Center
Anticipated Use
• Tracking state progress (based on APR info)• Compare data from each state to a series of
critical markers for summaries– X % of states met standard on each marker nationally– X% of states met standard on 7 out of 10 markers
• NOT a state by state report card• Share detailed info with each state, upon
request
4Early Childhood Outcomes Center
3 Areas
• Completeness of data
5Early Childhood Outcomes Center
Completeness of data
• Missing data concepts– Are whole forms missing? (Have data
from all kids expect to have data from)
– Are the forms that you have complete?
6Early Childhood Outcomes Center
3 Areas
• Completeness of data
• Accuracy of data
7Early Childhood Outcomes Center
Accuracy of Data
• Do the data reflect the “true” performance of children in the program– Unknowable, so….
• Are the patterns in data what you would expect to see if the data were accurate?
• Do they make sense? Or, are there red flags that raise more questions?
8Early Childhood Outcomes Center
3 Areas
• Completeness of data
• Accuracy of data
• State efforts related to tracking quality of data
9Early Childhood Outcomes Center
Criteria for Selection of Markers
• Important – necessary for or indicative of
high quality data• Accessible to ECO
– Information to determine presence or absence of marker is available
10Early Childhood Outcomes Center
Draft Markers: Completeness
• State calculates and publicly reports number of children missing outcomes data
• Percent of missing outcomes data is less than 5%. – Is this reasonable?
• Percent of missing data by proxy calculation is less than. – 40% of exiters for Part C– 20% of child count for 619
11Early Childhood Outcomes Center
Part C percent of exiters 2009-10
12Early Childhood Outcomes Center
Draft Markers: Accuracy/Patterning
• State % in a is not overly high (GT 5%)• State % in b is not overly low (LT 5%) or high (GT 50%)• State % in c is not overly low (LT 5%) or high (GT 50%)• State % in d is not overly low (LT 5%) or high (GT 50%)• State % in e is not overly low (LT 5%) or high (GT 65%)
13Early Childhood OutcGTomes Center
Knowledge and SkillsPart B 619 proportion of children that make no
progress (progress category a)
14Early Childhood Outcomes Center
Knowledge and SkillsPart B 619 proportion of children who improved functioning but not sufficiently to move nearer to functioning comparable to same age
peers(progress category b)
15Early Childhood Outcomes Center
Knowledge and SkillsPart B 619 proportion of children who improved functioning to a
level nearer to same age peers but did not reach it(progress category c)
16Early Childhood Outcomes Center
Knowledge and SkillsPart B 619 proportion of children who improved functioning to
reach a level comparable to same aged peers(progress category d)
17Early Childhood Outcomes Center
Knowledge and SkillsPart B 619 proportion of children who maintained
functioning at a level comparable to same aged peers(progress category e)
18Early Childhood Outcomes Center
Percent of states meeting the Critical Markers for 2009 - 10
19Early Childhood Outcomes Center
Accuracy of data Completeness of data
Both
Part C 66 59 47
Part B 619 63 66 41
Draft Markers: State Quality Review of Data
• State conducts data quality checks• State’s own analyses provide evidence of
high quality data
20Early Childhood Outcomes Center
21Early Childhood Outcomes Center
Completeness/ Missing Data
Accuracy/ Patterns
Location in Suggested APR Templatehttp://www.fpg.unc.edu/~eco/pages/fed_req.cfm#ECOSuggestedFormats
22Early Childhood Outcomes Center
State Quality Review of Data
Location in Suggested APR Templatehttp://www.fpg.unc.edu/~eco/pages/fed_req.cfm#ECOSuggestedFormats
Questions for you
• Are these markers important?– Related to high quality data– Things you value and might track or are already tracking
them?– Would tracking them help you improve your system?
• Is there anything else you can think of as a good marker of high quality data
23Early Childhood Outcomes Center
Summary – Draft Critical Markers
• State calculates and publicly reports number of children missing outcomes data
• Percent of missing outcomes data is less than 5%• Percent of missing data by proxy calculation is less than X • State % in a is not overly high (GT 5%)• State % in b is not overly low (LT 5%) or high (GT 50%)• State % in c is not overly low (LT 5%) or high (GT 50%)• State % in d is not overly low (LT 5%) or high (GT 50%)• State % in e is not overly low (LT 5%) or high (GT 65%) • State conducts data quality checks• State’s own analyses provide evidence of high quality data
24
Comments, Reactions, Questions?
25