cesa 6 improving the quality of child outcomes data

CESA 6IMPROVING THE QUALITY

OFCHILD OUTCOMES DATA

Ruth Chvojicek – Statewide Part B Indicator 7

Child Outcomes Coordinator

OBJECTIVES

• To discuss the significance of and strategies for improving child outcomes data quality

• To look at state, CESA and District data patterns as one mechanism for checking data quality

2

QUALITY ASSURANCE: LOOKING FOR QUALITY DATA

I know it is in here somewhere

ONGOING CHECKS FOR DATA QUALITY

Before Good training and assessmentEfficient data systemsTimely and accurate data entry

During

Ongoing supervision of implementationFeedback to implementersRefresher training

AfterReview sample of completed COSFsPattern checking analysis

4

PROMOTING QUALITY DATA - • Through data systems and verification, such as

• Monthly data system error checks e.g. missing & inaccurate data

• Monthly email data reminders• Indicator Training data reports• Good data entry procedures

5

LOOKING AT DATA

6

USING DATA FOR PROGRAM IMPROVEMENT = EIA

EvidenceInferenceAction

7

EVIDENCE Evidence refers to

the numbers, such as“45% of

children in category b”

The numbers are not debatable

8

INFERENCE How do you interpret the #s? What can you conclude from the #s? Does evidence mean good news? Bad news? News

we can’t interpret? To reach an inference, sometimes we analyze data in

other ways (ask for more evidence) “Drill Down”

9

INFERENCE Inference is debatable -- even

reasonable people can reach different conclusions

Stakeholders (district personal) can help with putting meaning on the numbers

Early on, the inference may be more a question of the quality of the data

10

ACTION• Given the inference from the numbers,

what should be done?• Recommendations or action steps• Action can be debatable – and often is• Another role for stakeholders• Again, early on the action might have

to do with improving the quality of the data

11

PROMOTING QUALITY DATATHROUGH DATA ANALYSIS

12

THE THREE OUTCOMES

Positive Social Emotional Skills

Acquisition & Use of Knowledge & Skills

Takes Appropriate Action to Meet Needs

Percent of preschool children with IEPs who demonstrate improved:

7-POINT RATING SCALEPLEASE REFER TO HANDOUT – “THE BUCKET LIST”

1• Child does not yet

show functioning expected of a child his or her age in any situation.

• Child’s functioning does not yet include immediate foundational skills upon which to build age-appropriate functioning.

• Child functioning reflects skills that developmentally come before immediate foundational skills.

• Child’s functioning might be described as like that of a much younger child.

2• Child occasionally

uses immediate foundational skills across settings and situation. More functioning reflects skills that are not immediate foundational than are immediate foundational.

3• Child does not yet

show functioning expected for a child of his or her age in any situation.

• Child uses immediate foundational skills, most or all of the time, skills upon which to build age-appropriate functioning.

• Functioning might be described as like that of a younger child.

4• Child shows

occasional age-appropriate functioning across settings and situations. More functioning is not age-appropriate than age-appropriate.

5• Child shows

functioning expected for his or her age some of the time and/or in some settings and situations. Child’s functioning is a mix of age-appropriate and not age-appropriate behaviors and skills.

• Childs functioning might be described as like that of a slightly younger child.

6• Child’s functioning

generally is considered appropriate for his or her age but there are some significant concerns about the child’s functioning in this outcome area. These concerns are substantial enough to suggest monitoring or possible additional support.

• Although age-appropriate, the child’s functioning may border on not keeping pace with age expectations.

7• Child shows

functioning expected for his or her age in all or almost all everyday situations that are part of the child’s life. Functioning is considered appropriate for his or her age.

• No one has any concerns about the child’s functioning in this outcome area.

PATTERN CHECKING - CHECKING TO SEE IF RATINGS ACCURATELY REFLECT CHILD STATUS• We have expectations about how child outcomes data

should look• Compared to what we expect• Compared to other data in the state• Compared to similar states/regions/school districts

• When the data are different than expected ask follow up questions

15

16

QUESTIONS TO ASK Do the data make sense?

Am I surprised? Do I believe the data? Believe some of the data? All of the data?

If the data are reasonable (or when they become reasonable), what might they tell us?

PATTERNS WE WILL BE CHECKING TODAY

1. Entry Rating Distribution

2. Entry Rating Distribution by Eligibility Determination

3. Comparison of Entry Ratings Across Outcomes

4. Entry/Exit Comparison by CESA

5. State Entry Rating Distribution by Race/Ethnicity

6. State Exit Rating Distribution

7. Progress Categories by State/CESA

8. Summary Statements by State/CESA

17

SMALL GROUP DISCUSSION QUESTIONS:

18

1. What do you notice about your local data?

2. What stands out as a possible ‘red flag’?

3. What might you infer about the data?

4. What additional questions does it raise?

5. What next steps might you take?

PREDICTED PATTERN #1

Children will differ from one another in their entry scores in reasonable ways (e.g., fewer scores at the high and low ends of the distribution, more scores in the middle).

Rationale:

Evidence suggests EI and ECSE serve more mildly than severely impaired children (e.g., few ratings/scores at lowest end). Few children receiving services would be expected to be considered as functioning typically (few ratings/scores in the typical range).

19

1 2 3 4 5 6 70.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

State 11-12 Entry Rating Distribution

21

PREDICTED PATTERN #2

Groups of children with more severe disabilities should have lower entry numbers than groups of children with less severe disabilities.

STATE 11-12 ENTRY RATING ELIGIBILITY PERCENTAGES

Autism 3.7%Cognitive Disability 0.9%Deaf-Blind 0.0%Emotional Behavioral Disability 0.9%Hearing Impairment 0.8%Orthopedic Impairment 0.9%Other Health Impairment 3.4%Significant Developmental Delay 21.2%Specific Learning Disability 0.2%Speech or Language Impairment 67.6%Traumatic Brain Injury 0.1%Visual Impairment 0.3%

Outcome 1 1 2 3 4 5 6 7Autism 20.4% 38.9% 14.3% 17.9% 7.1% 1.4% 0.0%Cognitive Disability 24.6% 33.3% 24.6% 10.1% 5.8% 1.4% 0.0%Deaf-Blind 0.0% 0.0% 50.0% 50.0% 0.0% 0.0% 0.0%Emotional Behavioral Disability 4.5% 21.2% 12.1% 39.4% 18.2% 4.5% 0.0%Hearing Impairment 3.4% 8.5% 5.1% 5.1% 40.7% 18.6% 18.6%Orthopedic Impairment 13.8% 10.8% 6.2% 6.2% 24.6% 23.1% 15.4%Other Health Impairment 17.3% 21.7% 14.2% 23.6% 15.4% 6.3% 1.6%Significant Developmental Delay 7.3% 24.5% 18.3% 24.7% 17.5% 5.6% 2.0%Specific Learning Disability 6.3% 0.0% 12.5% 12.5% 25.0% 31.3% 12.5%Speech or Language Impairment 0.4% 4.0% 5.8% 9.6% 23.8% 27.9% 28.6%Traumatic Brain Injury 36.4% 36.4% 9.1% 9.1% 9.1% 0.0% 0.0%Visual Impairment 0.0% 19.0% 14.3% 19.0% 4.8% 33.3% 9.5%

Outcome 2 1 2 3 4 5 6 7Autism 16.1% 32.5% 13.9% 20.7% 9.6% 6.4% 0.7%Cognitive Disability 37.7% 42.0% 14.5% 2.9% 2.9% 0.0% 0.0%Deaf-Blind 0.0% 0.0% 50.0% 50.0% 0.0% 0.0% 0.0%Emotional Behavioral Disability 0.0% 4.5% 4.5% 12.1% 19.7% 39.4% 19.7%Hearing Impairment 5.1% 13.6% 16.9% 18.6% 27.1% 13.6% 5.1%Orthopedic Impairment 16.9% 12.3% 9.2% 6.2% 21.5% 23.1% 10.8%Other Health Impairment 19.7% 20.5% 17.3% 10.6% 18.1% 8.3% 5.5%Significant Developmental Delay 8.6% 26.7% 23.6% 21.2% 14.2% 4.6% 1.1%Specific Learning Disability 6.3% 6.3% 25.0% 31.3% 12.5% 12.5% 6.3%Speech or Language Impairment 1.4% 8.7% 14.4% 20.2% 34.1% 17.6% 3.5%Traumatic Brain Injury 45.5% 36.4% 0.0% 9.1% 9.1% 0.0% 0.0%Visual Impairment 14.3% 23.8% 9.5% 0.0% 28.6% 23.8% 0.0%

Outcome 3 1 2 3 4 5 6 7Autism 11.8% 29.3% 18.9% 19.3% 12.5% 5.4% 2.9%Cognitive Disability 27.5% 49.3% 13.0% 4.3% 4.3% 1.4% 0.0%Deaf-Blind 0.0% 0.0% 0.0% 50.0% 50.0% 0.0% 0.0%Emotional Behavioral Disability 1.5% 4.5% 7.6% 18.2% 37.9% 15.2% 15.2%Hearing Impairment 5.1% 6.8% 6.8% 10.2% 16.9% 25.4% 28.8%Orthopedic Impairment 21.5% 16.9% 20.0% 16.9% 16.9% 7.7% 0.0%Other Health Impairment 20.5% 15.4% 17.7% 16.5% 18.5% 9.1% 2.4%Significant Developmental Delay 6.1% 16.6% 17.4% 23.2% 21.4% 12.0% 3.3%Specific Learning Disability 6.3% 6.3% 6.3% 0.0% 25.0% 37.5% 18.8%Speech or Language Impairment 0.5% 2.4% 4.4% 7.1% 16.1% 26.0% 43.5%Traumatic Brain Injury 45.5% 9.1% 18.2% 27.3% 0.0% 0.0% 0.0%Visual Impairment 4.8% 14.3% 14.3% 14.3% 19.0% 33.3% 0.0%

29

PREDICTED PATTERN #3

Functioning at entry in one outcome is related to functioning at entry in the other outcomes.

For cross tabulations we should expect most cases to be in the diagonal and the other to be clustered on either side of the diagonal.

1 2 3 4 5 6 71 13 62 4 47 17 6 2 3 13 20 33 13 2 2 14 9 21 51 25 145 1 9 27 52 70 13 16 5 13 22 91 48 27 1 7 18 84 123 10

Outco

me 1

Outcome 2

1 2 3 4 5 6 71 12 4 1 12 4 31 26 15 9 9 33 10 34 20 31 15 84 5 42 49 41 255 3 5 61 84 1216 1 2 22 43 1357 3 12

Outcome 3Out

come 2

1 2 3 4 5 6 71 13 4 1 12 2 30 21 14 9 3 13 1 6 29 19 10 5 14 3 10 32 48 22 55 1 6 13 77 57 196 1 1 5 22 71 817 2 1 6 37 197

Out

come 1

Outcome 3

PREDICTED PATTERN #4

Large changes in status relative to same age peers between entry and exit from the program are possible but rare.

When looking at the Entry/Exit Rating comparison for individual children we would expect very few children to increase more than 3 points.

33

Outcome 1 Exit

Outcome 1 Entry

1 2 3 4 5 6 7

1 1 2 1 1 1 1

2 4 3 6 6 2 3

3 1 1 1 4 6 2 3

4 4 6 5

5 1 2 4 11 20

6 1 4 18

7 2 19

Outcome 2 Exit

Outcome 2 Entry

1 2 3 4 5 6 7

1 2 1 1 1

2 1 3 6 4 12 2 2

3 1 4 5 9 10

4 1 6 10 4

5 10 10 15

6 1 9 15

7 1

Outcome 3 Exit

Outcome 3 Entry

1 2 3 4 5 6 7

1 3 3 1 1 1 1

2 3 2 4 1 2

3 1 2 3 5 6

4 2 3 9 6

5 1 5 7 13

6 1 1 2 21

7 4 32

PREDICTED PATTERN #5

If children across race/ethnicity categories are expected to achieve similar outcomes, there should be no difference in distributions across race/ethnicity.

Note:

Wisconsin began gathering race/ethnicity data for Indicator 7 on July 1, 2011. This impacts the data on the graphs being reviewed today.

37

Outcome 1 1 2 3 4 5 6 7Asian 4.3% 18.4% 9.9% 9.2% 23.4% 15.6% 19.1%Black 6.4% 15.3% 11.3% 16.7% 20.1% 15.2% 15.2%Hispanic 4.9% 11.9% 12.1% 13.5% 21.5% 21.7% 14.5%American Indian Alaskan 2.7% 10.0% 15.5% 19.1% 24.5% 17.3% 10.9%Hawaiian Other Pacific Islander 0.0% 31.6% 21.1% 10.5% 31.6% 0.0% 5.3%Two or More Races 4.5% 7.2% 11.7% 18.9% 26.1% 18.9% 12.6%White 2.8% 9.5% 8.1% 13.2% 21.4% 22.2% 22.8%

Outcome 2 1 2 3 4 5 6 7Asian 5.7% 14.9% 24.1% 20.6% 22.0% 12.8% 0.0%Black 7.1% 20.0% 23.6% 19.7% 20.4% 7.6% 1.6%Hispanic 7.6% 19.3% 22.8% 20.7% 19.8% 8.0% 1.7%American Indian Alaskan 2.7% 21.8% 17.3% 27.3% 21.8% 9.1% 0.0%Hawaiian Other Pacific Islander 10.5% 15.8% 15.8% 42.1% 10.5% 0.0% 5.3%Two or More Races 4.5% 12.6% 12.6% 23.4% 30.6% 10.8% 5.4%White 3.6% 11.8% 13.3% 19.1% 31.2% 17.1% 3.9%

Outcome 3 1 2 3 4 5 6 7Asian 4.3% 7.8% 5.7% 11.3% 15.6% 22.0% 33.3%Black 6.0% 10.5% 11.5% 13.5% 18.8% 20.3% 19.5%Hispanic 3.6% 9.9% 8.6% 13.0% 16.8% 25.6% 22.5%American Indian Alaskan 2.7% 5.5% 13.6% 15.5% 22.7% 20.0% 20.0%Hawaiian Other Pacific Islander 15.8% 5.3% 21.1% 15.8% 15.8% 5.3% 21.1%Two or More Races 2.7% 4.5% 7.2% 13.5% 20.7% 22.5% 28.8%White 2.7% 6.6% 7.8% 10.6% 17.0% 20.5% 34.9%

41

PREDICTED PATTERN #6

Children will differ from one another in their exit scores in reasonable ways. (At exit there will be a few children with very high or very low numbers.

42

OSEP PROGRESS CATEGORIES

43

PROGRESS CATEGORIESPLEASE REFER TO HANDOUT “CHILD OUTCOMES DATA CONVERSION”

Percentage of children who:

a. Did not improve functioningb. Improved functioning, but not sufficient to move

nearer to functioning comparable to same-aged peers

c. Improved functioning to a level nearer to same-aged peers but did not reach it

d. Improved functioning to reach a level comparable to same-aged peers

e. Maintained functioning at a level comparable to same-aged peers

44

PREDICTED PATTERN #7

Children will differ from one another in their OSEP progress categories in reasonable ways.

Note – A graph of this predicted pattern should have a similar distribution expected in entry & exit ratings (bell curve).

46

A B C D E0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

2010-2011 Outcome 1 Progress Category Percentages

Axis Title

A B C D E0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

2010-2011 Outcome 2Progress Category Percentages

Axis Title

A B C D E0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

50.0

2010-2011 Outcome 3Progress Category Percentages

Axis Title

SUMMARY STATEMENT #1 PLEASE SEE HANDOUT – “SUMMARY STATEMENTS FOR TARGET SETTING

1. Of those children who entered the program below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they exited the program.

c + d___ a + b + c + d

51

SUMMARY STATEMENT #2

2. The percent of children who were functioning within age expectations in each Outcome by the time they exited the program.

d + e__ a + b + c + d + e

52

SS1: Greater than Expected Growth SS2: Exited Within Age Expectations0

10

20

30

40

50

60

70

80

90

100

2010-2011 Outcome 1 Social Emotional

Axis Title

SS1: Greater than Expected Growth SS2: Exited Within Age Expectations0

10

20

30

40

50

60

70

80

90

100

2010-2011 Outcome 2 Knowledge & Skills

Axis Title

SS1: Greater than Expected Growth SS2: Exited Within Age Expectations0

10

20

30

40

50

60

70

80

90

100

2010-2011 Outcome 3 Action to Meet Needs

Axis Title

Wrap Up &

Next Steps

58

TAKE HOME MESSAGE

If you conclude the data are not (yet) valid, they cannot be used for program effectiveness, program improvement or anything else.

What do you if the data are not as good as they should be?

Answer: Continue to improve data collection through ongoing quality assurance

SO

60

KEY TO GOOD DATA

Have a good outcome

measurement

QUALITY CHECKS DURING DATA COLLECTION

During

Ongoing supervision of implementationFeedback to implementersRefresher training

61

QUALITY REVIEW OF COSF TEAM DISCUSSION:• Team discussion• Parent input• Documentation of rating

discussion• Multiple assessment

sources• Child’s everyday

functioning – Not test scores

• Child’s full range of functioning (AE/IF/F)

Ongoing Supervision:• Feedback to teams

critical• Refresher training• Beware of “auto pilot”

and “drift”

Methods:• Observation• Videos

62

QUALITY REVIEW OF COMPLETED COSFS

• Complete?• Adequate evidence?• Match the outcome area?• Based on functional behaviors?• Across settings and situations?• Ratings consistent with evidence?

63

NEXT STEPS?

1. What steps might you take after today’s discussion?

2. What other data patterns would be interesting / helpful to look at and interpret?

3. What types of TA/support/follow up might be appropriate next steps?

64

Thank You for Participating Today!

For more information:www.collaboratingpartners.com

http://sped.dpi.wi.gov/sped_spp-preout

http://projects.fpg.unc.edu/~eco/index.cfm