Outcomes and Evidence Outcomes and Evidence Statements Statements

Outcomes and Evidence Outcomes and Evidence Statements Statements

Prepared for the Michigan Results Group Lansing, MI July 2005

Kathy HebbelerSRI International

2

Objectives for this presentationObjectives for this presentation

Review the ECO outcomes Introduce some concepts related to

evidence statements to assist Michigan in formulating its evidence statements

Share ECO’s recommendations for evidence statements

Describe what OSEP will be requiring Introduce the measurement approach

ECO is developing

3

Early Childhood Outcomes (ECO) Early Childhood Outcomes (ECO) Center: MissionCenter: Mission

Promote the development and implementation of child and family outcome measures for infants, toddlers and preschoolers with disabilities that can be used in national and state accountability systems

4

ECO CenterECO Center

5-year project funded by OSEP in October 2003Collaboration among: SRI International Frank Porter Graham Child Development

Institute (U of North Carolina, Chapel Hill) Juniper Gardens Children’s Project (U of

Kansas) National Association of State Directors of

Special Education University of Connecticut

5

Public Policy ContextPublic Policy Context

Age of accountability Accountability increasingly means

looking at results – not just process Applies across all private and

public human service and education programs

6

President’s Commission on President’s Commission on Excellence in Special Education Excellence in Special Education (2002)(2002)

Major Recommendation 1: Major Recommendation 1: Focus on results – not processFocus on results – not process

“IDEA will only fulfill its intended purpose if it raises expectations for students and becomes result-oriented—not driven by process, litigation, regulation and confrontation. In short, the system must be judged by the opportunities it provides and the outcomes achieved for each child.”

7

Federal push for outcome data: Federal push for outcome data: Intro to AcronymsIntro to Acronyms

GPRA =GPRA =Government Performance Government Performance

and Results Actand Results Act PART = Program Assessment Rating PART = Program Assessment Rating

ToolTool OMB = Office of Management and OMB = Office of Management and

BudgetBudget IDEA = Individuals with Disabilities IDEA = Individuals with Disabilities

Education ActEducation Act

8

GPRA (Government Performance GPRA (Government Performance and Results Act)and Results Act)

IDEA goals and indicators established IDEA goals and indicators established Indicators and data collection further along for Indicators and data collection further along for

school age population than for ECschool age population than for EC For early childhood, data have been collected For early childhood, data have been collected

on: on:

• Number of children served (Part C)Number of children served (Part C)

• Settings (both Part C and 619)Settings (both Part C and 619) Note: Part C = Programs for 0-3 year olds; 619 = Programs for 3-5

9

PART (Program Assessment PART (Program Assessment Rating Tool)Rating Tool)

Tool used to review federal programsTool used to review federal programs Four critical Assessment Areas Four critical Assessment Areas

includingincluding

““results and accountability”results and accountability” Programs given ratings from ineffective Programs given ratings from ineffective

to effectiveto effective Purpose – “to enhance budget analysis”Purpose – “to enhance budget analysis”

10

PART Evaluation: Findings and PART Evaluation: Findings and RecommendationsRecommendations

Part C and 619 Findings: No long-term child outcome goals or data

The PART required the Department to: 1. Establish long-term outcome-oriented

objectives 2. Develop a strategy to collect performance

data Office of Special Education Programs (OSEP)

asked states to provide outcome data in their Annual Performance Reports (APRs)

11

Reality Check (aka Challenges)Reality Check (aka Challenges) Michigan will be required to submit data to

OSEP OSEP has recently released the indicators on

which it will be requiring data and the timeline is impossible

ECO is simultaneously working on many of the same issues the states are. We have a measurement approach in development but it is far from complete.

We have no choice but to live with the uncertainties and move forward as best we can

12

Steps in Development of an Steps in Development of an Outcomes SystemOutcomes System

1. Identify overall purpose for the system (including priorities)

2. Identify outcome areas for children and families

3. Develop outcome statements4. Formulate the evidence statements 5. Identify measurement approaches (e.g.,

select instruments) ***Steps 6 to 11 after that

13

Need to acknowledge the range of Need to acknowledge the range of emotional responses to outcomes emotional responses to outcomes systemssystems

Hate it, have to do it so I will Hate it, have to do it but I won’t do a very

good job of it because it is a waste of my time Neutral, just another fad that will pass Not sure, might be good but not likely and

probably will be useless to harmful Worried that this will take time away from

children and families Hopeful that this will live up to the potential Excited about the prospect of having data on

child outcomes

14

Identify outcomes areasIdentify outcomes areas

What are the important outcomes areas? Child Family

Everyone wants to know “How are children doing?” but need to decide “with regard to what?”

15

DefinitionsDefinitions

Outcome—a statement of a measurable condition(s) desired for the population of children or their families Children take appropriate action to

meet their needs Families know how to advocate for

the services they need

16

Identify outcome areasIdentify outcome areas

Outcome statements are the system’s overall vision for children and for families

Not the same as outcomes on an IFSP or IEP which are very specific outcomes for an individual child or family

Outcomes in an accountability system are global statements of what we are trying to do for children and families

17

Identify outcome areasIdentify outcome areas

How should we think about child outcomes? “…for more than three decades,

researchers and service providers have struggled with both the identification of significant child outcomes and their valid and reliable measurement.”

National Research Council and Institute of Medicine, 2000

18

Identify outcome areasIdentify outcome areas

How should we think about child outcomes? Domains (language, cognitive, adaptive,

etc.) Functional skills (feeding, dressing, mobility,

etc.) Underlying functional capacities (Self-

regulation, engagement, knowledge acquisition skills, etc.)

Pre-academic skills (Early reading, math, science, etc.)

19

Pitfall Alert!Pitfall Alert! What we want for children and families

(desired outcomes) reflects our values What we want for young children may not be

the same as what can be easily measured When what is valued and what can be

measured easily or well do not line up, EITHER: Outcomes will be determined by what can be

measured easily or well, OR Some of the outcomes will be more easily measured

or more adequately measured than others

20

ECO Family OutcomesECO Family Outcomes

1. Families know their rights and advocate effectively for their children.

2. Families understand their children's abilities and special needs

3. Families help their children develop and learn.

4. Families have social supports. 5. Families are able to gain access to desired

services, programs, and activities in their community.

21

Need for an overarching goal for Need for an overarching goal for childrenchildren

The ultimate goal is for young children to be active and successful participants now and in the future in a variety of settings– in their homes, in their child care, preschool or school programs, and in the community

22

ECO Child OutcomesECO Child Outcomes

1. Children have positive social relationships

2. Children acquire and use knowledge and skills

3. Children take appropriate action to meet their needs

23

Issues from stakeholder Issues from stakeholder discussionsdiscussions

Functional outcomes Best practice – kind of outcomes

recommended for IFSPs and IEPs Consistent with transdisciplinary

service delivery Not reflected well in a pure domains

framework Not captured well in current

assessment tools

Children have positive social relationships

Children acquire and use knowledge and skills

Children take appropriate action to

meet their needs

Relation-ships with

adultsRelation-

ships with

peers

Follows group rules

Masters the environ-ment

Symbol use, abstract thinking

Appliesknowledge

Listening PlayExploring

Playing

Being curious

Practicing

Touching

Attending

EngagingPersisting

Knowledge of physical world & culture

Self-care, health and safety

Elaboration of the ECO OutcomesElaboration of the ECO Outcomes

Children will be active and successful participants now and in the future in a variety of settings.

ECO Outcomes:

Overarching Goal:

Children have positive social relationships

Children acquire and use knowledge and skills

Children take appropriate action to meet their needs

Domains: Communication Cognition

Motor

Social-Emotional

Approaches to Learning

Content Areas:

Language Arts Math

Science Social Studies Art

Music

Note: Each of these can be broken down further into sub-areas

Processes: Memory Self-regulation Recognizing and interpreting sensory input

Listening Attending

Body Functions: Vision Seeing

Movement-flexibility-strength-postural response

Etc.

Speech production Etc.

Self Help

Example of how movement can be sub-divided

Alternative Ways of Thinking about Child OutcomesAlternative Ways of Thinking about Child Outcomes

26

What is happening with these What is happening with these outcome areas and statements?outcome areas and statements?

ECO’s work is independent of OSEP OSEP has been involved in ECO’s work

from the beginning ECO submitted these outcomes to OSEP

as its recommendations ECO submitted recommended indicators

based on these outcomes to OSEP for the APR/SPP instructions that were available for public comment

27

Evidence Statements(or what do you want to know about those

outcomes)

28

The Importance of Evidence The Importance of Evidence Statements Statements

What constitutes evidence of good outcomes?

If you were asked to testify before your state legislature to demonstrate the effectiveness of EI, what kind of evidence do you want to have? (Purpose = accountability)

What kind of evidence about outcomes is needed by states and local programs to improve services for children and families? (Purpose = program improvement)

29

Definition of Evidence StatementDefinition of Evidence Statement

Evidence Statement - a statement that incorporates a statistic and provides evidence as to whether not an outcome has been achieved

% of children who have positive social relationships

% of children who show progress toward acquiring knowledge and skills

Note: States will be asked to submit a specified kind of evidence to OSEP. Each state will need to be able to produce this kind of evidence but might want other kinds of evidence as well.

30

Features of a Good Evidence Features of a Good Evidence StatementStatement

Credible: Based on valid data Meaningful: The evidence can be

interpreted Powerful: The evidence is convincing

to those who will be receiving it (purpose = accountability)

Useful: The evidence is helpful to those who will use it (purpose = program improvement)

31

MeasurementMeasurement Evidence Evidence Statements Statements

Evidence statements are the ultimate product in an outcomes-based accountability system

The nature of the evidence statement that can be produced will depend on what is measured and how often

32

Possible Categories of Evidence Possible Categories of Evidence Statements for Outcome DataStatements for Outcome Data

1. Status (achievement at one point in time)

Example: “% of children who [achieved this..]

2. Progress (change relative to earlier status)

Example: “% of children who made progress in..]

33

Two Ways to Think about Two Ways to Think about Progress with Young ChildrenProgress with Young Children

Continuous Progress = acquisition of new skills and behaviors (almost all children show this)

Or Age-anchored Progress = change in

developmental trajectory (i.e., narrowing the gap between a child with a delay and typical development; “less delayed” after intervention)

34

Examples of Two Kinds of Examples of Two Kinds of ProgressProgress

Continuous– increase in vocabulary Age-anchored– change from being

1.5 standard deviations below norm to .5 standard deviation

Hypothetical Language Acquisition Rates for Three Groups of Children: Change in

Developmental Trajectory (Progress toward Closing the Gap)

0

10

20

30

40

50

60

70

1 6 11 16 21 26 31 36 41 46 51 56Age in Months

La

ng

au

ge

Sc

ore

Typically developing children

Children with delays without intervention

Children with delays after intervention

36

Advantages and Disadvantages of Different Evidence Statements

37

Status (where children are)Status (where children are)

Reports the percentage of children who achieved or could do X

Would likely involve some kind of comparison to expectations for children of a given age

Examples: 53% of early intervention graduates were

rated as “typical and proficient” with regard to having positive social relationships

38

Status Evidence StatementStatus Evidence Statement

Pro Requires one measurement point Could be strong evidence Nearly all K-12 evidence statements are of this

form (grad. rates, “proficient” reading)Con What is the benchmark that makes sense for

all children with disabilities? How does one interpret the data (is this finding

good news or bad news?)

39

Continuous Progress (change Continuous Progress (change compared to earlier status)compared to earlier status)

Reports % of children who improved Progress on a curriculum-based

assessmentExamples:

% of preschool children who made progress toward having positive social relationships

% of preschool children who showed an increase of 3 of more objectives related to positive social relationships scale after one year

40

Continuous ProgressContinuous Progress

Pro Straightforward Could quantify extent of child growth Incorporates all levels of gain, even the very

small gains made by children with severe impairments

Con Requires at least 2 points of data Nearly all children will improve over time Weak evidence. What constitutes “good

news”?

41

Age-Anchored Progress (closing the Age-Anchored Progress (closing the gap toward typical development)gap toward typical development)

Captures progress relative to same age peers

Examples: 23% of children moved from low performing

to age appropriate in positive social relationships by kindergarten entry

With regard to acquiring and using knowledge and skills, 75% of children made progress sufficient to maintain their functioning at an age appropriate level

42

Age-Anchored Progress (closing the Age-Anchored Progress (closing the gap toward typical development)gap toward typical development)

Pro Consistent with the intent of intervening Could produce strong evidence

Con Requires at least 2 time points of data Closing the gap is not an expectation for all

children with disabilities. How is the evidence to be interpreted?

If some children don’t close the gap, is this interpreted as lack of success for the program?

43

Deciding on desired evidenceDeciding on desired evidence

Not an either-or Different levels will want different

kinds of evidence Many different kinds of evidence

can be produced from the same set of data If the desired evidence statements

are identified as part of the planning

44

What OSEP will be requiring from states for Part C and Preschool

Special Education

45

Reporting Requirement: Part CReporting Requirement: Part C

Percent of infants and toddlers with IFSPs who demonstrate improved:

A. Positive social-emotional skills (including social relationships)

B. Acquisition and use of knowledge and skills (including early language/communication)

C. Use of appropriate behaviors to meet their needs

46

The detailsThe details

a. % of infants and toddlers who reach or maintain functioning at a level comparable to same-aged peers

b. % of infants and toddlers who improve functioning

c. % of infants and toddlers who did not improve functioning.

If children meet the criteria for a, report them in a. Do not include children reported in a in b or c.

Illustration of 5 Possible Paths

0

10

20

30

40

50

60

70

1 6 11 16 21 26 31 36 41 46 51 56Age in Months

Sco

re

Maintained functioning comparable to age peers

Achieved functioning comparable to age peers

Moved nearer functioning comparable to age peersMade progress; no change in trajectory

Did not make progress

48

3 OSEP Evidence Statements3 OSEP Evidence Statements

OSEP a Maintained functioning comparable to age

peers Achieved functioning comparable to age peersOSEP b Moved nearer functioning comparable to age

peers Made progress; no change in trajectoryOSEP c Did not make progress

49

Two kinds of outcomes questions for Two kinds of outcomes questions for each child with a disabilityeach child with a disability

1. How is this child doing relative to the individualized outcomes that were established through the IFSP or IEP process?

2. How is this child doing relative to same-aged peers?

50

Part C Indicators Part C Indicators Family OutcomesFamily Outcomes

Percent of families participating in Part C who report that early intervention services have helped the family Know their rights; Effectively communicate their

children’s needs; and Help their children develop and learn.

51

Measurement guidelinesMeasurement guidelines

State selected data source, sampling permitted

% of families who report that early intervention services have helped their family in each of the three areas

52

SPP/APRSPP/APRMeasurement implications Measurement implications Requires data collection from family

members/caregivers Data at one time period

Asks families to recognize that early intervention services contributed to the change or status listed.

Does not require family data to be linked with child data

53

Key Questions Key Questions

How can Michigan provide the indicators being requested by OSEP?

What kinds of evidence statements does Michigan want to be able to produce for its own use?

What do the answers mean for measurement?

54

Producing Data on the Outcomes

55

Rolling up assessment data to Rolling up assessment data to the outcomesthe outcomes

Given: Different children will have different kinds of

assessment data. Also:

The same child may have several different kinds of assessment data.

Challenge: How to roll up multiple pieces of information (e.g., data from different assessment tools) to a single score?

56

Children will be active and successful participants now and in the future in a variety of settings.

ECO Outcomes:

Overarching Goal:

Children have positive social relationships

Children acquire and use knowledge and skills

Children take appropriate action to meet their needs

Domains: Communication Cognition

Motor

Social-Emotional

Approaches to Learning

Content Areas:

Language Arts Math

Science Social Studies Art

Music

Note: Each of these can be broken down further into sub-areas

Processes: Memory Self-regulation Recognizing and interpreting sensory input

Listening Attending

Body Functions: Vision Seeing

Movement-flexibility-strength-postural response

Etc.

Speech production Etc.

Self Help

Example of how movement can be sub-divided

Different assessments on different childrenDifferent assessments on different children

Roll up Process

Assessment 1

Assessment 2

Assessment 3

Assessment 4

Single Score

Roll up Process

Assessment 1

Assessment 2

Parent Report

Informed Clinical Opinion

Single Score

Different pieces of information on the same childDifferent pieces of information on the same child

59

Rolling up from the different Rolling up from the different assessmentsassessments

ECO is working on a scale that will provide a way to give a child a numerical rating on each outcome It is NOT an assessment tool. It assumes teachers and providers are

already using an assessment tool to collect information about how children are doing.

The scale is a way to take what those closest to the child know and put a number on it.

60

What does the outcomes scale What does the outcomes scale look like?look like?

5 pt and a 7 pt version The precision question….

The high point (5 or 7) indicates outcome achieved at an age-expected level

The lowest point (1) indicates the farthest distance from age-expectations

61

What does the outcomes scale What does the outcomes scale look like?look like?

Challenge is in how to anchor and provide the descriptors for the middle points on the scale.

Progress is moving up the scale between time 1 and time 2.

62

Measuring the Family OutcomesMeasuring the Family Outcomes

ECO working on a tool ECO working on a document with

guidance for states that reviews key decisions and includes several options

63

Staying in touchStaying in touch

Web site: the-eco-center.org Follow developments related to work of the Center Obtain ECO resource documents Obtain other related resources

Email: staff@the-eco-center.org ECO sends to existing listservs Can join the ECO mailing list

64

VisionVision

To build a system where all the pieces will fit together smoothly

Meaningful outcomes data collected regularly that can used in programs by programs AND reported to the state (and feds)

Local data

State needs

68

When Michigan can realize this vision, children and families will

reap the benefits.