Overview of ENHANCEResearch Underway on the Validity of the

Child Outcomes Summary Form (COSF)

Lauren BartonDonna Spiker

July 31, 2010

Measuring Child and Family Outcomes ConferenceArlington, VA

Topics• Need for ENHANCE• Considerations - building a

validity argument• ENHANCE project and studies• When and how will studies

be conducted?• Suggestions for project to

better meet state needs

The Challenge

Child Outcomes Goal of Early Intervention and Early Childhood Special Education

“…To enable young children to be active and successful participants during the early childhood years and in the future in a variety of settings – in their homes with their families, in child care, in preschool or school programs, and in the community.”

(from Early Childhood Outcomes Center, http://www.fpg.unc.edu/~eco/pdfs/eco_outcomes_4-13-05.pdf)

Demands Grew

Data Data Data Data

The Need

High quality child outcomes data for accountability

FAST

The Need for

An approach or tool that:• Captures multiple sources of information and

child’s functioning across settings• Doesn’t require programs to change

assessments• Relates to age-expected child functioning• Measures progress over time• Is inexpensive• Is ready to be implemented now• Is valid and reliable

The Response

The Child Outcomes Summary Form (COSF)

The COSF

Captures multiple sources of information and child’s functioning across settings

Doesn’t require programs to change assessments

Relates to age-expected child functioningMeasures progress over time InexpensiveReady to be implemented now? Valid and reliable

Result

More information is needed– to improve guidance

about the COSF process, – to document its reliability

and validity, and – to inform appropriate use

of the data we have

COSF is in use in over 40 states

What is Your Experience?

• Who is asking you for validity information on the COSF?

• What do they want to know?• What would you like to be able

to tell them?

• Is your state taking actions to look at the validity of COSF data on your own?

Building a Validity Argument

Early Evidence

Belief in potential for COSF to be valid based on:• Existing literature: team-based decision-making can

be reliable and valid• Existing literature: teams are effective in identifying

individual children’s functioning so that they can plan and deliver appropriate services

• Early data from states: pilot sites, small n’s showing similarity in distributions, sensible patterns for subgroups

• Anecdotal data from trainers: participants reach decisions fairly easily and consistently

Validity – What Are We Trying to Demonstrate?

• Validity refers to the use of the information• Are you justified in reaching the conclusion

you are reaching based on the data?

Standards for Educational and Psychological Testing(1999) by American Educational Research Association, American Psychological Association, National Council on Measurement in Education

Validity Questions

The question is NOT “Are the data valid?”• The question is: “Are the data for valid for the purpose of….?”• Are the data sufficiently trustworthy to lead to

sound decisions? Examples: funding, TA, focused monitoring…

• How much error is acceptable? There will be error…

Implications for COSF Data

• Validity is NOT a characteristic of an assessment or measurement device.

• It is a characteristic of the data produced by the tool and how these data are used.

• Implications: State A’s COSF data could be valid; State B’s COSF data could not be.

Implications for Studying Validity of COSF Data in States

• Need to examine the conditions under which COSF data produce more or less valid conclusions– Study across states with variation in how they

implement the COSF– Examine if COSF data leads to appropriate

conclusions when it is used– Use findings to generate guidance

• States can use guidance to structure conditions that increase the likelihood of generating valid data with the COSF process.

Validation Process

• Validation process: – Develop propositions (validity argument) -

If data were valid for this use, then we would see….

– Collect evidence to examine those propositions in various locations with differences in implementation

Examples of propositions in the COSF Validity Argument

3. Children differ from one another with regard to level of functioning in the 3 outcome areas as reflected in COSF ratings.

7. Functioning (COSF ratings) in an outcome area at time 1 is related to functioning in that area at a later point in time.

9. COSF ratings will be related to the nature and severity of the child’s disability.

12. COSF ratings will not be related to:– the particular assessment tool used – extraneous child characteristics such as gender and ethnicity

14. Distributions of COSF ratings at entry will be similar across states serving similar populations and related to the percentage of children served in Part C or Part B Preschool.

ENHANCE: The Project and The Studies

ENHANCE• Project launched by the Early Childhood Outcomes

Center (ECO) and SRI International• Funded by the U.S. Dept. of Education, Institute for

Educational Sciences• Series of studies designed to find out:

– the conditions under which the Child Outcomes Summary Form (COSF) Process produces meaningful and useful data for accountability and program improvement

– the positive and/or negative impact of the COSF process on programs and staff

– what revisions to the form and/or the process are needed

Five ENHANCE Studies

1) State Data Study2) Comparison with

Child Assessments 3) Team Decision-Making4) Record Reviews5) Provider Survey

State Data Study

• Extant data from states using COSF• State level population data on children birth-5

with disabilities served under IDEA• Analyze characteristics of COSF data and

relationships to other variables• Interested in volunteers for state data study

Sites for 4 Other Studies

PART C6 states with 3 local programs each

PART B Preschool6 states with 3 local programs each

18 Part C + 18 Part B Preschool = 36 ProgramsStates: IL, ME, MN, NM, NC, SC, TX

Comparison with Child Assessments StudyGoals Over time, compare:

– COSF ratings and scores on assessments (correlations)– Conclusions derived (progress category determinations)

Process COSF ratings and external BDI-2

& Vineland-II assessment– near program entry– again at program exit

Sample 6 children at each site (216 total)

Team Decision-Making Study

Goals• Do teams reach a rating consistent

with evidence presented?• Examine understanding and

application of outcomes and the rating criteria

Process• Videotape teams discussing and

reaching consensus on a COSF ratingSample• 10 children at each site (360 total)

– ½ entry COSF meetings– ½ exit COSF meetings

Record ReviewsGoals• How well do the ratings reflect the

broader evidence in the child’s file?• Was understanding of the outcomes and

rating criteria applied consistently?• Quantity and quality of documentation

used to determine the COSF rating• Generate a protocol, procedures, and

a set of analytic methods for future state useProcess• Onsite review of child records, including COSF ratings

and documentation.Sample• 10 children at each site (360 total)

½ entry records; ½ exit records

Provider SurveyGoals• What processes are being implemented to determine

COSF ratings? • What is knowledge of outcomes, COSF criteria, age-

expected behavior? • What is the impact of COSF on their practice?

Process Survey responses (online or paper)

Sample All providers in program who participate in COSF

Timing

2010 2011 2012…

Aug

.Se

pt.

Nov

. or

Jan.

Mar

ch

June

Child Assessments

Team Videos

Provider Survey

Record Reviews

Until children exit program

Discussion Questions

• What facets of the COSF process do you want studied?

• What kinds of evidence-based guidance would be most useful for you?

• Other questions, comments, or reactions?

Find out more

• ENHANCE Website– http://ENHANCE.sri.com

• ECO Center Website– http://www.the-ECO-center.org

• Contact ENHANCE staff– E-mail: ENHANCE@sri.com

Questions? Comments? Reactions?