enhance update research underway on the validity of the child outcomes summary (cos) process eco...
TRANSCRIPT
![Page 1: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/1.jpg)
ENHANCE UpdateResearch Underway on the Validity of the Child Outcomes Summary (COS) Process
ECO Center Advisory Board Meeting
March 8, 2012Arlington, VA
ENHANCE is funded by grant R324A090171 from the the U.S. Dept. of Education, Institute for Educational Sciences
![Page 2: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/2.jpg)
Topics
• Project design - review• Data collection progress• What we are learning
– Implementation– Child assessment study – State data study
• Next steps
![Page 3: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/3.jpg)
Project Design
![Page 4: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/4.jpg)
1. Conduct a program of research to examine the validity of ratings generated by COS and identify conditions that lessen validity.
2. Revise the COS and supporting materials based on study findings.
3. Identify a series of validity analyses that can feasibly be conducted in states to allow each state to examine the validity of its own COS data on an ongoing basis.
ENHANCE Project Objectives
![Page 5: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/5.jpg)
Validity – What Are We Trying to Demonstrate?
• Validity is NOT a characteristic of an assessment or measurement device.
• Validity is a characteristic of the data produced by the tool and how these data are used.
• Are data valid for the purpose of…..• Implications:
– State A’s COS data could be valid; – State B’s COS data could not be.
Standards for Educational and Psychological Testing(1999) by American Educational Research
Association, American Psychological Association, National Council on Measurement in Education
![Page 6: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/6.jpg)
Validation Process
• Validation process: – Develop propositions (validity argument) -
If data were valid for this use, then we would see….
– Collect evidence to examine each of those propositions
![Page 7: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/7.jpg)
Examples of propositions in the COS Validity Argument
3. Children differ from one another with regard to level of functioning in the 3 outcome areas as reflected in COS ratings.
7. Functioning (COS ratings) in an outcome area at time 1 is related to functioning in that area at a later point in time.
9. COS ratings will be related to the nature and severity of the child’s disability.
![Page 8: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/8.jpg)
Design
Real World
Not controlled Conditions
![Page 9: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/9.jpg)
Design:37 Project Data Collection Sites
19 Programs Part C • Illinois• Maine• Minnesota• New Mexico• North Carolina• Texas • Virginia
18 Districts Part B Preschool • Illinois• Maine• Minnesota• New Mexico • South Carolina• Texas
9
![Page 10: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/10.jpg)
Studies and Data Collection Progress
![Page 11: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/11.jpg)
ENHANCE Studies
• Provider Survey• Team Decision-Making• Comparison with
Child Assessments • State Data Study
![Page 12: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/12.jpg)
Provider Survey
Goals• Learn about COS implementation – processes in use• Identify providers’ knowledge and training experiences • Describe perceptions about if COS produces an accurate
rating and influences on that• Understand impact of COS on practice• Process & Sample
• Online survey responses• All providers in program who participate in COSStatus• Survey underway, continues through April
![Page 13: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/13.jpg)
Team Decision-Making Study
Goals• Examine understanding and
application of outcomes and rating criteria
• Describe team process • Identify if ratings are consistent with
evidence discussedProcess & Sample• Video teams discussing COS ratings
• 210 children’s teamsStatus• Data collection underway• Code videos this summer & fall
![Page 14: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/14.jpg)
What Are We Learning?
![Page 15: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/15.jpg)
Considerable variability across states and even across programs, within a state Training Ongoing staff support and quality assurance Teaming (not just for COS) Parent involvement Timing and Process
Implication: Results will tell us about COS validity under
real-world conditions
Implementation
![Page 16: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/16.jpg)
Number of Providers in COS Ratings - Preliminary
1 2 3 4 5 or more0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
EI % (n = 99)ECSE % (n = 61)
Perc
enta
ge o
f C
OS f
orm
s
Number of providers
![Page 17: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/17.jpg)
Goals• Compare entry and exit COS ratings
to BDI-2 and Vineland-II scores• Compare conclusions from COS
and assessments
Process & Sample• Longitudinal, external assessor at
program entry & program exit • 216 children
Study Status• Local, trained assessors in place• Recruiting families since Aug. 2010• Sample shows expected variability,
including initial COS ratings, tool scores
Comparison with Child Assessments Study
![Page 18: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/18.jpg)
Comparison with Child Assessments Study – Preliminary…
![Page 19: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/19.jpg)
3. There is variability in children’s functioning in the three outcome areas and that variability is reflected in the COS ratings.
Validity argument claims
![Page 20: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/20.jpg)
Distributions of Preliminary COS Ratings (1-7)
EI(n=71)
ECSE(n=49)
Ratings
![Page 21: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/21.jpg)
3. There is variability in children’s functioning in the three outcome areas and that variability is reflected in the COS ratings.
10. COS ratings in the corresponding outcome areas are moderately correlated with:
o the social-emotional (Outcome 1), o cognitive (Outcome 2), o communication (Outcome 2), and o adaptive (Outcome 3) domain scores
of assessment tools.
Validity argument claims
![Page 22: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/22.jpg)
• Methods table
Methods
![Page 23: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/23.jpg)
• Preliminary correlations between COS Ratings and assessment tools
• What expect to see?
Methods
.70
.42
.00
1.0
![Page 24: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/24.jpg)
Preliminary Correlations: COS Ratings & Assessment Scores
ECSE larger COS-Assessment Correlations than EI
COS with relevant assessment tool domains EI Mean (range)
ECSE Mean (range)
Positive Social Relationships 0.33 (.20-.52) 0.61 (.51-.68)
Acquiring and Using Knowledge and Skills 0.25 (.22-.28) 0.70 (.67-.75)Taking Appropriate Action to Meet Needs 0.49 (.36-.64) 0.47 (.40-.57)
![Page 25: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/25.jpg)
Correlations: BDI-2 and Vineland-II Domains
Vineland-II and BDI-2 EI Mean (range)ECSE Mean
(range)
Personal-Social 0.54 0.78
Cognitive/Communication 0.5 (.37-.63) 0.74 (.69-.83)
Adaptive /Motor 0.49 (.43-.67) 0.73 (.66-.86)
ECSE larger BDI-Vineland Correlations than EI
![Page 26: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/26.jpg)
Methods
• COS Group 1 – Ratings of 1, 2, 3
• COS Group 2 – Ratings of 4, 5
• COS Group 3 – Ratings of 6, 7
![Page 27: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/27.jpg)
Outcome 1: Positive Social Relationships
EI(n=71)
ECSE(n=49)
![Page 28: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/28.jpg)
EI(n=71)
ECSE(n=49)
Outcome 2: Acquiring and Using Knowledge and Skills
![Page 29: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/29.jpg)
EI(n=71)
ECSE(n=49)
Outcome 3: Taking Appropriate Action to Meet Needs
![Page 30: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/30.jpg)
Summary of preliminary findings
Patterns• Means for groups generally follow expected directions
on assessment tools• Group comparisons showed expected differences• Effect sizes were nearly all larger for ECSE than EI
For COS – assessment tool comparisonsFor comparisons between assessment tools by the
same external assessor
More data are needed for final conclusions
![Page 31: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/31.jpg)
State Data Study
Goals• Examine characteristics of COS data
and relationships to other variables• Look for consistency in patterns
across states to test claimsSample• All valid COS data within the state for a reporting year • 15-18 states conducting all analyses• Additional states sharing select analysesStatus• Piloted procedures with 3 Part C, 3 Part B Preschool states• Now working with 4 Part C, 6 Part B Preschool states• Recruiting more states, requesting data
![Page 32: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/32.jpg)
7. Functioning, as reflected in the COS rating, in an outcome area at time 1 is related to functioning in that area at a later point in time.
Validity argument claims
![Page 33: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/33.jpg)
Year Taking Appropriate Action to Meet Needs
State A 08-09 0.47State A 09-10 0.52State A 10-11 0.50State B 08-09 0.61State B 09-10 0.61State C 10-11 0.59
Correlations:Entry and Exit Ratings Part B 619
Preliminary state data
![Page 34: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/34.jpg)
3. There is variability in children’s functioning in the three outcome areas and that variability is reflected in the COS ratings.
14. COS rating distributions at entry will be related to the disability-related characteristics of the population served by states.
Validity argument claims
![Page 35: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/35.jpg)
Part C 08-09 entry ratings across states Taking appropriate action to meet needs
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
State AState BState D
1 2 3 4 5 6 7
COS Ratings
![Page 36: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/36.jpg)
Next Steps and Reactions
![Page 37: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/37.jpg)
Next steps
• Gather more state data• Complete data collection
involving local programs/districts• Analyze provider survey results• Code videos
![Page 38: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/38.jpg)
• Questions? Reactions?• Implications for the national
data? • Implications for ECO?
Questions? Reactions?
![Page 39: ENHANCE Update Research Underway on the Validity of the Child Outcomes Summary (COS) Process ECO Center Advisory Board Meeting March 8, 2012 Arlington,](https://reader038.vdocuments.net/reader038/viewer/2022110212/56649e215503460f94b0ca73/html5/thumbnails/39.jpg)
Find out more
• ENHANCE Website– http://ENHANCE.sri.com
• ECO Center Website– http://www.the-ECO-center.org
• Contact ENHANCE staff– E-mail: [email protected]