measuring impact with a single case design: evaluating training on the wisconsin indian child...
TRANSCRIPT
![Page 1: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/1.jpg)
Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D.
NHSTES 2013
![Page 2: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/2.jpg)
The Logic of Single Subject Designs
![Page 3: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/3.jpg)
The Basics
Repeated measures Individuals are their own controlsBaseline phase
Obtains a profile of variation absent the intervention
Allows identification of systematic patterns indicative of maturational effects, seasonality, history
Provides a basis of comparison for treatment phase
Treatment phase Measurements taken during time the treatment is applied
Phases are compared to make inferences about treatment effect
![Page 4: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/4.jpg)
Overall Requirements
Consistent measures over time Intervention that can be described fully and implemented with fidelity
Systematic introduction of the intervention
Replication (looking for a functional relationship not an isolated incident of change)
![Page 5: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/5.jpg)
Requirements for Dependent Variables
Observable, quantifiable target behavior (dependent variable)
Can be measured repeatedlyCan be measured with a high degree of inter-observer agreement
For training Training could be expected to impact it Impact would be relatively immediate
![Page 6: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/6.jpg)
Examples of CW Training related Dependent variables
Increased identification of children subject to ICWA
Increase in timely/accurate completion of risk and safety assessment tools
Increase in use of SMART objectives in case plans
Increased presence of concurrent plans in case files
![Page 7: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/7.jpg)
Threats to Validity with Repeated Measures Designs
History-another event occurring at the same time as the intervention that could affect the dependent variable
Maturation-normal developmental processes occurring over time that could explain the results
Are others but these are particularly relevant
![Page 8: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/8.jpg)
Stability of Data
Do the data represent a stable pattern or are they unpredictable?
Minimum of 3 separate, consecutive observations required per phase (Tankersley NHSTES 2012)
The more variable the data, the more data points are needed
Are several methods for representing background variability (e.g. putting a confidence interval around a phase mean)
![Page 9: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/9.jpg)
Length of Baseline and Stability
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
T11
T12
T13
T14
T15
0
10
20
30
40
50
60
70
80
Hypothetical Baseline A
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
T11
T12
T13
T14
T15
0
10
20
30
40
50
60
70
Hypothetical Baseline B
![Page 10: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/10.jpg)
Common Types of Single Subject Designs
![Page 11: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/11.jpg)
Types of Designs: “B Design”
Monitors the dependent variable during treatment
Shows trend but can’t make causal inference
![Page 12: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/12.jpg)
Types of Designs: “AB Design”
Shows change from baseline to treatment
May allow causal inference, but doesn’t control for history
![Page 13: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/13.jpg)
Types of Designs: “ABAB” Design
Allows causal inference but only works where treatment can realistically be withdrawn
Jan. Feb. Mar. Apr. May Jun. Jul. Aug. Sept. Oct. Nov. Dec. Jan. Feb. Mar. Apr.0
10
20
30
40
50
60
70
Percentage of Court Reports Completed on Time
A B A B
![Page 14: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/14.jpg)
Types of Designs: “Multiple Baseline”
Staged start for the intervention for different groups
Allows causal inference where intervention cannot be withdrawn
![Page 15: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/15.jpg)
Project Background
![Page 16: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/16.jpg)
“Best Outcomes for Indian Children”
Tribally-driven collaborative effort between
WDCF, MCWIC, and the 11 Wisconsin Tribes
![Page 17: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/17.jpg)
WICWAlaw
•The Project is focused on the state-wide implementation of the Wisconsin Indian Child Welfare Act (WICWA), which became law in December 2009
• The WICWA is a codification into state law of the Federal Indian Child Welfare Act (ICWA), which became law in 1978.
![Page 18: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/18.jpg)
Goals of project
• SHORT TERM • Train CW agencies on
tribal child welfare practices
• Modify DCF Tribal child welfare approaches
• Incorporate WICWA requirements into court procedures and the legal process
• Update Adm. Rules and program standards to integrate WICWA
• Improve Tribal/State child welfare relationships
• LONG TERM• Strengthen relationships
b/w state, county, adoption agencies, state and tribal courts
• Increase state wide understanding of the history and purposes of the acts in child welfare system
• Increase identification of ICWA eligible children
• Increase formal notice to tribes
• Increase adherence to WICWA placement preferences
![Page 19: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/19.jpg)
State Advisory Board
Variety of disciplines involved in the child welfare continuum
•Recommends policy and practice changes based on stakeholder input
•Three working subcommittees: • Curriculum• Qualified Expert Witness• Active Efforts
![Page 20: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/20.jpg)
Cross-systems integration
• Legislative Branch – Codification • Judicial Branch – State Court Office - Children’s court
Improvement Program– On going judicial training – Revised ICWA Court Forms
• Wisconsin Public Defenders Association • Executive Branch – Department of Children and Families • Specific Programs and internal
Departments
![Page 21: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/21.jpg)
Key Implementation Drivers(NIRN)
Specific drivers to effect system change in this project:
•Leadership
•Training
•Coaching
•Systems Intervention
•Facilitative Administration
•Decision Support Data Systems
National Implementation Research Network (NIRN)
![Page 22: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/22.jpg)
“Pulling multiple levers”
Advisory Board and stakeholder workgroups
New training on WICWA offered to all child welfare staff, supervisors, central office
Specialized legal training for attorneys, briefings for judges
Revised WDCF policies Desk aids for case workersChanges to eWiSACWIS system Revisions to CQI system for review of ICWA cases
![Page 23: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/23.jpg)
Examples of outcomes: ICWA records generated
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July -D
ec. 1
20
200400600800
10001200
162 146 167347
519668 714
963
Number of ICWA Records Cre-ated in eWiSACWIS January 2009 -
December 2012
![Page 24: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/24.jpg)
Examples of outcomes: ICWA identifications
![Page 25: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/25.jpg)
Summary of Project Overview
Training is one of multiple systemic interventions, with overlapping implementation periods
Strong evidence that trainees are satisfied, are learning a lot, and are motivated/plan to use what they’ve learned on the job when they have opportunity
Emerging evidence of improved outcomes in eWiSACWIS data
Can we determine the impact of training over and above that of the new WICWA tab in eWiSACWIS?
![Page 26: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/26.jpg)
Analysis of Single Subject Data: Our findings
![Page 27: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/27.jpg)
Types of Analysis for Single Case Designs
Visual Look for obvious contrast between phases in
Level Trend Overlap Variability
More and larger contrasts are evidence of importance of change
Immediacy is evidence of importance of change
Statistical Apply statistical tests of significance to patterns
![Page 28: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/28.jpg)
Advantages and Disadvantages
Visual Analysis
Simple; e.g. graphs and descriptive statistics
Differences that are obvious are more likely to be meaningful
Low power
Danger of confirmatory bias and over-interpretation of random variation
Low inter-rater reliability
Statistical Analysis More complex; e.g.
regression discontinuity models
Higher power
Less prone to human error and biases
Statistical significance ≠ practical significance
Require a long time series
Must meet assumptions about independent distribution of residuals (autocorrelation)
![Page 29: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/29.jpg)
Combined Graphical and Statistical Analysis
Pros Graphical aids like trend lines and means can aid in interpretation and improve inter-rater agreement about change
Cons Still suffer from risk of over-interpreting random fluctuations in a short time series
Recommendations (Nugent 2010)• Use both mean referenced and and trend
referenced representations of background variability to supplement interpretation
![Page 30: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/30.jpg)
AB DesignVisual Analysis Immediacy of Change
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July -D
ec. 1
20.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Percentage of American Indian Children1 Discharged from Out of Home Placement
Identified as ICWA Children Jan-uary 2009 -December 2012
![Page 31: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/31.jpg)
AB DesignLevel of Change: Visual Analysis Phase Means and Medians
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July -D
ec. 1
20.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Percentage of American Indian Children1 Discharged from Out of Home Placement
Identified as ICWA Children Jan-uary 2009 -December 2012
![Page 32: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/32.jpg)
AB DesignCombined Method:
Background Variability Relative to Mean; 2 SD method
Nourbakhsh and Ottenbacher (1994) Ja
n.-Ju
ne 0
9
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July -D
ec. 1
20.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Percentage of American Indian Children1 Discharged from Out of Home Placement
Identified as ICWA Children Jan-uary 2009 -December 2012
![Page 33: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/33.jpg)
AB DesignCombined Method: Percentage of Data Points Exceeding the Median (PEM)
Ma (2006)Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July -D
ec. 1
20.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Percentage of American Indian Children1 Discharged from Out of Home Placement
Identified as ICWA Children Jan-uary 2009 -December 2012
![Page 34: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/34.jpg)
AB Design: Visual Analysis Trend Based
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July-D
ec. 1
20%
20%
40%
60%
Percentage of American Indian Chil-dren1 Discharged from Out of Home
Placement Identified as ICWA Children January 2009 -December
2012
Observed Predicted
![Page 35: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/35.jpg)
AB Design:
Combined Method Trend Based
![Page 36: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/36.jpg)
WICWA Training Evaluation: What we hoped to see
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July-D
ec. 1
20
20
40
60
80
Average Percentage of Indian Chil-dren Identified as ICWA by Time
Period and Training Group
Group 1 Group 2 Untrained
![Page 37: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/37.jpg)
Combination DesignAB1
AB1B2
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July-D
ec. 1
20.00%
30.00%
60.00%
Untrained
B
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July-D
ec. 1
20.00%
20.00%40.00%60.00%
Training Group 1
A
A
B1 B2
![Page 38: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/38.jpg)
Data Considerations, Lessons Learned, and Recommendations
![Page 39: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/39.jpg)
Data Considerations
Need repeated measures over timeNeed sufficient numbers at each time period
Definitions of the measures need to remain constant
Administrative data sources are both promising and challenging
Offer access to measurements of child and family outcomes over time
Not designed for research Extracts needed for analysis and how they are drawn matters
Getting to the right variables can be like peeling an onion!
![Page 40: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/40.jpg)
The Unit of Analysis: Unique Child
Unique Child Most Recent Completed Episode of Care
Unique Child Earliest Removal in Study Period
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July-D
ec. 1
20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Average Percentage of Indian Children Identified as ICWA by
Time Period and Training Group
untrainedTrng1 May-Dec11
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July-D
ec. 1
20
0.10.20.30.40.50.60.70.80.9
Average Percentage of Indian Children Identified as ICWA by
Time Period and Training Group
untrained trng1 may-Dec11
![Page 41: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/41.jpg)
The Unit of Analysis: Snapshot (duplicated count)
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July-D
ec. 1
20
0.2
0.4
0.6
Average Percentage of Indian Chil-dren Identified as ICWA by Time
Period and Training Group
Untrained Trng1 May-Dec11
![Page 42: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/42.jpg)
Questions for Discussion
What types of training applications might this work for?
How do we separate the contribution of training from other factors affecting implementation?
Is it even possible, feasible, or necessary to tease out the effects of multiple contemporaneous interventions?
![Page 43: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/43.jpg)
Further Reading
Nugent, William R., (2010). Analyzing Single System Design Data. New York : Oxford University Press
Tankersley, Harjusola-Webb, and Landrum (2012) Using single-subject research to establish the evidence base of special education. Intervention in School and Clinic. 44(2). Pp. 83-90
![Page 44: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/44.jpg)
Exercise
![Page 45: Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D. NHSTES](https://reader035.vdocuments.net/reader035/viewer/2022081603/56649e235503460f94b10651/html5/thumbnails/45.jpg)
In groups
Discuss a training evaluation situation where single case methods might be appropriate
On your worksheet list Your evaluation question The type of single case design you would use
Data sources Potential issues/pitfalls
Report out