![Page 1: Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact Study Presenters: Alan Werner, co-Principal](https://reader036.vdocuments.net/reader036/viewer/2022081504/5697bfd81a28abf838caef56/html5/thumbnails/1.jpg)
Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact StudyPresenters:
Alan Werner, co-Principal Investigator
Laura Peck, co-Principal Investigator
Project Director:
Gretchen Locke
APPAM Conference November, 2013
![Page 2: Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact Study Presenters: Alan Werner, co-Principal](https://reader036.vdocuments.net/reader036/viewer/2022081504/5697bfd81a28abf838caef56/html5/thumbnails/2.jpg)
Abt Associates | pg 2
Presentation Overview
What we know and don’t know about what works
The HPOG Impact Study
Strategies to “Get Inside the Black Box”
Q&A
![Page 3: Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact Study Presenters: Alan Werner, co-Principal](https://reader036.vdocuments.net/reader036/viewer/2022081504/5697bfd81a28abf838caef56/html5/thumbnails/3.jpg)
Abt Associates | pg 3
Effects of Training: Progress to Date
It is well-established that vocational training and employment support programs for low-income individuals work
– Strong experimental impact research supports claims of effectiveness of specific approaches
– Best when program model tested in multiple sites and/or when program models tested head-to-head (e.g., HCD vs. LFA in NEWWS)
– But, new program models within Career Pathways framework-based programs, and…
![Page 4: Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact Study Presenters: Alan Werner, co-Principal](https://reader036.vdocuments.net/reader036/viewer/2022081504/5697bfd81a28abf838caef56/html5/thumbnails/4.jpg)
Abt Associates | pg 4
Challenges in Evaluating Training
Challenge has been to get inside the “black box” of program to develop findings more useful for policy and program design, for example:
– What “parts” of a tested program are most responsible for impacts?
– What accounts for variation in impacts across multiple program realizations?
– What design and implementation strategies work better than others?
– What works best for whom?
![Page 5: Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact Study Presenters: Alan Werner, co-Principal](https://reader036.vdocuments.net/reader036/viewer/2022081504/5697bfd81a28abf838caef56/html5/thumbnails/5.jpg)
Abt Associates | pg 5
HPOG and Its Impact Evaluation
Career Pathways framework-based training for TANF and low-income individuals to pursue healthcare sector careers
HPOG-Impact is part of a rich research “portfolio” at ACF
Impact Evaluation involves an experimental design, with randomization of eligibles to control and treatment groups, with randomization to enhanced treatment in some locations
![Page 6: Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact Study Presenters: Alan Werner, co-Principal](https://reader036.vdocuments.net/reader036/viewer/2022081504/5697bfd81a28abf838caef56/html5/thumbnails/6.jpg)
Abt Associates | pg 6
Study Sample and Data Collection
Sample size– Individuals: about 10,500 overall: 7,000 T; 3,500 C
– Study sites: 38 study sites programs across 20 grantees
– Planned variation sample (TBD)• Peer support
• Emergency financial assistance
• Non-cash incentives
Data collection– At baseline (before RA), from PRS & supplement
– Quarterly wage data (NDNH)
– Follow-up surveys at 15 months post-randomization
– Implementation study site visits
– Grantee and other surveys from NIE
![Page 7: Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact Study Presenters: Alan Werner, co-Principal](https://reader036.vdocuments.net/reader036/viewer/2022081504/5697bfd81a28abf838caef56/html5/thumbnails/7.jpg)
Abt Associates | pg 7
Research Questions
HPOG-Impact will address the following questions:
1. What impacts do the HPOG programs as a group have on the outcomes of participants and their families?
2. To what extent do those impacts vary across selected subpopulations?
3. Which locally-adopted program components influence average impacts?
4. To what extent does participation in a particular component (or components) change the impacts experienced by individual trainees?
![Page 8: Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact Study Presenters: Alan Werner, co-Principal](https://reader036.vdocuments.net/reader036/viewer/2022081504/5697bfd81a28abf838caef56/html5/thumbnails/8.jpg)
Abt Associates | pg 8
Research Questions
HPOG-Impact will address the following questions:
1. What impacts do the HPOG programs as a group have on the outcomes of participants and their families?
2. To what extent do those impacts vary across selected subpopulations?
3. Which locally-adopted program components influence average impacts?
4. To what extent does participation in a particular component (or components) change the impacts experienced by individual trainees?
![Page 9: Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact Study Presenters: Alan Werner, co-Principal](https://reader036.vdocuments.net/reader036/viewer/2022081504/5697bfd81a28abf838caef56/html5/thumbnails/9.jpg)
Abt Associates | pg 9
Sources of Variation
Generating new evidence on the role of program components in generating impacts from HPOG sites will encompass:
Natural variation in program features across sites
Planned variation in certain sites where subsets of participants are randomized to gain access to a specific program enhancement in addition to the basic program
Person-to-person variation in components of the offered intervention actually received by individuals
![Page 10: Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact Study Presenters: Alan Werner, co-Principal](https://reader036.vdocuments.net/reader036/viewer/2022081504/5697bfd81a28abf838caef56/html5/thumbnails/10.jpg)
Abt Associates | pg 10
Sources of Variation
Generating new evidence on the role of program components in generating impacts from HPOG sites will encompass:
Natural variation in program features across sites
Planned variation in certain sites where subsets of participants are randomized to gain access to a specific program enhancement in addition to the basic program
Person-to-person variation in components of the offered intervention actually received by individuals
![Page 11: Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact Study Presenters: Alan Werner, co-Principal](https://reader036.vdocuments.net/reader036/viewer/2022081504/5697bfd81a28abf838caef56/html5/thumbnails/11.jpg)
Abt Associates | pg 11
Intervention Components by Site and RA Sample: An Illustration
SiteP Q R S N
Type of Program Component: Naturally occurring, universal F F F F F Naturally occurring, varied – G E H – Randomized enhancement – E – – EComponents Provided for Each Random Assignment Sample:
C group sample A B C D LT1 sample A, F B, F, G C, F, E D, F, H L, FT2 sample – B, F, G, E – -- L, F, E
![Page 12: Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact Study Presenters: Alan Werner, co-Principal](https://reader036.vdocuments.net/reader036/viewer/2022081504/5697bfd81a28abf838caef56/html5/thumbnails/12.jpg)
Abt Associates | pg 12
Sources of Variation
Extracting the best information on the role of program components in generating impacts from HPOG sites will encompass:
Natural variation in program features across sites
Planned variation in certain sites where subsets of participants are randomized to gain access to a specific program enhancement in addition to the basic program
Person-to-person variation in components of the offered intervention actually received by individuals
![Page 13: Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact Study Presenters: Alan Werner, co-Principal](https://reader036.vdocuments.net/reader036/viewer/2022081504/5697bfd81a28abf838caef56/html5/thumbnails/13.jpg)
Abt Associates | pg 13
Conceptually…
When exposed to treatment…
• used program component (e.g., emergency assistance)
• achieved interim outcome (e.g., recognized credential)
If exposed to treatment, would have…• used program component
(e.g., emergency assistance)• achieved interim outcome
(e.g., recognized credential)
![Page 14: Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact Study Presenters: Alan Werner, co-Principal](https://reader036.vdocuments.net/reader036/viewer/2022081504/5697bfd81a28abf838caef56/html5/thumbnails/14.jpg)
Abt Associates | pg 14
Practically… Step 1
Step 1: Use baseline (exogenous) characteristics to predict subgroup membership– To capitalize on the internal validity of the experimental
design, use symmetric identification of T and C subgroups
HPOG application– Use baseline variables…
• demographics, supplemental baseline Qs on efficacy, work preferences, barriers/needs
– …to predict participation in various program components:• used emergency assistance, supports, used child care
support/services, accessed majority of available supports
– …or to predict selected achieved/short-term outcomes (mediators):
• earned recognized credential, found healthcare job
![Page 15: Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact Study Presenters: Alan Werner, co-Principal](https://reader036.vdocuments.net/reader036/viewer/2022081504/5697bfd81a28abf838caef56/html5/thumbnails/15.jpg)
Abt Associates | pg 15
ASPES: Steps 2 & 3
Step 2: Estimate impacts on predicted subgroups– Impact estimates are unbiased
Step 3: Convert estimated impacts for predicted subgroups to represent actual subgroups– Conversion rests (1) on an assumption of the homogeneity
of impacts among those predicted to be in a subgroup; and (2) on the foundation of an experimental impact estimate
![Page 16: Getting Inside the “Black Box” – Capitalizing on Natural and Random Variation to Learn from the HPOG Impact Study Presenters: Alan Werner, co-Principal](https://reader036.vdocuments.net/reader036/viewer/2022081504/5697bfd81a28abf838caef56/html5/thumbnails/16.jpg)
Further Information
Molly IrwinFederal Project Officer, HPOG HHS/ACF/[email protected]
Alan Werner & Laura PeckCo-Principal InvestigatorsAbt Associates [email protected][email protected]