pushing forward from bhr through random individual-level variation in program components within...

Pushing Forward from BHR through Random Individual-Level Variation in Program Components within Sites: The HPOG Impact Study

Stephen H. Bell

APPAM Research Conference

November 7, 2013

Abt Associates | pg 2

Levels and Mechanisms of Program Variation in an RCT Context

MECHANISMLEVEL Random NaturalSite Cluster randomization

Multi-level estimationMulti-level explanatory models (e.g., BHR)

Individual Three-way within-site randomizationStandard experiment-al estimation

Analysis of symmetrically predicted endogenous subgroups (ASPES; Peck, 2003; Bell & Peck, 2013)


Strengthen multi-level explanatory model using three-way random assignment of individuals in a subset of sites (e.g., Health Profession Opportunity Grants evaluation)

_______________________________________

1. Omitted variable bias in BHR

2. How third experimental arm removes bias for the randomized element

3. How third experimental arm reduces bias for BHR-style cross-site comparisons

4. Implications / next steps

Goal and Outline


HPOG and Its Impact Evaluation

Career Pathways framework-based training for TANF and low-income individuals to pursue healthcare sector careers

HPOG-Impact is part of a rich research “portfolio” at ACF

Impact Evaluation involves an experimental design, with randomization of eligibles to control and treatment groups, with randomization to enhanced treatment in some locations


Study Sample and Data Collection

Sample size– Individuals: about 10,500 overall: 7,000 T; 3,500 C– Study sites: 38 study sites programs across 20 grantees– Planned variation sample (TBD)

• Peer support • Emergency financial assistance • Non-cash incentives

Data collection– At baseline (before RA), from PRS & supplement – Quarterly wage data (NDNH)– Follow-up surveys at 15 months post-randomization– Implementation study site visits– Grantee, staff/management and other surveys


T

Estimating this model gives , . . . , as non-experimental estimates of the influence on impact magnitude of the P program features

Two-Level Model of Site-Level Determinants of Intervention Impacts


What if there is an omitted site-level factor, , that

• Influences impact magnitudes

• Is not included in nor in site level covariates

• Correlates with one or more ?

Plug formula for ) into equation:

Omission of biases wherever Cov(

Omitted-Variable Bias Concern


= + + +

where = 1 if person i in site j is assigned to the basic intervention (T1)

= 0 otherwise (T2, C)

and = 1 if person i in site j is assigned to the enhanced intervention (T2)

= 0 otherwise (T1, C)

= +

Two-Level Model with Three-Arm Random Assignment


Add and subtract to expression for :

+ ( + ) + ( - ) +

This replaces with

)

which, unlike, does not depend on F no omitted variable bias on estimate of incremental impact of the enhancement

How the Third Arm Removes Bias


bias

Bias is smaller when and – the site-comparison-based non-experimental (BHR) estimate and the within-site experimental estimate – for the randomized element are closer together

Tweak the model – especially the impact equations – to reduce - reduces all bias

All estimates are distorted by the same F factor “whittle down” this distortion for one and do so for all

Using the Experimental Evidence to REDUCE the Bias Risk in Non-Experimental Estimates

𝜋𝓂∧𝑆 𝜋𝑒∧𝑆 𝜋𝑒∧𝑋


A) Vary as many individual program features as possible through 3-arm random assignment within sites (e.g., HPOG)

B) Better yet . . .

– get many dozens of sites

– randomize program features across sites

Use A to approximate B by adding lots of natural variation sites to a small number of three-arm random assignment sites

______________________________________

Next: Determine statistical power of the design for HPOG Impact

Then: Determine whether site-focused methods can support / enhance individual-focused methods like ASPES . . . or vice versa

Implications for Design / Extensions


Levels and Mechanisms of Program Variation in an RCT Context

MECHANISMLEVEL Random NaturalSite Cluster randomization

Multi-level estimationMulti-level explanatory models (e.g., BHR)

Individual Three-way within-site randomizationStandard experiment-al estimation

Analysis of symmetrically predicted endogenous subgroups (ASPES; Peck, 2003, Bell & Peck, 2013)

Further Information

Molly IrwinFederal Project Officer, HPOG HHS/ACF/[email protected]

Stephen BellPrincipal ScientistAbt Associates [email protected]

mailto:[email protected]

mailto:[email protected]

pushing forward from bhr through random individual-level variation in program components within...

Documents