c hendricks brown...funding contexts, fidelity monitoring and feedback systems used to deliver a...

Implementation Science: An Introductory Workshop for Researchers, Clinicians, Policy Makers and Community Members

Advanced Dissemination and

Implementation Research Designs

C Hendricks [email protected]

Director, Center for PrerventionImplementation Methodology for Drug Abuse

and HIV

http://cepim.northwestern.edu/MT-DIRC

http://www.cepim.northwestern.edu

http://www.feinberg.northwestern.edu/



mailto:[email protected]

http://cepim.northwestern.edu/


Acknowledgements

NIDA Funding P30DA027828 Center for Prevention Implementation Methodology for Drug Abuse and HIV (Ce-PIM, Brown & Mustanski)

Co-AuthorsLarry Palinkas USC – Sustainment R34 from NIDAGreg Aarons UCSD, Marisa Sklar UCSD, Brian Mustanski Northwestern, Nanette Benbow Northwestern

1




OutlineA. Responding to QuestionsB. Brief Review of Basics on Implementation

DesignsC. Roll-Out Designs for ImplementationD. Scaling Out Designs for Delivering

Interventions to Different Populations or through Different Delivery Systems

E. Predictors of SustainmentF. Responding to Questions

2




An Example Why Different Methods are Needed: Relationship between Dosage and

Outcome

• Traditional Clinical Dosage Trial– Randomly assign units to a dose– Measure outcome– Summarize Dose Response

3Innovative Methodology: Bloomberg SPH at Johns




Typical Dose Response Trial





Real Implementation: Amount of MH Treatment Last 10 Years vs Current Sx





Amount of MH Treatment Predicting Current Symptoms





A. Responding to Questions

• -Would like to learn more about the "Dynamic Wait List Control" design (Pros, Cons, etc.)

• -What are some important design considerations for hybrid efficacy + implementation design? Resources and examples would be wonderful.

• -Study designs/methodologies for translating data on barriers + facilitators into implementation strategies.

7




More Questions• -What are some RCT options for observational

studies with smaller samples. AND How do you choose the number of sites/organizations.

• -You have done quantitative work showing variation in physician practice or in patient uptake of something. Then you did qualitative studies asking about barriers and facilitators-now how do you take all this and design an intervention for a grant application to actually address some of the barriers you found?

8




More Questions

• -Walk through an example of aligning methods to the research question. Folks seem to want to use a fancy method (eg. Most, SMART) before clarifying the question.

9




B. Brief Review of D&I Designs

• Traditional Translational Pipeline• Head-to-Head Implementation Trials• Evaluation using Stages of Implementation

Completion• MOST and SMART Designs for Implementation• Quality Improvement and Control Charts

10




An Overview of Research and Evaluation Designs for Dissemination and Implementation

C Hendricks Brown (Northwestern)Geoff Curran (VA / UAMS)Lawrence A. Palinkas (USC)Linda Collins (Penn State)Ken Wells (UCLA)Loretta Jones (Healthy African American Families II)Greg Aarons (UCSD)Rachel Tabak (Washington University)

Andrea Wallace (Univ of Iowa)Naihua Duan (Columbia)Brian Mittman (VA / Kaiser)Lori Ducharme (NIAAA), David Chambers (NCI), Gila Neta (NCI), Tisha Wiley (NIDA)Ken Chung (Columbia)John Landsverk (Oregon Social Learning Center)Gracelyn Cruden (Northwestern)

11




Could a program work?

Does a program work?

Making a program

work

Efficacystudies

Effectivenessstudies

Real

-wor

ld re

leva

nce

Time

Exploration

Preparation

Implementation

Sustainment

Local knowledge

Generalizable knowledge

Implementation Research

Traditional Translational Pipeline

Brown et al., ARPH 2017

Implementation Practice

Preintervention




Implementation Phases (EPIS Aarons et al)

• Exploration Phase is the organizational or community attention to and investigation of one or more approaches to improve a population’s outcomes

• Adoption38/Preparation Phase involves the decision to, introduce, change, or experiment with one or more programs, policies, or practices and how these relate to an evidence-base.

• Active Implementation Phase focuses on organizational, political and funding contexts, fidelity monitoring and feedback systems used to deliver a clinical/preventive intervention.

• Sustainment Phase95 refers to the continued use, effective delivery, credentialing96, or improvement over time, expansion or scaling97, or survivability of a clinical/preventive intervention.

Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence based practice

Prev Med Cardiovascular Epidemiology 2/13/2015

13




Existing Implementation

Supports for County, Agency,

Group Home

MTFC Intervention

MTFC Implementation

Supports for County, Agency,

Clinicians, Parent

Two-Arm TrialsEffectiveness vs. Implementation

14

MTFC Intervention MTFC

Intervention

CDTImplementation

Supports for County

Control Condition

Standard Implementation

Supports for County

Youth Youth

Youth Youth




Head-to-Head Trial of Two Implementation Strategies of the Same Clinical/Preventive Intervention

For Implementation, the Program Delivery System, rather than the Clinical/Preventive Intervention, is in the Foreground

Clinical/Preventive Intervention

Multilevel, Program Delivery System I

Landsverk J, Brown CH, et al., Design and Analysis in Dissemination Research 2017.

Clinical/Preventive Intervention

Multilevel, Program Delivery System II

Different

Same

Randomize





Randomize 51 Counties in CA and OH to Implementation Strategy and Time (Cohort)

Randomized Roll-Out Design*

16

40 CA Counties

26 Wait LIsted

CDT

Stnd

Wait Listed

13 Wait LIsted

COHORT 1 COHORT 2 COHORT 3 COHORT 4

*Brown, et al. 2009 Ann Rev PH

11 OH Counties

Prev Med Cardiovascular Epidemiology 2/13/201551 Counties Randomized to both Implementation and Timing




Process and Output Measures in CAL-OH Trial

Stages of Implementation Completion (SIC)

Chamberlain et al. Admin Policy Ment Hlth Ment Hlth Res. 2008;35(4):250-260Chamberlain et al. Implementation Science. 2011;6(1):116-124.http://www.implementationscience.com/content/pdf/1748-5908-6-116.pdf

Wang et al . Implementation Science. 2010;5(1):72.http://www.implementationscience.com/content/pdf/1748-5908-5-72.pdf

Saldana L (2014). Implementation Science 2014, 9:43http://www.implementationscience.com/content/pdf/1748-5908-9-43.pdf

17



http://www.implementationscience.com/content/pdf/1748-5908-6-116.pdf




Summary of Findings of CAL-OHBrown CH, Chamberlain P, Saldana L, Wang W, Padgett, C.,

Cruden G. (2014). Implementation Science, 9:134http://www.implementationscience.com/content/pdf/s13012-014-0134-8.pdf• Mixed Results

• No evidence that – CDT affected rate of adoption– CDT changed speed of implementation– Composite Score

• Evidence that – CDT increased numbers

of families served– CDT counties completed

implementation more thoroughly

18

0 5 10 15 20 25 30

05

1015

2025

30

CDT

IND

Number Served Quantiles for CDT versus IND (EQQ Plot)




http://www.implementationscience.com/content/pdf/s13012-014-0134-8.pdf


Is there contamination across Implementation Condition Due to Peer Influences?


StandardImplementation

CmmunityDevelopment Team Implementation

19




Evaluating Multiple Components of an Implementation Strategy : Multiphase Optimization

Strategy Trial (MOST) for Implementation• A comprehensive implementation strategy often requires a component

that is directed towards the system leadership, one at the clinic level, and one at the clinician level. Components also can be aimed at key processes: planning, educating, financing, restructuring, managing quality, and policy

Practices Randomized to: Feedback on prescribing rates w or w/o• health board comparator• text based messages• 9 month feedback• http://www.implementationscience.com/content/pdf/1748-5908-9-50.pdf

3 factors 8 strategies4 factors 16 strategies


20





3 Phases of a MOST Design• Selection (and pilot testing) of components• Optimization, test each component, typically in a balanced

factorial design• Test the single optimal combination against usual care in a

randomized trial

Dziak, J.D., Nahum-Shani, I., & Collins, L.M. (2012). Multilevel factorial experiments for developing behavioral interventions: Power, sample size, and resource considerations. Psychological Methods, 17, 153-175


21




Testing Adaptive Implementation Strategies: SMART Designs

SMART designs allow interventions to be tuned to prior responses.

Collins, et al. (2014). Optimization of behavioral dynamic treatment regimens based on the sequential, multiple assignment, randomized trial (SMART). Clinical Trials. Advance online publication.


22




A SMART Trial for Colorectal Cancer Screening CDC Guidelines: Yearly FOBT OR 5 Year Sigmoidoscopy +

3 Year FOBT OR 10 Year Colonoscopy

23

Yearly FOBT

Sigmoidoscopy + FOBT

Adherent

Adherent

Continue

Randomize

Monitor

Randomize

Randomize

Sigmoidoscopy + FOBT

Colonoscopy

Colonoscopy

Yearly FOBT

Non Adherent

Non Adherent




SMART Designs for Adaptive Implementation Strategies Kilbourne et al., Impl Sci 2014


158 community outpatient clinics using Re-Engage to implement evidence-based programs to address mood disorders.

24

Re-Engage

Responsive

Not Responsive

6 Months 12 Months 18 Months

Re-Engage

R

Enhanced

Enhanced

Standard

Standard





Quality Improvement and Optimization: Taking an engineering perspective

• Working systematically toward development of an implementation strategy that meets specific criteria

Produce LOCAL KNOWLEDGEWu S, Duan N, Wisdom JP, Kravitz RL, Owen RR, Sullivan G, Wu AW, Di Capua P et al., (2014). Admin Policy MH, Online First.

Collins, L. M., Nahum-Shani, I., & Almirall, D. (2014). Clinical Trials. Advance online publication.

25




Doing Better: Quality Improvement Strategies

• Statistical Control ChartsProblematic for Low Rates


26




Prev Med Cardiovascular Epidemiology 2/13/2015 27

Number of Youth Suicide Deaths from 1988 to 2002 in County

years

deaths

1988 1990 1992 1994 1996 1998 2000 2002

01

23

45

6




Doing Better: Quality Improvement Strategies

• Statistical control

Monitor One of the Key Hypothesized Change Factors

Gatekeeper Training: Attitudes and Self-Reported Behaviors


28




29

Attitudes Changed through QPR Training Wyman et al., 2008

Improvements from Training and Time Effect Size

Null Low Med High

Knowledge of Warning Signs and QPR behaviors

0.46

Attitudes about Suicide Prevention

0.89

Self-Evaluation of Suicide Prevention Knowledge

1.06

Knowledge of Clinical Resources

0.99

Efficacy to Perform Gatekeeper Role

1.22

Reluctance to engage with suicidal students

0.29





5 10 15 20

57

911Control ChartSelf Efficacy for Gate

Time

Effic

acy

Benneyan et al., 2003


30




C. Roll-Out, Stepped Wedge, and Dynamic Wait-Listed Designs

31




Randomized Roll-Out Designs Wyman et al., Prev Sci 2015

Units are randomized to when they get the intervention (roll-out)Randomize by Place and Time -- CH Brown et al., (2009) Ann Rev PH

OPRE Design Meeting

Stepped Wedge Design – Brown CA & Lilford, BMC Med Research Meth, 2006

Control Periods

Intervention Periods

Rand

omize

d O

rder




Randomized Roll-Out Designs as a Categorical Name

• More functional and appealing to communities than jargon like Stepped-Wedge and Dynamic Wait-Listed Designs

• Misuse of the term Stepped-WedgeStandard Condition Single Active

Intervention

Standard Condition Randomize to Intervention A or B• There are many examples of roll-out designs

OPRE Design Meeting




Advantages of Roll-Out Designs From a Policy Maker or Community Perspective

Ethical IssuesNo one should be denied a potentially useful program, as long as it can be delivered with fidelity – Roll-out trials Traditional research designs like RCTs are unacceptable or foreign in some minority communities and for many policy makersAllows for programs to improve over time

Decision on which subregion gets the intervention first is fair.Go First: Immediate access to a potentially beneficial programGo Later: Program potentially improved through experience




Power increases with even a few subregions

OPRE Design Meeting




Advantages Even with a Few Units to Randomize

N = 2Mpowerment Young MSM (Kegeles AJPH

1987)

OPRE Design Meeting

EugeneSanta

Barbara Randomized Eugene Mpowerment Sustained

Santa Barbara Control Mpowerme

Baseline Year 1 Year 2




Rollout of Repeated Pairs of Randomized Communities:

Cumulative Trials (Brown et al., Ann Rev PH 2009)

OPRE Design Meeting

Pair 1

Tx

Ctl

Pair 2

Tx

Ctl

Pair K

Tx

Ctl. . .

Time

Pairwise Enrollment Roll-Out Design- Wyman et al., Prev Sci 2015




S4

S3

Single Selection Roll-Out of Randomly Selected Communities

OPRE Design Meeting

S1S2

Tx

CtlTx

Ctl Tx

Ctl

. . .

Time

Single Selection Roll-Out Design- Wyman et al., Prev Sci 2015




D. Scaling Out Designs for Delivering Interventions to Different Populations or through Different Delivery Systems

39



Scaling Up an Evidence-Based Intervention

Community and Macro Level Context

Delivery System EBI

Evidence-Based Determined by Effectiveness Trial

Scaling Up to Similar Contexts, Populations, and Delivery Systems 40


Scaling OutAarons et al., Implementation Science 2017

• Scaling-out is an approach to adapting and delivering EBIs Different service systems Different target populationsOr Both

• When can scaling-out be expected to produce effects similar to those found in previous studies?

Aarons, G. A., Sklar, M., Mustanski, B., & Benbow, N., & Brown, C. H., (2017). “Scaling-out” evidence-based interventions to newpopulations or new health care delivery systems. Implementation Science, 12(1), 111.

41




3. Three types of scaling-out

Type I: Population fixed, different delivery system• EBI is delivered toSame PopulationDifferent Delivery System• Ex: Familias Unidas: Family-

based intervention for Hispanic families with adolescents

• Parent Groups/Home Visits -> eHealth through primary care

Type II: Delivery system fixed, different population• EBI delivered to a Different population Same service system where it has previously been tested

Ex: PreP to adolescent MSM through local health department

Hurlburt, M., Aarons, G.A., Fettes, D., Willging, C., Gunderson, L., & Chaffin, M. (2014). Interagency Collaborative Team Model for Capacity Building to Scale-Up of Evidence-Based Practice: Structure and Process. Children and Youth Services Review, 39, 160-168.

42




Type III Scaling Out: Different Population and Different Delivery System

Example: Adolescent African American MSM not Connected to Health System

Highest Risk AND Least Likely to Use PrEP

• EBI Efficacy: PrEP can reduce HIV incidence by 90% from unprotected sex

• Tested Population: Adult Men who have Sex with Men (MSM), nearly all white, Delivery System: Full Service LGBT Services

• Different Population: Adolescent African American MSMCurrently Adolescents NOT recommended by CDC -- Off-Label

Different Different Delivery System:Home HIV Testing Services

43




Scaling Out Perspective on Implementation Researchand EPIS Implementation Framework

Local Knowledg

e

Generalized Knowledge

Scale Up

Scale Out Across Diverse Contexts

Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4-23.

44




Levels of evidence for scale-up and cale-out evaluations

• RE-AIM model

45

Effective?Adopt? Impleme

nt?Reach?Maintain

?

DELIVERY SYSTEM

Four levels of evidence

0 Minimal or No New Measures1 Inexpensive Proxy Measure2 Direct Empirical Evidence3 Full Randomized Hybrid Effectiveness-Implementation Trial




Level of Evidence

Implementation Fidelity

Intervention Fidelity

Reach & Exposure Adoption Sustainment Effect on Health

Outcome Potential Use

0: Minimal or no new

empirical evidence

Not measured

Training certification of facilitator and/or clinician

prior to new implementation

Numbers of individuals exposed

Attendance of organizational

representatives at trainings

Not measured Not measured

Demonstration program that

explicitly follows an intervention manual

1: Proxy empirical evidence

Leadership + staff self-efficacy to support EBI

Facilitator and/or clinician; self-assessment of fidelity

Attendance for behavioral

intervention; filled prescriptions

Formal acknowledgement by organizations of

adoption

Completion of yearly reports by

implementing agencies

Assessment of intermediate and/or

proximal health outcome

Inexpensive large-scale

implementation evaluation

2: Direct empirical evidence

Measurement of milestone attainment;

speed, quality + quantity of implementation

Independent assessment of fidelity

Ratings of quality of behavioral homework, medication adherence

Quality of staff training

Sustained number of staff + number of subjects exposed to

intervention w/ fidelity

Change in primary health outcome from baseline

Formal implementation

evaluation to establish evidence

base through mediationalmechanisms

3: Full randomized hybrid trial

Evaluate intervention vs. comparison on

primary outcome

Type II hybrid trial to directly establish full evidence base

46

Expanded RE-AIM and 4 Levels of Evidence Aarons et al., 2018




Assuming a conceptual theory of mediation holds for scaling-up

1. Cook’s (1991) Principle of Proximal Similarity– Correspondence between original trial and where/what we hope to

generalize to – Requires similarity on prototypical components

Scaling-Up

47




Level of Evidence

Implementation Fidelity

Intervention Fidelity

Reach & Exposure Adoption Sustainment Effect on Health

Outcome Potential Use

0: Minimal or no new

empirical evidence

Not measured

Training certification of facilitator and/or clinician

prior to new implementation

Numbers of individuals exposed

Attendance of organizational

representatives at trainings

Not measured Not measured

Demonstration program that

explicitly follows an intervention manual

1: Proxy empirical evidence

Leadership + staff self-efficacy to support EBI

Facilitator and/or clinician;self-assessment of fidelity

Attendance for behavioral

intervention;filled prescriptions

Formalacknowledgement by organizations of

adoption

Completion of yearly reports by

implementing agencies

Assessment of intermediate and/or

proximal health outcome

Inexpensive large-scale

implementationevaluation

2: Direct empiricalevidence

Measurement of milestone attainment;

speed, quality + quantity of implementation

Independent assessment of fidelity

Ratings of quality of behavioral homework, medication adherence

Quality of staff training

Sustained number of staff + number of subjects exposed to

intervention w/ fidelity

Change in primary health outcome from baseline

Formal implementation

evaluation to establish evidence

base through mediationalmechanisms

3: Full randomized hybrid trial

Evaluate intervention vs. comparison on

primary outcome

Type II hybrid trial to directly establishfull evidence base

Scaling-up to similar contexts

48




Type I Scaling Out: Deliver Familias Unidas through Primary Care (Family Medicine)

Delivered through eHealth, Molleda et al., J Ped Health Care 2017)

Effectiveness Impact on drug use 3 years later

Implementation• Feasibility and acceptability of eHealth in primary care• Addressed clinic flow, training of clinic personnel• Collect data on organizational context and climate

49

eHealth Familia Unidas delivered through Primary Care trialPrado PI




Community Context

Evaluation Guided by How a Program Should Work --Arrows represent Feedback

Implementation Agency

Intervention Agent

EBI

TargetPopulation

Reach, Equity

Proximal Outcome

Distal Health Outcome

Fidelity

Acceptance

50




Chambers et al., Imp Sci2013

“Sequential” Mediational Model

Roman Letters Represent Mean Values

Greek Letters Represent Regression Coefficients

From Aarons et al., Imp Sci 3017

51

Simplified Sequential Mediational Model




4. Cook’s (1991) Principle of Empirical Interpolation and ExtrapolationGeneralizing to an unsampled range of values on a particular variable

Relation between Adherence and log RR (Fonner)

10%

25%

100%

52




SYSTEMDomain

(mean, regression) Same Different

Scaling-UpType I Scaling-Out: Population -fixed,

different delivery system

POPU

LATI

ON

Sam

e

Implementation Fidelity (a , α )Intervention Fidelity ( b , β)Reach (c , γ )Adoption ( a , α )Sustainment ( a , α )Health Outcome (d, e , δ )

a = 1-2, α = 0b = 1-2, β = 0c = 1, γ = 0a = 1, α = 0a = 1, α = 0d, e = 0, δ = 0

a = 2, α = 0b = 1-2, β = 0c = 1, γ = 0a = 2, α = 0a = 2, α = 0d, =2 ,e =0, δ = 0

Type II Scaling-Out: Delivery System Fixed, different

populationType III Scaling-Out: Different

Population and Delivery System

Diffe

rent

Implementation Fidelity (a, α )Intervention Fidelity ( b , β)Reach (c , γ )Adoption ( a , α )Sustainment ( a , α )Health Outcome (d, e , δ )

a = 1,2, α = 0b = 1-2, β = 0c = 2, γ = 0a = 1, α = 0a = 1, α = 0d, e = 0, δ = 0

a = 2, α = 1-2b = 2, β = 1-2c = 2, γ = 0a = 1, α = 0a = 1, α = 0d, e = 2-3, δ = 0-3

Recommended Levels of Evidence for Type I Scaling-Out0=Minimal, 1=Proxy, 2=Direct, 3=Full Scale Hybrid Trial

53




Conditions where baseline parent child communication moderates the

mediation of Familias Unidas –Perrino et al., Prev Sci 2014

54

Poor parent-child communication leads to impact




Summary of Scaling Out

• With an even partial understanding of the underlying causal model, Scaling Out studies can often borrow strength from existing studies.

• Scaling-out can accelerate the rates of implementation of effective interventions

• Scaling-out can reduce burden and costs of implementing new practices

55




E. Predictors of Sustainment

• Palinkas et al., Imp Sci 2016 and Under Review• Barriers and Facilitators Implementation

Strategies

• Sustainment Measurement System

• Contact Larry Palinkas for survey

56




Conceptualizing and Measuring Sustainability of Prevention Programs and

InitiativesNIDA R34DA037516 (Palinkas)




Acknowledgements

58

Grants: NIDA R34DA037516 & NIDA P30DA027828




Acknowledgements

• Center Dir. Frances Harding, CSAP• Center for Substance Abuse Prevention (CSAP)

– Division of Community Programs– Division of State Programs

• Center for Mental Health Services (CMHS)– Division of Prevention, Traumatic Stress, and

Special Programs




Project Aims• Identify core components and their interrelationships

across time for sustainability of prevention programs and their support infrastructures.

• Design a measurement system for monitoring and providing feedback regarding sustainment.

• Pilot test the predictability of the Sustainment Measurement System (SMS) and the feasibility and acceptability of this system to evaluate and improve sustainment likelihood.




SAMHSA Ce-PIM Partnership• Four SAMHSA Prevention Programs:

– Strategic Prevention Framework State Initiative Program (SPF-SIG)

– Drug-Free Communities Support Program (DFC)/Sober Truth on Preventing Underage Drinking (STOP ACT)

– Garrett Lee Smith State and Tribal Youth Suicide Prevention (GLS)

– Implementing Evidence-Based Prevention Practices in Schools (PPS)




Methods• Data Collection

• Open-ended questions about experience with implementation and sustainment and identification of barriers and facilitators to sustainment.

• Free list exercise to elicit participant conceptions of what is meant by the term sustainment, what elements of their program they wish to see sustained, and what it will take to sustain those program elements.

• Checklist of domain elements from the Consolidated Framework for Implementation Research (CFIR: Damschroder et al., 2009).




Table 2.2 Percent of free list nominations of definition of sustainment, recommendations for what should be sustained,

and requirements to sustainment




The Sustainment Measurement System

SUSTAINMENT INDICATORS (n = 4)The project continues to operate as described in the original application for funding.

1 2 3 4 5 6 7

The project continues to deliver prevention services to its intended population

1 2 3 4 5 6 7

This project periodically measures the fidelity of the prevention services that are delivered.

1 2 3 4 5 6 7

The project continues to deliver prevention services that are evidence-based.

1 2 3 4 5 6 7




Steps

• CFIR Interviews• Survey• Grantee Reports• Visualization

65




• Form a lasting structural partnership between service and research, outside a contractual relationship

• Essential to Get Ahead of the Curve on Sustainment

• Develop a Sustainment Measurement System using Implementation Science to monitor and support sustainment of SAMHSA grantees while funded.

SAMHSA/Ce-PIM Mutual Self-Interest and Formation of a Strategic Plans “This partnership aims to improve overall population and individual level behavioral health through the use of research and utilization of effective practices/interventions/strategies [and] impact SAMHSA’s mission of preventing mental and substance use disorders.”Moving to Action on a Sustainment Measurement System• SAMHSA/Ce-PIM Partnership Communications Protocol : Cultural

Integration Stage• CFIR Interviews with 45 Grantees• Surveys completed by 64 Grantees• Text Mining of Grantee Reports and Visualization with SAMHSA Oversight

Service and Research Objectives

Meaning of Sustainment (Palinkas et al., under review)

At-A-Glance Information on Financial Sustainment Extracted from Grantee Reports

Results

Partnership Accomplishments

Conclusions

A Service/Research Partnership to Prevent Substance Misuse,

Background and Mutual Self-Interest

ALL GRANTSINDIVIDUAL GRANTS

• Service and Research Partnership follows: Kellam Partnership Model (2012) BOX A, Cultural Exchange Theory (Palinkas et al. 2011) FIG A, Strategic Prevention Framework (SPF, Harding et al., 2016) FIG B.

• Administrative Structure of Partnership: SAMHSA identified sustainment and prevention programs: STOP-Act, SPF-SIG/ Partnerships for Success Program, GLS Suicide Prevention Program, Implementing Evidence-based Prevention Practices in School, Minority AIDS Initiative & Prevention Navigator Programs. Research activities funded by P30 (Brown) and R34 Palinkas. Cross-Center Meetings (CSAP, CMHS, Ce-PIM) held every month.

• Activities: Mixed Methods Research: Consolidated Framework for Implementation (CFIR) Interviews with SAMHSA grantees and SAMHSA Project Officers, surveys of grantees, text mining of grantee reports

GLSN=11

PPSN=5

SPF-SIG

N=18

STOP-ActN=7

TotalN = 39

% % % % %What should be sustained

Program-specific activities 100.0 20.0 66.7 100.0 84.6***Training 36.4 60.0 27.8 57.1 41.0Coalitions/collaboration/networking 18.2 20.0 50.0 14.3 33.3Approach/strategies 0.0 40.0 38.9 0.0 23.1*Evaluation/monitoring 0.0 0.0 38.9 0.0 17.9**Integration with other programs 18.2 20.0 11.1 0.0 12.8Funding 0.0 0.0 27.8 0.0 12.8Media campaign 0.0 0.0 22.2 14.3 12.8Partnerships 0.0 0.0 22.2 0.0 10.3Positive outcomes 0.0 0.0 22.2 0.0 10.3Working groups 0.0 0.0 22.2 0.0 10.3

Percent of free list nominations of definition of sustainability, recommendations for what should be sustained, and requirements to sustainability

References

* p < 0.10, ** p < 0.05, *** p < 0.01

Garraza, Lucas Godoy, Christine Walrath, David B. Goldston, Hailey Reid, and Richard McKeon. "Effect of the Garrett Lee Smith memorial suicide prevention program on suicide attempts among youths." JAMA psychiatry 72, no. 11 (2015): 1143-1149.Kellam, Sheppard G. "Developing and maintaining partnerships as the foundation of implementation and implementation science: reflections over a half century." Administration and Policy in Mental Health and Mental Health Services Research 39.4 (2012): 317-320.Palinkas, Lawrence A., et al. "Mixed method designs in implementation research." Administration and Policy in Mental Health and Mental Health Services Research 38.1 (2011): 44-53.Harding, Frances M., et al. "Underage drinking: a review of trends and prevention strategies." American journal of preventive medicine 51.4 (2016): S148-S157.

• Followed Kellam’s Steps for Partnership formation and sustainment1. Analyze which organizations are required for support: SAMHSA lead

PH agency for BH2. Learn about the organization, work through trust with each

leader/organization:7 years of monthly meetings, NIH grant, joint presentations

3. Search for mutual self interests: Research to develop and then implement Sustainment Measurement System within SAMHSA

4. Form an operations group with oversight: Work conducted under aegis of SAMHSA

5. Carried out mutual self interest program for Sustainment• Achieved Cultural Integration 3rd Stage of Cultural Exchange Theory

• SAMHSA grantees expected to sustain their successful programs after SAMHSA funding ends.

• Sustainment plan required for every SAMHSA prevention program.

• Sustainment directly related to continued impact of SAMSA/CMHS’s Garrett Lee Smith (GLS) youth suicide program (Garraza et al. 2015).

• Sustainment is the least studied phase of implementation science and practice.

C Hendricks Brown1, Lawrence Palinkas2, Juan A. Villamar1, Sheppard Kellam3, Carlos Gallo1, Suzanne Spear4, Sapna Mendon2, Charles Reynolds5, Costella Green5, Charlotte Olson5, Audrey Adade5, James Wright5, Gail Ritchie5.1Northwestern Feinberg School of Medicine, 2Univ of Southern California School of Social Work, 3Bloomberg School of Public Health at Johns Hopkins University, 4California State University, 5Substance Abuse and Mental Health Services Administration.

• SAMHSA is not positioned to track grantees after their SAMHSA funding has ended.• Ce-PIM, funded by NIDA, is positioned to conduct research on sustainment

• Analysis of Needed Partners : SAMHSA is the primary behavioral health prevention agency; Ce-PIM single NIH Center devoted completely to Implementation Methodology.

• Goal: Develop a Sustainment Measurement System (SMS) that can be used inside SAMHS to monitor and predict long-term sustainment, and give feedback and provide assistance during program funding to improve likelihood of sustainment.

Service Delivery

Partners

New Partnershi

p Cultur

e

STAGE ICultural

Assessment

STAGE IICultural

Accommodation

STAGE IIICultural

Integration

DEVELOPMENTAL PROCESSFIG A

Communication

Collaboration Compromise

Research Partners

Service Delivery

Partners

Research Partners

Formation of Partnership: SAMHSA Administrator designated Ce-PIM as its research partner at NIDA Council

BOX A

FIG B

Mental Disorders and Suicide and HIV Transmission: The SAMHSA/Ce-PIM Partnership

NIDA P30DA027828 & R34DA037516




F. Responding to Questions• -Would like to learn more about the "Dynamic

Wait List Control" design (Pros, Cons, etc.)Extended Roll-Out Designs as general class

• -What are some important design considerations for hybrid efficacyeffectiveness + implementation design? Resources and examples would be wonderful.

Poduska et al IS 2009, Brown et al., 2009 : “sequential hybrid”

• Brown CH, Mason WA, Brown EC (2014). Translating the Intervention Approach into an Appropriate Research Design -- The Next Generation Designs for Effectiveness and Implementation Research. In Z Sloboda and H Petras (Eds.), Advances in Prevention Science: Defining Prevention Science, Springer Publishing.

67




More Questions• -What are some RCT options for observational studies with

smaller samples. AND How do you choose the number of sites/organizations.

Pairwise Rollout• -You have done quantitative work showing variation in

physician practice or in patient uptake of something. Then you did qualitative studies asking about barriers and facilitators-now how do you take all this and design an intervention for a grant application to actually address some of the barriers you found?

Palinkas sustainment measurement system

68




More Questions

• -Walk through an example of aligning methods to the research question. Folks seem to want to use a fancy method (eg. Most, SMART) before clarifying the question.

69




Community and Organizational are Much More Involved in Design Decisions and their

Ownership• Legal responsibility• Moral responsibility• Ethical responsibilityKey Areas• developing and maintaining partnerships with diverse stakeholders• Recognizing under-resourced communities or other vulnerable populations have

substantial historical trust concerns • leadership is within a partnered participatory research framework• methodological and design strategies that may apply when D&I research is

conducted from a participatory, stakeholder perspective





Challenges of Forming and Sustaining Research-Practice-Policy – and Community Partnerships

Kellam, Sheppard G. "Developing and maintaining partnerships as the foundation of implementation and implementation science: reflections over a half century." Administration and Policy in Mental Health and Mental Health Services Research 39.4 (2012): 317-320.Palinkas, Lawrence A., et al. "Mixed method designs in implementation research." Administration and Policy in Mental Health and Mental Health Services Research 38.1 (2011): 44-53.

Mutual Self-Interest – Alinksy, KellamCultural Exchange Theory - Palinkas

• Followed Kellam’s Steps for Partnership formation and sustainment1.Analyze which organizations are required for

support: SAMHSA lead PH agency for BH2.Learn about the organization, work through

trust with each leader/organization:7 years of monthly meetings, NIH grant, joint presentations

3.Search for mutual self interests: Research to develop and then implement Sustainment Measurement System within SAMHSA

4.Form an operations group with oversight: Work conducted under aegis of SAMHSA

5.Carried out mutual self interest program for Sustainment




Faculty Position at Northwestern in Implementation Science

http://cepim.northwestern.edu/facultyposition

72



http://cepim.northwestern.edu/facultyposition


73



c hendricks brown...funding contexts, fidelity monitoring and feedback systems used to deliver a...

Documents