making the most of what we have: the role of evidence, the importance of context and working...

Making the most of what we have:The role of evidence, the importance of context and working together to achieve high quality implementation to improve outcomes for children and families.

Robyn Mildon, PhD

Director of Knowledge Exchange and Implementation

2

Cradle to Kinder Evaluation Proposal

Parenting Research Centre

- non-government, non-profit, independent research, program development, implementation specialist and evaluation organisation

-support the use of evidence in practice and participatory program development and evaluation

- use evidence from the science and practice of implementation to support effective and full implementation of quality services for children and families

Where are we at..

The ultimate goal of services and supports in child welfare service systems is to decrease the prevalence of child abuse and neglect and increase child permanency, safety and wellbeing.

Over the two decades, thinking about how to accomplish this has involved a number of strategies..

Evidence-informed programs consist of collections of practices that are performed within known parameters (philosophy, values, service delivery structure, and treatment components) and are put together in the form of a structured manual containing details of procedures and tools (Fixsen et al., 2005).

Evidence-informed practices are effective skills, techniques, and strategies that can be used by an individual practitioner. Such practices describe core intervention components that have been shown to reliably produce desirable effects and can be used individually or in combination to form more complex procedures or programs (Embry, 2004).

Evidence to practice gapDespite these efforts, widespread implementation with sustainment has been difficult to achieve across human services

What is implemented often disappears with time and staff turnover

Moving effective interventions from development and research settings to the practice setting, with fidelity and good effect, involves far more then making the interventions available

(Fixsen et al, 2005; Kauffman Foundation Best Practices Project; Rogers 1995; )

Only 9% of those who completed all

3 phases of training followed through to

implementation.

Alone, these will not do. Identification and cataloguing of “evidence-based” practices and programs.

Making laws and policy directives

Providing funding

One way dissemination of information (tip sheets, websites, lectures like this one...)

Implementation without changing supporting roles and functions does not work

Training alone

(Embry & Biglan, 2008; Fixsen et al., 2005 )

Children and families cannot benefit from something they do not receive

10

The get the most out of what we have.. The role of evidence (the what)

Evidence-informed interventions and/or practices

High quality implementation (the how):

Effective implementation frameworks (e.g. strategies to change and maintain behaviour of practitioners and create hospitable organisational systems)

The importance of context, within which the programs and practices will be delivered (community, organisations, political context)

Effective interventions and practices+

Effective implementation=

Positive outcomes

Necessary conditionsMotivation: intent to change

- perceived advantages of innovation + culture + climate + resources + leadership + structure

Organisational capacity

- context, culture, current infrastructure, organisational processes (sufficient staffing, leadership, well established network)

Innovation-specific capacity

- Knowledge, skills, abilities needed for the innovation; program champion; specific implementation climate supportScaccia, Cook, Lamont, Wandersman, Castellow, Katz & Beidas, in press

Implementation

...the planned and intentional use of strategies to put into practice effective innovations within real-world settings (Fixsen et al. 2005; Mitchell, 2011).

Implementation is a process, not an event, and should be distinguished from adoption, which is defined as the formal decision to use an effective practice/s (Mitchell, 2011).

Implementation Science

Is the study of methods to promote the integration of research findings and evidence into health, education and human services

15

What we choose to do matters

The What..

Human services field is moving toward providing services that are demonstrably effective as one way to reduce the prevalence of most common problems and increase wellbeing.

In the main, efforts to do this have focused on identifying and disseminating evidence-informed programs.

What next..

Increasing the adoption and implementation of effective programs and practice

Start with outcomes

Pay attention to the “implementability” of EIPs

Considering programs AND a common elements approach

Considering core elements and adaptable characteristics

Starting with outcomesOutcome: describes how the target population (who) might change based upon successful completion of your strategies.

Improved parent and child safety

Improved parental and family functioning

Promote a secure, predictable and dependable relationship between the parents/children

Increase child and parent self-efficacy

Promote responsive and warm parenting

Improved parenting coping skills

Improve parent and child problem-solving skills

Increase positive communication

Improved child and adult social connections/resources

Identify and assess best fit evidence-informed programs/practices

•Clarify needs of children and families the agency support.

•Clarify desired aims and outcomes of the program or service.

•Collect and summarise the evidence base.

•Map practice.

•Select and adapt knowledge to local context.

•Select, tailor and/or design program or Practice Model.

Programs and practices

Where we are looking to achieve change outcomes with a particular target group, and a manualised EIP has been shown to achieve this, we should strive to effectively implement that EIP with high fidelity.

This will often mean changing the service setting in which the program will be implemented (staff skill, supervision, organisational systems and structures, service frameworks)

Challenges with EIPs for service system• challenges with respect to implementation in agencies serving children and families with complex and multiple needs.

• EIPs can be too constraining of practitioners’ practice as a result of rigid manualization and structure (Lyon et al, under review); • there are difficulties in implementing an EIPs widely with fidelity and maintaining effectiveness (Embry & Biglan, 2008);

• in the real-world there are often difficulties in accessing adequate training in a variety of ESIs that have been developed (Kazdin & Blase, 2011)

• many existing interventions that are treated as EIPs may also have limited effectiveness, modest effect sizes, issues with scalability, limited generalisation, and difficulties with maintaining or sustaining the program without adaptation in child welfare service settings (Embry & Biglan, 2008).

A common elements approach

Common elements refer to the individual treatment practices that reflect specific content (e.g., psycho-education, exposure, and rewards) that comprise an intervention

Treatment elements are selected to match particular child and family characteristics, allowing for practitioners to establish therapeutic alliances, utilize clinical judgment, and still follow evidence-based practice protocols.

Practice elements are defined by their content, not by duration, periodicity or location within a manual.

(Chorpita, Daleiden, & Weisz, 2005a; Chorpita, Daleiden, & Weisz, 2005b)

Common elements - recognizes the benefits to be gained by understanding more about current practice when trying to build effective services across contexts

- offers a genuine possibility in terms of facilitating more effective adoption and implementation

- conductive to a high level of adaptability to context and reciprocal adaptation between intervention and context, and these may serve to decrease the substantial barriers to adopting EIP’s that exist in provider attitudes, and characteristics of client populations

Fidelity and Adaptation

One of the most critical challenges in implementation of EIPs - tension between adaptation (i.e., flexibility) and attaining fidelity

Adaptation.

“To make fit (as for a specific or new use or situation) often by modification”

(Aarons et al 2012)

27

Fidelity and AdaptationExposure : amount of an intervention that is offered to the participants in relation to the amount prescribed in the validated intervention model (the number of sessions or hours of programmed activity offered).

Adherence: extent to which the techniques/practices were delivered according to the program developer’s specifications for content.

Quality of delivery: pertains to practitioners/manager performance on dimensions that are thought to enhance delivery of the intervention (e.g., enthusiasm, style, ability to facilitate client participation, etc.).

Implementation matters

The How

“Evidence” on effectiveness helps you select what to implement for whom

“Evidence” on these outcomes does not help you implement the program or practice

Fixsen & Blase (2008)

Implementation Matters (from Fixsen et al., 2005)

Effective Not Effective

Effective

Not Effective

Implementation: The How

Inte

rven

tion

– Th

e W

hat

(Institute of Medicine, 2000; 2001; 2009; New Freedom Commission on Mental Health, 2003; National Commission on Excellence in Education,1983; Department of Health and Human Services, 1999)

Inconsistent; not sustainable; poor outcomes

Poor outcomes Poor outcomes; sometimes harmful

Actual benefits

Implementation matters Durlak & DuPre (2008) review of over 500 implementation studies in the field of prevention and health promotion programs for children and youth

found that the magnitude of mean effect sizes were at least 2 to 3 times higher when programs were implemented well with few or no problems in the implementation.

full implementation of programs was associated with better outcomes, particularly when fidelity and dosage were used to measure levels of implementation.

Implementation matters

“. . . in some analyses, the quality with which the intervention is implemented has been as strongly related to recidivism effects as the type of program, so much so that a well-implemented intervention of an inherently less efficacious type can outperform a more efficacious one that is poorly implemented.”

Lipsey et al (2010)

descriptions of the implementation process which include key attributes, facilitators, and challenges related to implementation

provide an overview of practices that guide the implementation process and provide guidance to researchers and practitioners by describing specific steps to include in the planning and execution of implementation efforts

Frameworks for implementation

Letting it happen– Diffusion; networking; communication

Helping it happen– Dissemination; manuals; websites

Making it happen (Implementation Science) – Purposeful and proactive use of implementation practice and science

Based on Hall & Hord (1987); Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou (2004); Fixsen, Blase, Duda, Naoom, & Van Dyke (2010)

38

Quality Implementation Framework (Meyers, Durlak and Wandersman, in press)

National Implementation Research Network (Fixsen, Naoom, Blase, Friedman & Wallace, 2005)

Getting to Outcomes Framework (Wandersman, Imm, Chinman, & Kaffarian, 2000)

Quality Implementation Framework

Phase One : Initial Considerations regarding the host setting (elements)

1. Conducting a Needs and Resources Assessment

2. Conducting a Fit Assessment

3. Conducting a Capacity/ Readiness Assessment

4. Possibility for Adaptation

5. Obtaining Explicit Buy-in from Critical Stakeholders and Fostering a Supportive Organizational Climate

6. Building General/Organisational Capacity

7. Staff recruitment/ maintenance

8. Effective Pre-innovation Staff Training

Phase Two : Creating a Structure for Implementation

9. Creating Implementation Teams:

10. Developing an implementation plan:

Phase three : Ongoing Structure Once Implementation Begins

11. Technical assistance/ Coaching/Supervision:

12. Process evaluation

13. Supportive feedback mechanism

Phase Four : Improving Futures Applications

14. Learning from experience

Knowledge to Implementation Cycle (based on Fixsen et al 2005)

© Fixsen & Blase, 2008

Performance Assessment

Coaching

Training

Staff selection

Systems Intervention

Facilitative Administration

Decision Support Data System

Technical

Integrated & Compensatory

Com

pete

ncy D

river

s

Com

pete

ncy D

river

s Organization Drivers

Organization Drivers

LeadershipLeadership

Improved outcomes for children and families

Adaptive

Effective implementation (fidelity)

Implementation drivers

Competency Drivers

•develop, improve, and sustain ability to implement an intervention with fidelity and benefits to consumers. Drivers include: Selection, Training, Coaching, and Performance Assessment

Organizational Drivers

•create and sustain hospitable organizational and systems environments for effective services. Drivers include: Decision Support Data System, Facilitative Administration, and Systems Intervention

Leadership Drivers

•methods to manage technical problems

TOWARD AN EVIDENCE INFORMED SYSTEM FOR INNOVATION SUPPORT

From Barbee et al 2011.

To AchieveDesired

Outcomes

Training +

QI/QA +

Tools +

TA +

=Current Level of Capacity

+

GTO Steps: (1) Needs & Resources; (2) Goals & Desired Outcomes; (3) Science-based practices; (4) Fit; (5) Capacity ; (6) Plan; (7) Implementation & Process Evaluation; (8) Outcome evaluation; (9) Continuous Quality Improvement; and (10) Sustainability

Actual Outcomes Achieved

Context Matters

Implementation is affected by organisational context

Relationship between organisational support for EBP, attitudes towards EBP and use of EBPs in practice.

Findings:

•gap between public and private sector organisations regarding innovation and implementation.

•private agencies provided greater support for EBP implementation

•staff working for private agencies reported more positive attitudes toward adopting EBPs.

•organisational support was significantly positively associated with attitudes toward EBP and EBP use in practice.

Aarons et al. (2009)

Stages of Implementation

50

52

Population of concern Intervention Strategies (WHAT)

Intervention Outcomes

Children and families where neglect is present.

IFSS workers skilfully implement the evidence-based practices within the IFSS Practice Model

Reduction in child neglect according to the ‘Child Neglect Index’

IFSS Logic Model

The logic model continued

Population of concern

Intervention Strategies (WHAT) Intervention Outcomes

Children and families where neglect is present.

IFSS workers skilfully implement the IFSS Practice Model


Populations Implementation Strategies (HOW) Implementation Outcomes

IFSS workers Team Leaders carefully select staff and arrange for skilful, timely training; coaching; performance assessments; and use data in supportive administrative environments

IFSS workers competently use the IFSS Practice Model to improve child neglect and strengthen families

53

The logic model continued


Intervention Strategies (WHAT) Intervention Outcomes

Families where child neglect is present.






Supervisors/Team Leaders

Agency managers engage with technical assistance providers; plan for release time for training, coaching, and ongoing consultation services; installation of data systems to monitor fidelity; review and revise agency policies and procedures

Team Leaders ensure access to skilful, timely training; coaching; performance assessment and supportive administrative environments for workers implementing the IFSS Practice Model

54

The logic model continuedPopulation of

concernIntervention Strategies (WHAT) Intervention Outcomes

Families where child neglect is present.






Supervisors /Team Leaders


Team Leaders ensure access to skilful, timely training; coaching; performance assessment and supportive administrative environments for workers implementing the IFSS Practice Model

Provider Agency Managers

FaHCSIA funds and supports an Implementation Support Service for IFSS: formal mechanisms and structures created to enhance policy-practice feedback loops (CIT): streamline reporting requirements


55


Intervention, Strategies and Measures (WHAT)

Intervention Outcomes

Children and families.



Populations Practice, Organisational and Systems Change Strategies (HOW)

Implementation Related Outcomes

IFSS Workers Team Leaders carefully select staff and provide or arrange for skilful, timely training; coaching; performance assessments; and use data in supportive administrative environments


Team Leaders Provider agency managers facilitate agreements with IFSS trainers, quality consultants, and technical assistance providers; plan for release time for training, coaching, and ongoing consultation services; manage the installation of data systems to monitor fidelity and Service outcomes.

Team Leaders ensure access to skilful, timely training; coaching; performance assessment and supportive administrative environments for caseworkers implementing the IFSS Practice Model


FaHCSIA develops a common mission for supporting implementation of IFSS: formal mechanisms and structures created to enhance policy-practice feedback loops; streamline reporting requirements; facilitating collaborative relationships to support system infrastructure.

Agency leaders ensure access to skilful, timely training, and implementation supports to Team Leaders: creation of hospitable organisations for the IFSS Practice Model 56PR

C IM

PLEM

ENTA

TIO

N S

UPP

ORT

TEA

M

The logic model continuedPopulation of

concernIntervention, Strategies and Measures (WHAT) Intervention Outcomes

Children and families.



Populations Practice, Organisational and Systems Change Strategies (HOW)

Implementation Related Outcomes

IFSS Workers Supervisors carefully select staff and arrange for skilful, timely training; coaching; performance assessments; and use data in supportive administrative environments


Team Leaders Provider agency managers facilitate agreements with trainers, quality consultants, and technical assistance providers; plan for release time for training, coaching, and ongoing consultation services; manage the installation of data systems to monitor fidelity.

Team Leaders ensure access to skilful, timely training; coaching; performance assessment and supportive administrative environments for caseworkers implementing the IFSS Practice Model


PRC Implementation Team develops a specific training program; provides coaches and develops measures and a data capture system so that provider agencies have improved knowledge, skills and resources in the delivery of the IFSS Practice Model

Agency leaders ensure access to skilful, timely training, and implementation supports to Team Leaders: and the creation of hospitable organisations for the IFSS Practice Model

57

Contact UsLevel 5, 232 Victoria Parade East Melbourne, Victoria, Australia 3002

E: [email protected]: +61 3 8660 3500F: +61 3 8660 3599

making the most of what we have: the role of evidence, the importance of context and working...

Documents