![Page 1: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/1.jpg)
Dr. Brian Rush
Centre for Addiction and Mental Health, Toronto, OntarioandDepartments of Psychiatry and School of Public Health, University of Toronto
EVALUATION AND QUALITY IMPROVEMENT IN SUBSTANCE USE TREATMENT AND SUPPORT: GOING FROM “SIMPLE” TO “COMPLEX” INTERVENTIONS.
![Page 2: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/2.jpg)
SOME BASICS ABOUT EVALUATION
![Page 3: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/3.jpg)
WHAT DO WE MEAN BY “EVALUATION”?
Evaluation involves the systematic collection of feedback about a program or intervention. The feedback is used in making decisions.
![Page 4: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/4.jpg)
need assessment
participant satisfaction
outcome evaluation
economic evaluation
Different Types of Evaluation
Process
Need
Satisfaction
Outcome
Economic
![Page 5: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/5.jpg)
WHY DO WE DO IT?
Accountability (service or system) - promises we make with public funds
Quality Improvement (service or system) – using feedback to improve what we do
Research/testing theory(service or system) – finding out things that others will be interested in and sharing it with them
![Page 6: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/6.jpg)
Precontemplation – never think about it; actively avoid; only if pressured; don’t like to ask too many questions
Contemplation – do the minimum stats; ok but how do I start? Interested but afraid of the answers; went to a workshop once but no time to do it; bought a book about evaluation
Action – have a logic model and evaluation plan for one of my services; been talking to a researcher to help out; found a good way to start and made a proposal to the administration
Maintenance – wake up every morning thinking about how to evaluate my service; have a plan for each service; routine
DOES ANY ONE CARE ABOUT IT?
![Page 7: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/7.jpg)
need assessment
participant satisfaction
outcome evaluation
economic evaluation
Establishing a Healthy Culture for Evaluation
The Importance of a Healthy Evaluation Culture
Process
Need
Satisfaction
Outcome
Economic
![Page 8: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/8.jpg)
Treatment modality (CBT, motivational interviewing, naltrexone, methadone)
Service/program level (e.g., inpatient unit, day or evening drop-in service; family group, opiate substitution service)
Organization level (e.g., hospital, NGO, therapeutic community)
System/network level (e.g., substance use and mental health services, primary and urgent care, hospital and community networks, justice, education)
EVALUATION CAN BE DONE AT DIFFERENT LEVELS
![Page 9: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/9.jpg)
PURPOSES AND LEVELS OF EVALUATION
Accountability/ Monitoring
Quality Improvement
Research/ Development
Service/ Intervention
Program/ Organization
System/Network
![Page 10: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/10.jpg)
THE IMPORTANCE OF “EVIDENCE”
![Page 11: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/11.jpg)
GOING FROM THEORY TO PRACTICE IN THE COMMUNITY
![Page 12: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/12.jpg)
THE IMPORTANCE OF “EVIDENCE” IN SUBSTANCE USE TREATMENT AND
SUPPORT
Increase chances of really helping people (and at reasonable cost to society)
Research evidence does play a critical role in treated safely and respectfully
Research evidence is needed to ensure one’s “approach” gains the acceptance of professionals, funders and the community as a whole
Current challenge – there are many kinds of research and many ways of “knowing” something
![Page 13: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/13.jpg)
THE “EVIDENCE PYRAMID” BEHIND CURRENT MEDICAL
PRACTICE
Meta-Analysis
Systematic Reviews
RCT”s
Cohort Studies
Case Control Studies
Case Reports
Animal Research
![Page 14: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/14.jpg)
ADAPTATION OF THE PYRAMID FOR
QUALITATIVE EVIDENCE
I - Generalizable conceptual studies
II- Descriptive Studies
III- Single Case Studies
![Page 15: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/15.jpg)
BUT THIS IS THE PYRAMID THAT”RULES”
Meta-Analysis
Systematic Reviews
RCT”s
Cohort Studies
Case Control Studies
Case Reports
Animal Research
![Page 16: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/16.jpg)
EVOLVING MODELS OF RESEARCH EVIDENCE
More recognition of the limitations of RCT’s Who is actually represented in the research populations?
Does this research design really work for highly complex and system-level interventions?
More recognition now of “practice-based evidence” to complement evidence-based practice
More recognition of the challenges with this model in working with indigenous populations and studies of traditional medicine - a different view on the nature of “knowing”
![Page 17: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/17.jpg)
CHALLENGE(S) MEASURING “SYSTEMS CHANGE”
![Page 18: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/18.jpg)
CHALLENGE(S) MEASURING “SYSTEMS CHANGE”, “INTEGRATION”,
“COLLABORATION”
in most intervention research OUTCOME = intervention MINUS “noise” internal validity trumps external validity
in systems or complexity–based research OUTCOME = intervention PLUS context nothing can be understood outside of the context in which it exists
![Page 19: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/19.jpg)
CHALLENGE #1
The goals or anticipated benefits of a systems change are not always shared or explicit among the key actors
![Page 20: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/20.jpg)
HE SAYS:
“THIS IS THE PART I ALWAYS HATE!”
![Page 21: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/21.jpg)
Substance Abuse
Treatment and Support
System
Mental Health
Treatment and Support
System
A process is underway in many countries to improve this problem
![Page 22: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/22.jpg)
The personal and clinicalexperience of co-occurring disorders
![Page 23: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/23.jpg)
Mental Disorders
Overlap in Canadian General Population
Substance Use
Disorders15-20%
![Page 24: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/24.jpg)
Mental Disorders
Substance Use Services (Ontario)
Substance Use
Disorders70- 80%
![Page 25: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/25.jpg)
Mental Disorders
Overall Mental Health System (Ontario)
Substance Use
Disorders15-20%
![Page 26: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/26.jpg)
Mental Disorders
Young males in inpatient MH settings
Substance Use
Disorders55%
![Page 27: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/27.jpg)
Mental Disorders
To help keep things in perspective…
Where are we with the justice system?
Substance Use
Disorders80-90%
![Page 28: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/28.jpg)
WITH RESPECT TO THE INTEGRATION OF MENTAL HEALTH AND ADDICTIONS IN CANADA AND
ELSEWHERE …...
The train has clearly left the station!
![Page 29: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/29.jpg)
ANTICIPATED BENEFITS OF “INTEGRATION”
To reduce stigma and discriminationTo improve access to servicesTo get better retentionTo improve continuity of careTo improve outcomes
??? ….. improved prevention of SU (e.g. addressing mental health and determinants of MH at young age)
![Page 30: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/30.jpg)
DOES THE INTEGRATION OF HEALTH CARE SYSTEMS HAVE PREVENTION
IMPLICATIONS?
Prevention
Resources
Treatment
Resources
![Page 31: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/31.jpg)
![Page 32: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/32.jpg)
OTHER POSSIBLE GOALS???
Increased efficiency/less managementMore competitive positioning for resourcesInter-disciplinary issues of world
view/power/control (e.g., medical – non-medical)
Leveling the playing field (wages, credentialing, workforce mobility)
Expectations and perceived benefits are essentially “values based”
![Page 33: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/33.jpg)
CHALLENGE # 2
“Systems Integration” does not mean the same thing to the various actors
![Page 34: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/34.jpg)
MANY FIND IT HELPFUL TO CONSIDER “INTEGRATION” ALONG A CONTINUUM…
Full segregation
Co-operation
![Page 35: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/35.jpg)
SO MANY WAYS TO CONSIDER “INTEGRATION”
Services vs Systems integration
Functional vs Horizontal
Normative integration (values based)
![Page 36: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/36.jpg)
Information continuity: the use of information on past events and personal circumstances to make current care appropriate for each individual
Management continuity: a consistent and coherent approach to the management of a health condition that is responsive to a person’s changing needs
Relational continuity: an ongoing therapeutic relationship between a person and one or more providers
DIFFERENT ASPECTS OF INTEGRATION OR “CONTINUITY OF CARE”
![Page 37: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/37.jpg)
SO MANY WAYS TO PUT “INTEGRATION” INTO PRACTICE
integrated clinical teams
centralized access and intake
shared electronic records
linkage managers/case management
the devil is in the details
every situation is unique
![Page 38: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/38.jpg)
CHALLENGE # 3
“It is not clear how we should go about achieving “Systems Integration”
![Page 39: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/39.jpg)
THE MECHANISMS/PROCESSES TO ACHIEVE IMPROVED “INTEGRATION” ARE NOT CLEAR OR EVIDENCE-BASED
May have a model in mind and its about “best practice” implementation – implementation science calls for active implementation supports
May be driven by a defined quality improvement process (Plan, Do, Study, Act) or change management process
Top down (Big world) vs bottom up (small world)Muddling along
Whatever the initial vision or model one typically works with the existing services and sub-systems as starting point(s) – leads to a basic strategy of “incrementalism” and ”contextual adaptation”
![Page 40: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/40.jpg)
ESSENTIALLY A MIRACLE OCCURS….
![Page 41: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/41.jpg)
THE MECHANISMS/PROCESSES TO ACHIEVE IMPROVED “INTEGRATION” ARE NOT CLEAR OR EVIDENCE-BASED
Whatever the initial vision or model, one typically works with the existing services and sub-systems as starting point(s) – leads to a basic process of “incrementalism” and ”contextual adaptation”
![Page 42: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/42.jpg)
CHALLENGE # 4
“Health outcomes are often distal not proximal to the change efforts”
![Page 43: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/43.jpg)
INTERMEDIARY OUTCOMES
(Process)Outcomes
- Access- Continuity of care- Information moves
Integration activity or
intervention
(Health)Outcomes
- SU- MH- QOL
![Page 44: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/44.jpg)
CHALLENGE # 5
“Systems are never static – always responding to external context – can you really isolate the intervention from its dynamic context”
![Page 45: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/45.jpg)
SYSTEMS ARE ALWAYS RESPONDING TO THE
ENVIRONMENT (TO SURVIVE)
![Page 46: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/46.jpg)
SYSTEMS ARE ALWAYS RESPONDING TO THE
ENVIRONMENT (TO SURVIVE)
Aging population
Resource
AvailabilityNew needs
Political
reform
Health Determinants
![Page 47: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/47.jpg)
CHALLENGE # 6
“Systems are never what they appear to be”
![Page 48: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/48.jpg)
ARCHEOLOGICAL SYSTEMS AND ORGANIZATIONS
SecretsMemories
Myths
Old warriorsLies Truths
![Page 49: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/49.jpg)
NEED TO DIG DEEP TO UNDERSTAND
SecretsMemories
Myths
Old warriorsLies Truths
![Page 50: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/50.jpg)
CHALLENGE # 7
“Implementation of any systems change is never smooth – rarely do you end up with what you set out to achieve”
![Page 51: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/51.jpg)
![Page 52: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/52.jpg)
CHALLENGE # 8
“Routine information systems will always present challenges for evaluation of system change”
![Page 53: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/53.jpg)
CHALLENGES WITH ROUTINE INFORMATION SYSTEMS
health data rarely collected for research and evaluation purposes
challenges with data linkage
mental health and addictions information can be lost
regionalization challenges commonality in data elements and quality
![Page 54: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/54.jpg)
IS THERE ANY ROOM FOR OPTIMISM IN THE EVALUATION OF
INTEGRATION?
![Page 55: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/55.jpg)
EMERGENT REALIST EVALUATION MODEL
Intervention + context = outcome
Systems analysis and complexity science
Mixed methods - qualitative and quantitative
Realist research synthesis and contribution analysis
Working on it and sharing the experiences
![Page 56: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/56.jpg)
Thank you and good luck in your personal and professional work!
![Page 57: Integration of substance abuse treatment: what is it and how can it be evaluated?](https://reader035.vdocuments.net/reader035/viewer/2022070302/5485403eb47959ce0c8b4db0/html5/thumbnails/57.jpg)
BUT WHAT DO WE DO?