cmha hrb vces eqip presentation
TRANSCRIPT
VCES Data: The Virtual Reality at CMHA Halton Region Branch
E-QIP QI Innovations ConferenceApril 13, 2021
Ed Castro, CLSSBB, Director, Planning and ServicesMegan McNeil, RSSW, Manager of Case Management ServicesAngela Gaetano, MSW, RSW, Clinical Planner and Case Manager
VCES Response data accurate as of: Wednesday, December 09, 2020Client response rate estimated based on historical EMHWare “Program History” data from October 2020, based on reports pulled on: 13 January 2021
Objective:To share some learnings from the process of designing and implementing our recent quality improvement pilot project.
Agenda:
1. Overview of the VCES Pilot Project2. Learnings:
A. Introductions & The Voice of Lived Experience: Language Matters!B. Basic Survey Information
I. Always plan for “n”II. Importance of Strategic Communication
C. Demographics: Plan Output Before InputD. Technology:
I. Expanding Helpful Input/OutputsII. Notice Who & What You’re Missing
E. Client Experiences: What if your results are all positive?F. Preliminary Statistical Analysis: Planning for Statistical Analysis
3. Next Steps
Overview of VCES Pilot ProjectProblem Statement:Lack of data measuring client services satisfaction with the pivot to virtual supports during the COVID-19 pandemic
Why a pilot?:• To test design of data collection system & identify quality improvement activities in
it prior to rolling it out agency wide.
• Began with 3 non-crisis Case Management programs providing video supports.
Why CAMH’s Virtual Client Experience Survey (VCES)?• Brief literature review showed few easily accessed tools specifically targeted to our
community programs & client populations.
• CAMH trusted & widely known; potential broader use of VCES in system may allow for knowledge sharing with multiple system partners.
Virtual Client Experience Survey (VCES)• CAMH; Drs. Allison Crawford, MD, PhD, FRCPC & Eva Serhal, MBA PhD• Self Report; Full and Brief Versions• Includes items from the Ontario Perception of Care (OPOC)• Is free to download and use from CAMH: https://edc.camhx.ca/redcap/surveys/?s=7CRKNYT7FY
Full version includes items on: Client satisfaction with virtual mental health and addictions supports.
Some demographics
Presence of some common health and accessibility factors, including:• Chronic Illness• Mental health and/or substance use challenges• Ability differences (cognitive, physical, & sensory)
Technology, including:• What tech is used to access appointments• Overall comfort level with technology
Prep (PLAN)
Administer(DO)
Data Collection(STUDY)
Analysis (STUDY)
Report Back &Identification of QI
Activities(ACT)
Tool Identification via Literature Review
Input from Lived Experience
(Peer Mentor Meeting23 September 2020)
Administration Process Planning
Staff Training Development
Training Staff in programsidentified for Pilot.
Creation of online surveyinput sustem anonymous
(SurveyMonkey)
Staff to send link to allactive clients receivingvideo supports duringthe month of October:
Staff to communicatevalue & deadline to client
(script available)
Survey monkey collectsresults as surveys are
completed
Client informs staff they have completed by end
of October(honor system)
Staff emails client namesto Clinical Planner for
draw by draw deadline:06 November 2020
Participants who wish tobe are entered in draw.
Manager draws numbersfor 5 $25 Walmart GiftCards in a November
team meeting
Data Analyst aggregatessurvey results (outcomedata) by Nov 16, 2020
Meet with staff to reportoutcome data & identified
QI goalsTimeline Target: end of
November
Project Team to meet on24 Nov 2020 to analyseoutcome data and start
Identifying QI goals & nextsteps including action
items and ideas re: potential project teams
Manager to remind staffat team meetings in October regularly re:
providing link to surveyto clients
VCESImplementation Pilot PDSA Cycle
Fall 2020
CMHA Halton Region Branch
Pilot now complete; Project team begins
planning for completion of identified QI changes to
survey to a second (fuller) implementation of survey.
Proceed with Next Project: Fuller implementation of
survey.
Report back to Board re:outcome data & identified
QI goals
We are here: Pilot Project Now Complete!
Proceed with NextProject:
Fuller implementation of
survey with QI changes made.
“I appreciated the opportunity to contribute to the conversation through the lens of ‘What is important to the peer/client’.
Framing the questions and using Language which reflected ‘we are listening’ and peers/clients input and experience is very important to the organization.”
-CMHA Halton Peer Mentor
• Peer Mentors & voice of lived experience
• Person-centered language to communicate: • Importance of your voice• Value of your input• Respect & Person-centered values• How information will be used• Impact of information received &
value for service users
• Mitigating the impact of jargon & clinical language by adding an introduction
• Collaboratively wrote an intro for the survey monkey & optional introduction script forCase Managers.
• Honorarium
Learning: Language Matters!Introductions & The Voice of Lived Experiences
Basic Survey InformationTotal Responses Estimated Response
Rate*Completion Rate Typical Time Spent
Completing
33 18.76* 100% 9 min 34 sec
Challenge: • Project team was focused on so many moving
parts, we did not work in a process for consistently tracking how many invitations to the survey were sent out (the “n” in research terms)
• This made it difficult to determine an accurate response rate.
Solution: • To address this gap in information, a “proxy”
estimate was calculated based on known caseload data from the electronic records database (reported above*).
Important Learning for QI: Always get the “n”Moving forward, this will be a required data input for future launches of the survey & a process for Case Managers to record and report the number of surveys handed out is
being developed for our wider roll out.
“Missing the Forest for the Trees”
Response Timeline - Communication with Case Management (CM) Team
DATES
# O
F R
ESPO
NSE
S
CM Team Meeting
CM Team Meeting & Clinical Planner spoke re: VCES
Email reminder sent to CM Team
Intended stop of pilot project.
LEGEND:
Learning: Importance of Strategic Communication
Demographics• Full VCES collected basic demographic information on:
• Gender• Age• Geography (via postal code)• Racial/Ethnic Group• Number of years living in Canada• Presence of:
• Chronic Physical Health Conditions• Mental Health Challenges• Substance Use Challenges• Ability Differences (cognitive, physical, and sensory)
• Notably does not include income or sexual orientation
• QI Activity: May add additional demographic questions in upcoming second implementation
0123456789
10R1A 2B3V1C 9K7N8K 0M7N2G 1L8V7D 8L9Q2C 9H6N3K 0L8R1A 0M7N2G 1L8L8L 6G7R1A 2B3
V1C 9K7N8K 0M7N2G 1L8V7D 8L9Q2C 9H6N3K 0L8R1A 0M7N2G 1L8L8L 6G1R1A 2B3V1C 9K7
N8K 0M7N2G 1L8V7D 8L9Q2C 9H6N3K 0L8R1A 0M7N2G 1L8L8L 6G1*Not actual screen shots of inputs or postal code responses; created for illustrative purposes only
OUTPUT*: OUTPUT:
Learning: Plan Output Before InputPostal Code vs City Name: Free text vs Discrete Choice Input
ORIGINAL DATA INPUT*: REVISED DATA INPUT*:
Years in Canada: Open ended comment box vs forced choice
ORIGINAL DATA INPUT*: REVISED DATA INPUT*:
“1978”“At 10 years old”“01 January 2003”“In childhood”“1996”“2007”“2013”“I can’t recall”“Last year.”“5 years ago”
OUTPUT*:
25
2015
25
10
Years in Canada
0 to 5 5 to 10 10 to 15 15 to 20 20 +
*Not actual screenshots of inputs or actual responses; created for illustrative purposes only
OUTPUT*:
Learning: Plan Output Before Input
Technology Data• Full VCES collects information on:
• Where you accessed the appointment from.
• What online platform/software you used to access your appointment.
• What type of device you used to access your appointment.
• How comfortable you are with technologyin your daily life.
Question 11: Where did you access the virtual mental health and/or addiction appointment from?
75.76%
21.2%
3.03%
Home Healthcare Organization Other
Question 13: What type of device did you use to access the virtual mental health and/or addiction appointment?
9.09%
48.48%
42.42%
Tablet Computer Phone
Potential QI Activity: Expanding “Other” to identify community partners for program development
Learning: Expanding Helpful Input/Outputs
Question 14: How comfortable do you feel with technology in your daily life?
QI Learning:
• This population of participants is already accessing virtual supports because of pandemic; this may skew this data.
• It is important to get information on digital inclusion & who is not accessing virtual supports.
• How do you get information on who is accessing virtual services during the pandemic when that’s most of what you can offer?
• Beginning development of processes to collect this information.
• Began communicating with regional data partners.
Learning: Notice What & Who You’re Missing
Client Experiences: • Overall, participants were highly satisfied.
• Hopeful the fuller roll out to a larger population (more agency programs) will allow for more nuanced data & reveal more areas for improvement.
Question 15: For each of the following statements, please indicate if you Strongly Disagree, Disagree, Agree or Strongly Agree.*Select items from the “Ontario Perception of Care Tool for Mental Health and Addictions (OPOC-MHA).”Copyright © 2015 by CAMH.
1. It was easy to access virtual mental health and/or addiction care at this organization.
3. It was easy to book my virtual mental health and/or addiction appointment.
4. During my virtual mental health and/or addiction appointment, I was able to see the healthcare provider clearly.
5. During my virtual mental health and/or addiction appointment, I was able to hear the healthcare provider clearly.
6. I am confident that the healthcare provider at this organization and my other service providers are working as a team.
7. I feel that there was an adequate amount of time allotted for the virtual mental health and/or addiction appointment.
2. *The wait time for services was reasonable for me.
8. I felt comfortable during my virtual mental health and/or addiction appointment.
9. I believe virtual mental health and/or addiction care is just as effective as in-person healthcare.
Learning: What if your results are all positive?
Q15: Continued.
10. I was able to get a virtual mental health and/or addiction appointment sooner than an in-person healthcare appointment.
11. *I was assured my personal information was kept confidential.
12. *Staff understood and responded to my needs and concerns.
13. *I was treated with respect by program staff.
14. I received compassionate virtual mental health and/or addiction care.
15. I felt safe (emotionally and physically) during my virtual mental health and/or addiction appointment.
16. The healthcare provider spoke with me about my mental health and/or addiction in a way that I could understand.
17. *I was involved as much as I wanted to be in decisions about my treatment services and supports.
18. The healthcare provider explained to me the benefits and risks of any treatments or interventions that were recommended during my virtual mental health and/or addiction appointment.
This may sound strange, but….It may sound counter-intuitive, but we’d actually like to see LESS positiveresponses so that we can find ways to improve!
We anticipate that a more diverse range of experiences exists within our client population, and that by increasing our pool of participants, we willget a more representative sample of their satisfaction level.
19. I am confident that I will be able to follow the healthcare provider's recommendations.
20. I understand what to do if I have a mental health and/or addiction emergency following this appointment.
21. The physical location of my virtual mental health and/or addiction appointmen was convenient for me to get to.
22. Overall, I am satisfied with my virtual mental health and/or addiction appointment.
Q15: Continued.
… if your results are all positive, keep digging!
Preliminary Statistical Analysis• Lean Six Sigma Black Belt tools
• Experimented with running statistical analysis tools designed to output correlation calculations and illustrative scatterplots
• Learned we were able to investigate correlations between responses to different VCES items
Preliminary statistical analysis: A few interesting findings
R² = 0.0495
0
0.5
1
1.5
2
2.5
3
3.5
0 10 20 30 40 50 60 70
Age
No relationship found between age & opinion on effectiveness
Age
Que
stio
n 9
VS
Question 9: I believe virtual mental health and/or addiction care is just as effective as in-person healthcare.
Respondent’s Age
Significant correlation between feeling
understood and responded to & perceived ability to follow through on staff’s
recommendations. R² = 0.425
0
0.5
1
1.5
2
2.5
3
3.5
0 0.5 1 1.5 2 2.5 3 3.5
12. *Staff understood and responded to my needs and concerns.
Que
stio
n 19
Question 12
VS
Question 19: I am confident that I will be able to follow the healthcare provider’s recommendations
Question 12: Staff understood and responded to my needs and concerns.
Weak correlation between feeling safe and perceived ability to follow through on staff’s recommendations.
R² = 0.3609
0
0.5
1
1.5
2
2.5
3
3.5
0 0.5 1 1.5 2 2.5 3 3.5
15. I felt safe (emotionally and physically) during my virtual mental health and/or
addiction appointment.
Que
stio
n 19
Question 15
VS
Question 19: I am confident that I will be able to follow the healthcare provider’s recommendations
Question 12: I felt safe (emotionally and physically) during my virtual health and/or addiction appointment
Significant correlation between feeling confident
in ability to follow recommendations & overall
satisfaction with appointment
Que
stio
n 19
Question 22:
VS
Question 19: I am confident that I will be able to follow the healthcare provider’s recommendations
Question 22: Overall, I am satisfied with my virtual mental health and/or addiction appointment
• Again, data input matters: For statistical analysis, text questions must be converted to/coded as numbers
• It’s easier to have this be an automated input, rather than manually code afterward
Learning: Planning for Statistical Analysis
StronglyDisagree
Disagree Agree Strongly Agree
1 2 3 4
• You can run literally thousands of correlations: choose potential research questions wisely.
• Challenging Stereotypes: Further data on age and comfort with technology to see if lack of correlation is accurate.
• Supporting Newcomers: Is there a correlation between how long you’ve been in Canada and your satisfaction with virtual supports?
• Health Equity: Do diverse communities have different experiences of virtual supports?
Learning: Planning for Statistical Analysis
• Be open to surprises: the data may suggest unexpected research questions.
Next Steps:
QI Activities broadly summarized into themes:
• Changing data inputs to allow for more effective and efficient data collection and analysis.• Creating a process to consistently and accurately collect “n”• Strategically planning communications with staff to maximize the number of responses.• S.M.A.R.T. Research Questions:
• Mindfully crafting targeted research questions, especially around demographics and health equity, to be able to hone in on which correlations to run.
• Removing and/or adding questions to tailor our data set to our research questions.• Reviewing the revised processes with Lived Experience to
ensure continued attention to person-centered values. • Creating complimentary data collection projects to gather information
on digital inclusion & staff experiences delivering virtual supports.
Anticipated Outcome: Collect more diverse client experience data which will help define QI targets for changes virtual program delivery to increase client satisfaction.
QI Comes First: Prior to second implementation, we will be implementing over 20 quality improvement ideas to survey items and collection process;
May 2021: Roll out the survey to a wider number of programs to increase our data pool.