evaluating engagement in research: promising …...recruitment & retention • recruitment...
TRANSCRIPT
Organization Logo Here(150 dpi Resolution jpeg or
png file)
The 2015 PCORI Annual Meeting:
Progress in Building a Patient-Centered Comparative Clinical Effectiveness Research Community
The 2015 PCORI Annual Meeting:
Progress in Building a Patient-Centered Comparative Clinical Effectiveness Research Community
Evaluating Engagement in
Research: Promising Practices
from PCORI’s Portfolio
Laura Forsythe, PhD, MPH
Associate Director, Evaluation & Analysis
Michelle Johnston-Fleece, MPH
Engagement Officer, PCORI
October 7, 2015
Organization Logo Here(150 dpi Resolution jpeg or
png file)
What is Research Engagement?
Patient and other stakeholder involved in
• Formulating research questions;
• Defining essential characteristics of study design, participants, and comparators;
• Identifying and selecting outcomes that matter to patients;
• Monitoring study conduct and progress; and/or
• Designing and carrying out plans for dissemination and implementation activities.
2
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Why is Research Engagement
Important?
• Early evidence suggests that engagement may result in
– Improved understanding of study populations’ needs
– More relevant research studies
– Better understanding of research results
• Yet, little evidence exists about the most promising practices for how to engage and for the effects of engagement in terms of the research process, uptake of findings, and impact on health
3
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Who Are Our Stakeholders?
Purchasers
Caregivers/Family MembersPayers
Patients/Consumers
Clinicians
Training Institutions
Policy Makers
Hospitals/Health SystemsIndustry
Patient/Caregiver Advocacy Organizations
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Objectives for Measuring
Engagement
• Describe engagement in PCORI funded projects
• Evaluate impact on PCORI strategic goals
• Guide current awardees, future applicants, and others
interested in PCOR
• Support project progress based on learnings
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Conceptual Model of PCOR
PCOR Principles
Frank L, Forsythe L, Ellis L, Schrandt S, Sheridan S, Gerson J, Konopka K, Daugherty S. Conceptual and Practical Foundations of Patient Engagement in Research at the Patient-
Centered Outcomes Research Institute Plans. Quality of Life Research 2015. 24(5):1033-1041 DOI 10.1007/s11136-014-0893-3
Actions
OutcomesFoundational Elements
• Readiness
• Channels
• Policies
• Resources
• Near-term
• Intermediate
• Long-term
• Initiate and maintain
partnerships
• Communicate
• Capture and use
perspectives
• Ensure influence
• Train
• Share
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Additional Engagement Resources
• PCORI has developed other resources to help guide your engagement activities, housed on the “What We Mean by Engagement” page on the PCORI website:
http://www.pcori.org/funding-opportunities/what-we-mean-engagement
• Resources include:– Framework for Financial Compensation of Patient, Caregiver and
Patient Organization Stakeholders
– Engagement Rubric
– Sample Engagement Plans
– PCORI Stakeholder Groups
– Short Videos on Engagement in Research
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Methods
8
Organization Logo Here(150 dpi Resolution jpeg or
png file)
• Annual data collection from awardees
• Awardees nominate research partners to be invited to respond
Data collection on Engagement
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Methods for Qualitative Analysis
Partnership with American Institutes for Research
• Developed & applied codebook based on research questions and review of the open-text responses
• Identified major themes
• Mapped themes to conceptual model of PCOR1
(N= 178 researchers, 257 patients and stakeholders)
1 Frank L, Forsythe L, Ellis L, Schrandt S, Sheridan S, Gerson J, Konopka K, Daugherty S. Conceptual and practical foundations of
patient engagement in research at the patient-centered outcomes research institute. Qual Life Res. 2015 Jan 6.
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Results
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Results:
Type of Stakeholders Engaged
Year 1 (N = 91)
Organization Logo Here(150 dpi Resolution jpeg or
png file)
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1Percent of Projects
Surveys
Advisory Group or Group Forum
Research Team
CO-Investigator
Approaches to Engagement
Results:
Approaches to Engagement
Year 1 (N=90)
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Results:
Stages of Engagement
Year 1 (N = 91)
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
Dissemination
Results Interpretation
Data Analysis
Data Collection
Recruiting Participants
Study Design
Adding People to the Team
Developing the Budget
Proposal Development
Identifying Research Topics or Questions
Stages of Engagement
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Conceptual Model of PCOR
PCOR Principles
Actions
OutcomesFoundational Elements
• Readiness
• Channels
• Policies
• Resources
• Near-term
• Intermediate
• Long-term
• Initiate and maintain
partnerships
• Communicate
• Capture and use
perspectives
• Ensure influence
• Train
• Share
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Results:
Initiate and Maintain Partnerships
• Patients and stakeholders noted the usefulness of being involved early or experienced a desire to be involved earlier
• Researchers noted several challenges: – keeping patients engaged
throughout the project
– setting expectations for
project funding
– lacking funds for early
involvement
I wish they would have contacted us earlier in
the grant process so we may have been able to
work in more areas of the state vs. a small
section. (Patient/Stakeholder)
It is always hard to go back to
stakeholders…when a project has not been
funded. This, in my opinion, is one of the
greatest challenges to engaging with patients
in the conceptualization and planning
phases. (Researcher)
We were able to avoid some obstacles and a
learning curve by learning about the issues from
families prior to starting. (Researcher)
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Conceptual Model of PCOR
PCOR Principles
Actions
OutcomesFoundational Elements
• Readiness
• Channels
• Policies
• Resources
• Near-term
• Intermediate
• Long-term
• Initiate and maintain
partnerships
• Communicate
• Capture and use
perspectives
• Ensure influence
• Train
• Share
Organization Logo Here(150 dpi Resolution jpeg or
png file)
<< Develop infrastructure for D&I >>Researchers need to understand patients and
how to communicate with them, especially if
they are not in the same age group or cultural
background. (Patient/Stakeholder)
Results:
Communication
• Managing power differentials
• Managing diverse groups
• Using plain language
It is sometimes difficult to speak the same
language at group meetings. In other words,
the language style tends to be dominated by
researchers or clinicians. (Researcher)
How does one facilitate conversations across
different stakeholder groups when there are
strong feelings that can conflict? (Researcher)
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Conceptual Model of PCOR
PCOR Principles
Actions
OutcomesFoundational Elements
• Readiness
• Channels
• Policies
• Resources
• Near-term
• Intermediate
• Long-term
• Initiate and maintain
partnerships
• Communicate
• Capture and use
perspectives
• Ensure influence
• Train
• Share
Organization Logo Here(150 dpi Resolution jpeg or
png file) Board of Governors Meeting, May 4, 2015
Inventory of Engagement Activities
• Identifying research topics
• Developing research questions
• Proposal development
• Developing the budget
• Expanding the research team
• Study Design
Planning the Study
• Recruiting & retaining participants
• Data collection
• Data analysis
• Results review, interpretation, & transition
Conducting the study
• Disseminating/Sharing research findings
Dissemination
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Capturing the Patient Perspective:
Study Design
• Help choose research methods: measures, interventions, comparators, and outcomes
• Decide on data collection methods
• Review and revise study plans and materials
We presented aspects of study design to the
group and solicited their input. I.e. - what
should the control group be for the RCT - is it
‘standard practice’ … or should it be a
currently existing but potentially ineffective
intervention? (Researcher)
Helped the investigators decide what cohort of
patients to include. (Patient/Stakeholder)
Gave clinical input into choice of screening
measures. (Patient/Stakeholder)
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Capturing the Patient Perspective:
Recruitment & Retention
• Strategize for recruitment and retention
• Serve as liaison between research team and groups to be recruited
• On-the-ground recruiting of study participants, practices, and partner organizations
We helped the researchers to understand
potential barriers to enrollment, particularly for
minority candidates, and identified responses
to these barriers. (Patient/Stakeholder)
The stakeholder Co-I's relationship with
individuals similar to those recruited for this
study allowed her to provide insights to this
population that is often difficult to recruit and
maintain over the course of the study. (Researcher)
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Conceptual Model of PCOR
PCOR Principles
Actions
OutcomesFoundational Elements
• Readiness
• Channels
• Policies
• Resources
• Near-term
• Intermediate
• Long-term
• Initiate and maintain
partnerships
• Communicate
• Capture and use
perspectives
• Ensure influence
• Train
• Share
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Ensure Influence:
Study Design
• Changes to study design to make it more responsive to patient needs, feasible in clinical setting
• Range of impact reported, from minimal to major
Contributed to the approach taken and to
creating conditions that would allow maximum
participation on the part of both patients and
providers. (Patient/Stakeholder)
This led us to modify our original 2-group
research design and include a 3rd group;
community based group exercise. (Patient/Stakeholder)
High impact - changed design, outcomes, flow
of study. (Researcher)
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Ensure Influence:
Recruitment & Retention
• Recruitment procedures more responsive to patient needs
• Making more potential participants aware of the study
• Improvements in recruiting and retaining difficult-to-reach populations
Since discussing our challenges with
recruiting and retaining study participants, we
have had only one participant decline to
participate. (Patient/Stakeholder)
Outreach materials, recruitment procedures
were modified significantly. (Researcher)
Brought in quite a few new people to
participate. (Patient/Stakeholder)
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Conceptual Model of PCOR
PCOR Principles
Actions
OutcomesFoundational Elements
• Readiness
• Channels
• Policies
• Resources
• Near-term
• Intermediate
• Long-term
• Initiate and maintain
partnerships
• Communicate
• Capture and use
perspectives
• Ensure influence
• Train
• Share
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Results:
Training for Partnership
• Training needs– Topic background
– Research methods
– How to provide input effectively
– How to communicate about research
– Training for researchers on how to engage partners
I have searched for training webinars and
other tools to help…my staff to better
understand our role. (Patient/Stakeholder)
There is a steep learning curve to
understanding research and how to conduct
research. (Researcher)
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Conceptual Model of PCOR
PCOR Principles
Actions
OutcomesFoundational Elements
• Readiness
• Channels
• Policies
• Resources
• Near-term
• Intermediate
• Long-term
• Initiate and maintain
partnerships
• Communicate
• Capture and use
perspectives
• Ensure influence
• Train
• Share
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Early Impacts Beyond the Project
• Increased knowledge and skills about research
• Increased knowledge about or engagement in health
The work on this project did inform me of the
importance of patient/ family engagement in
health care decision-making and has
prompted me to adopt some of these
approaches in my personal life.(Patient/Stakeholder)
I have watched my staff improve in their
professional skills as well through this
project… We have learned skills and
developed tools that will enhance our success
in the future. (Patient/Stakeholder)
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Early Impacts Beyond the Project
• Increased interest in patient/ stakeholder engagement
• Feeling like participation had impact beyond project
I am more comfortable suggesting inclusion of
patients on research projects. (Researcher)
It allowed me to feel like a more rounded physician
because I am doing research to help the general
community. It expands my influence on the
community in which I live. (Patient/Stakeholder)
Expanded our interest and relationships with other
researchers in our community… We also brought
together several of the researchers in our
community to discuss who we are and what we do
as to try and coordinate projects geared towards
senior adults. (Patient/Stakeholder)
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Conclusions
• Patients and other stakeholders are involved in PCORI projects in a variety of ways across the research continuum
• Early engagement is desirable for many partners, particularly if challenges can be addressed
• Fostering an environment for open communication is important
• There is preliminary evidence for the impact of engagement across stages of the research, including study design and recruitment
• Additional impacts outside the research project identified
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Turning Information into Action
• Small group discussions (45 min)
– Describe an “AHA!” moment from your experiences with research engagement.
– What have been your biggest successes in research engagement? What methods did you employ to ensure success?
– Your biggest challenges? What might improve your partnerships in the future?
• Report back to the group (30 min)
– Themes, compelling anecdotes, practical ideas you want to implement in your work
– What will you do differently?
Board of Governors Meeting, May 4, 2015
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Breakout Sessions
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Large Group Discussion
34
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Learn More
35
www.pcori.org
#PCORI2015
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Organization Logo Here(150 dpi Resolution jpeg or
png file)
Questions?
Organization Logo Here(150 dpi Resolution jpeg or
png file)
References
1. Staley K. Exploring impact: public involvement in NHS, public health and social care research. Eastleigh, United Kingdom: INVOLVE; 2009
2. Brett J, Staniszewska S, Mockford C, Seers K, Herron-Marx S, Bayliss H: The PIRICOM Study: A Systematic Review of the Conceptualisation, Measurement, Impact and Outcomes of Patients and Public Involvement in Health and Social Care Research. Warwick: University of Warwick.
3. Domecq JP, Prutsky G, Elraiyah T, Wang Z, Nabhan M, Shippee N, Brito JP, Boehmer K, Hasan R, Firwana B, Erwin P, Eton D, Sloan J, Montori V, Asi N, Dabrh AM, Murad MH. Patient engagement in research: a systematic review. BMC Health Serv Res. 2014 Feb 26;14:89. doi: 10.1186/1472-6963-14-89.
4. Forsythe LP, Szydlowski V, Murad MH, Ip S, Wang Z, Elraiyah TA, Fleurence R, Hickam DH. A systematic review of approaches for engaging patients for research on rare diseases. J Gen Intern Med. 2014 Aug;29 Suppl3:S788-800. doi: 10.1007/s11606-014-2895-9.
5. Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving from theory to practice. J Comp Eff Res. 2015;4(2):133-45. doi: 10.2217/cer.14.79.
6. Forsythe LP, Ellis LE, Edmundson L, Sabharwal R, Rein A, Konopka K, Frank L. Patient and Stakeholder Engagement in the PCORI Pilot Projects: Description and Lessons Learned. J Gen Intern Med. 2015 Jul 10.
7. Concannon TW, Fuster M, Saunders T, Patel K, Wong JB, Leslie LK, Lau J. A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research. J Gen Intern Med. 2014 Dec;29(12):1692-701. doi: 10.1007/s11606-014-2878-x. Epub 2014 Jun 4.
8. Concannon TW, Meissner P, Grunbaum JA, McElwee N, Guise JM, Santa J, Conway PH, Daudelin D, MorratoEH, Leslie LK. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J Gen Intern Med. 2012 Aug;27(8):985-91. doi: 10.1007/s11606-012-2037-1. Epub 2012 Apr 13.
Organization Logo Here(150 dpi Resolution jpeg or
png file)
The 2015 PCORI Annual Meeting:
Progress in Building a Patient-Centered Comparative Clinical Effectiveness Research Community
The 2015 PCORI Annual Meeting:
Progress in Building a Patient-Centered Comparative Clinical Effectiveness Research Community
Thank You!Laura Forsythe, PhD, MPH
Associate Director, Evaluation & Analysis
Michelle Johnston-Fleece, MPH
Engagement Officer, PCORI