evaluation framework (draft) - pcori · pcori draft evaluation framework 4 . 1. what is the overall...

Evaluation Framework (DRAFT)

April 22, 2014

About PCORI

PCORI was authorized by the Patient Protection and Affordable Care Act of 2010 as a non-profit, nongovernmental organization and is charged with helping patients, clinicians, purchasers, and policy makers make better-informed health decisions by “advancing the quality and relevance of evidence about how to prevent, diagnose, treat, monitor, and manage diseases, disorders, and other health conditions.” It does this by producing and promoting high-integrity, evidence-based information that comes from research guided by patients, caregivers, and the broader healthcare community. PCORI’s strong patient-centered orientation directs attention to individual and system differences that may influence research strategies and outcomes. PCORI is charged with producing useful, relevant clinical evidence through the support of new research and the analysis and synthesis of existing research. PCORI is committed to transparency and a rigorous stakeholder-driven process that emphasizes patient engagement. PCORI uses a variety of forums and public comment periods to obtain public input to enhance its work. Patient-Centered Outcomes Research Institute 1828 L St., NW, Suite 900 Washington, DC 20036 Phone: (202) 827-7700 Fax: (202) 355-9558 Email: [email protected] Follow us on Twitter: @PCORI

PCORI Draft Evaluation Framework 2

Preface

The purpose of the PCORI Evaluation Framework is to capture all of the questions that our stakeholders and we at PCORI have about our work and about PCOR in general and convey whether and how we plan to answer those questions. This draft of a portion of the framework focuses on the set of questions pertaining to “The PCORI Way” and its impact on Comparative Effectiveness Research (CER), on attainment of our strategic goals, and ultimately on health decisions, health care, and health outcomes. Each section has a figure that depicts our theory about how our approach is supposed to work (If we do A, it will result in B), followed by a table that outlines the questions about that theory (Is A resulting in B?). It is not necessary to understand the figures in order to understand the questions in the tables. We anticipate a wide variety of readers for this document, and the figures are provided for those who find this kind of display of information helpful. The first section of the framework covers the “Overall Impact of PCORI.” The subsequent sections “zoom in” on six key ingredients in the PCORI recipe (zooming in is represented through figures with a red circle on the model of the overall impact of PCORI, followed by a more detailed model of the circled section). The detailed theory underlying how each ingredient contributes to PCORI’s success in attaining its goals is indicated with dashed arrows in the figure, and the questions about it are outlined in the corresponding table. We endeavor to capture all of the questions about PCORI’s work that are important to our stakeholders. We invite you to review this first draft of the questions in the section of the framework that pertains to PCORI’s approach to CER and whether it’s making a difference and let us know if you see your own questions reflected in it. We recognize that this framework is lengthy and packed full of information. That’s simply a reflection of the numerous issues it seeks to address and the many important questions our stakeholders have about our work. We’re striving to keep the questions as straightforward and understandable as possible, and we welcome your thoughts on how to best do that. Please feel free to tackle the entire draft or focus on one or two sections that interest you the most. We value all comments and appreciate your input. Thank you for your help!


http://www.pcori.org/public-feedback-on-pcori-evaluation-framework/

http://www.pcori.org/public-feedback-on-pcori-evaluation-framework/

Contents

Preface ................................................................................................................................................... 3

1. What Is the Overall Impact of PCORI? ................................................................................................ 5

Figure 1. Model for Evaluating the Overall Impact of PCORI

Table 1. Evaluation Questions for the Overall Impact of PCORI

2. What Is the Impact of Engagement in Research? ........................................................................................ 8

Figure 2a. Model for Evaluating the Impact of PCORI: Where Does Engagement in Research Fit?

Figure 2b. Model for Evaluating Engagement in Research

Table 2. Evaluation Questions for Evaluating the Impact of Engagement in Research

3. What Is the Impact of PCORI’s Approach to Infrastructure Development? ............................................... 11

Figure 3a. Model for Evaluating the Impact of PCORI: Where Does PCORI’s Approach to Infrastructure Development Fit?

Figure 3b. Model for Evaluating PCORI’s Approach to Infrastructure Development

Figure 3c. Model for Evaluating PCORI’s Approach to Infrastructure Development

Table 3. Evaluation Questions for PCORI’s Approach to Infrastructure Development

4. What Is the Impact of PCORI’s Approach to Merit Review? ...................................................................... 15

Figure 4a. Model for Evaluating the Impact of PCORI: Where Does PCORI's Approach to Merit Review Fit?

Figure 4b. Model for Evaluating PCORI’s Approach to Merit Review

Table 4. Evaluation Questions for PCORI’s Approach to Merit Review

5. What Is the Impact of PCORI’s Approach to Topic Generation and Research Prioritization? .................... 18

Figure 5a. Model for Evaluating the Impact of PCORI: Where does PCORI’s Approach to Topic Generation and Research Prioritization Fit?

Figure 5b. Model for Evaluating PCORI’s Approach to Topic Generation and Research Prioritization

Table 5. Evaluation Questions for PCORI’s Approach to Topic Generation and Research Prioritization

6. What Is the Impact of PCORI’s Approach to Developing a PCOR Community? ......................................... 21

Figure 6a. Model for Evaluating the Impact of PCORI: Where Does PCORI's Approach to Developing a PCOR Community Fit?

Figure 6b. Model for Evaluating PCORI’s Approach to Developing a PCOR Community

Table 6. Evaluation Questions for PCORI’s Approach to Developing a PCOR Community

7. What Is the Impact of PCORI’s Approach to Communication, Dissemination, and Implementation? ...... 24

Figure 7a. Model for Evaluating the Impact of PCORI: Where Does PCORI’s Approach to Communication, Dissemination, and Implementation Fit?

Figure 7b. Model for Evaluating PCORI’s Approach to Communication, Dissemination, and Implementation

Table 7. Evaluation Questions for PCORI’s Approach to Communication, Dissemination, and Implementation


1. What Is the Overall Impact of PCORI? Figure 1. Model for Evaluating the Overall Impact of PCORI


Table 1. Evaluation Questions for the Overall Impact of PCORI

Question Prioritization (In

Development)

Metrics/ indicators

(In Development)

Methods (To be

Developed)

Sources (To be

Developed)

Does the creation of a dedicated entity (PCORI) and funding stream (PCORTF) result in:

• increased total funding for CER,

• increased number of CER studies,

• increased availability of CE information for decision makers,

• higher impact of CER: o better informed health decisions, o better health care, and o better health outcomes?

How can information from PCORI studies be characterized in terms of:

CORE

M

EAS-

URES

• the usefulness of information,

• uptake of information, and

• cost of research?

How does information resulting from PCORI’s approach to CER (patient-centered CER) compare with that from others in terms of:

• usefulness, • uptake in clinical practice, • impact, and • cost to produce?

What is the impact of PCORI’s work among a variety of key stakeholder groups (patients, clinicians, researchers, payers, caregivers, policy makers, etc.) on:

• attitudes toward CER,

• use of CER for health decision making,

• attitudes toward engagement of patients and other stakeholders in CER, and



Development)

Metrics/ indicators

(In Development)

Methods (To be

Developed)

Sources (To be

Developed)

• engagement of patients and other stakeholders in CER?

How does PCORI’s work influence others, including:

• adoption of CER and PCORI terminology,

• use of PCORI guidance regarding patient-centered CER,

• use of PCORI approaches to research topic generation and prioritization, and merit review,

• use of PCORI-supported curricula or training,

• collaborations/co-funding with other funders,

• use and support of PCORnet for the conduct of CER,

• use of PCORI Methodology Standards for patient-centeredness,

• use of PCOR methods evidence,

• endorsement, promotion, and dissemination of PCORI work, and

• funding investments in PCOR?

What is the relationship between PCORI’s usefulness criteria and uptake of research information?


2. What Is the Impact of Engagement in Research?

Figure 2a. Model for Evaluating the Impact of PCORI: Where Does Engagement in Research Fit?


Figure 2b. Model for Evaluating Engagement in Research


Table 2. Evaluation Questions for Evaluating the Impact of Engagement in Research


Development)

Metrics/ indicators

(In Development)

Methods (To be

Developed)

Sources (To be

Developed)

What is the impact of Engagement in Research, particularly: • who (types of stakeholders) is involved • when engagement occurs (stages of research) • how much (the nature of engagement: information, consultation, collaboration,

patient/stakeholder direction) • level of perceived influence of patients and stakeholders • stakeholder experiences as engaged partners (satisfaction, empowerment) • principles of engagement (trust, co-learning, etc.)

on:

Research processes and characteristics for studies that matter that to patients:

o relevance of research questions/goals,

o study design,

o recruitment and retention,

o recruitment and retention of hard-to-reach populations,

o experiences of participants in PCORI studies,

o quality (appropriate methods, amount of missing data, completion of trial, etc.),

o study completion,

o dissemination of findings,

o end-user understanding,

o end-user trust,

CORE

M

EAS-

URES


• uptake of information, and • cost of research?

What is the role of contextual factors (e.g., study population, study setting, etc.) in the associations between engagement and research processes and characteristics the usefulness of information, and the uptake of information?


3. What Is the Impact of PCORI’s Approach to Infrastructure Development?

Figure 3a. Model for Evaluating the Impact of PCORI: Where Does PCORI’s Approach to Infrastructure Development Fit?


Figure 3b. Model for Evaluating PCORI’s Approach to Infrastructure Development (the impact of engagement in the development of PCORnet)


Figure 3c. Model for Evaluating PCORI’s Approach to Infrastructure Development (the impact of PCORnet)


Table 3. Evaluation Questions for PCORI’s Approach to Infrastructure Development


Development)

Metrics/ indicators

(In Development)

Methods (To be

Developed)

Sources (To be

Developed)

What is the effect of Engagement of patients and other stakeholders in the development of research networks on:

• enrollment in the networks,

• type(s) of data contributed by patients,

• patient-reported outcomes collected,

• mechanisms to synthesize and provide data to patients in the Network,

• cohort participant trust in the Network,

• cohort participant perceptions of data use,

• cohort participant opportunities for engagement in the studies conducted in the Network,

• attainment of continued funding for the network,

• number of studies conducted through the Network, and

• Network participants’ receipt of study results?

Compared to other CER studies, for studies conducted in the PCORnet, what is the impact of PCORnet on:

• experiences of participants in PCORnet studies,

• recruitment for CER studies,

• proportion of completed CER studies,

• completion time for CER studies,

• richness of data collected,

• study participants trust in the research process,

CORE

M

EAS-

URES


• uptake of information, and

• cost of research?


4. What Is the Impact of PCORI’s Approach to Merit Review?

Figure 4a. Model for Evaluating the Impact of PCORI: Where Does PCORI's Approach to Merit Review Fit?


Figure 44b. Model for Evaluating PCORI’s Approach to Merit Review


Table 4. Evaluation Questions for PCORI’s Approach to Merit Review


Development)

Metrics/ indicators

(In Development)

Methods (To be

Developed)

Sources (To be

Developed)

What is the impact of PCORI’s approach to Merit Review (inclusion of patients and other stakeholders, unique criteria, and application of these criteria) on:

• perspectives incorporated in the review process, and • selection of research projects for funding?


5. What Is the Impact of PCORI’s Approach to Topic Generation and Research Prioritization (TGRP)?

Figure 5a. Model for Evaluating the Impact of PCORI: Where does PCORI’s Approach to Topic Generation and Research Prioritization (TGRP)Fit?


Figure 6b. Model for Evaluating PCORI’s Approach to Topic Generation and Research Prioritization (TGRP)


Table 5. Evaluation Questions for PCORI’s Approach to Topic Generation and Research Prioritization (TGRP)


Development)

Metrics/ indicators

(In Development)

Methods (To be

Developed)

Sources (To be

Developed)

What is the impact of PCORI’s approach to Topic Generation, Prioritization, and Selection (inclusion of patients and other stakeholders and methods for ranking and selection) on:

• perspectives incorporated into topic selection process, and • the topics selected for funding?

Compared to broad funding announcements, what is the effect of targeted funding announcements on: • the impact of information?

Compared to funding opportunities developed with input from scientists only, what is the effect of funding opportunities developed based on multistakeholder input on:

• the impact of information?


6. What Is the Impact of PCORI’s Approach to Developing a PCOR Community?

Figure 6a. Model for Evaluating the Impact of PCORI: Where Does PCORI's Approach to Developing a PCOR Community Fit?


Figure 6b. Model for Evaluating PCORI’s Approach to Developing a PCOR Community


Table 6. Evaluation Questions for PCORI’s Approach to Developing a PCOR Community


Development)

Metrics/ indicators

(In Development)

Methods (To be

Developed)

Sources (To be

Developed)

What is the effect of PCORI’s unique approach to developing a PCOR community on: • the availability of stakeholders prepared to engage in research, • the quality of contributions of engaged partners, and • stakeholder participation in dissemination of findings?

What is the effect of PCORI Engagement Awards, including the pipeline to proposal awards, on: • the availability of stakeholders to engage in research, • the quantity of patient-centered CER, and • PCORI’s dissemination and implementation efforts?


7. What Is the Impact of PCORI’s Approach to Communication, Dissemination, and Implementation?

Figure 7a. Model for Evaluating the Impact of PCORI: Where Does PCORI’s Approach to Communication, Dissemination, and Implementation Fit?


Figure 7b. Model for Evaluating PCORI’s Approach to Communication, Dissemination, and Implementation


Table 7. Evaluation Questions for PCORI’s Approach to Communication, Dissemination, and Implementation


Development)

Metrics/ Indicators

(In Development)

Methods (To be

Developed)

Sources (To be

Developed)

What is the effect of including patents and other stakeholders in planning dissemination on: • how findings are disseminated, • to whom the findings are disseminated, • end-user understanding,

• end-user trust, and

• uptake of information?


evaluation framework (draft) - pcori · pcori draft evaluation framework 4 . 1. what is the overall...

Documents