the pcor cds-ln environmental scan: spurring action by ... · 11/4/2016 · spurring action by...

PCOR CDS Learning Network Transforming Patient-Centered Outcomes Research Findings into Action

The PCOR CDS-LN Environmental Scan: Spurring Action by Identifying Barriers and Facilitators To the Dissemination of PCOR through PCOR-Based Clinical Decision Support Prepared by

Joshua E. Richardson, PhD, MS, MLIS Blackford Middleton, MD, MPH, MSc

Jerome A. Osheroff, MD Melissa Callaham, MS

Laura Marcial, PhD, MLIS Barry H. Blumenfeld, MD, MS

RTI International 040 E. Cornwallis Road

Research Triangle Park, NC 27709

iii

List of Interviewees Sarah Corley, MD, FACP NextGen Healthcare Laura Crawford, MIS Eli Lilly and Company e-Patient Dave deBronkart Society for Participatory Medicine e-Patient Dave, LLC Doug Fridsma, MD, PhD American Medical Informatics Association Patricia Gallagher, MLS, MA, AHIP National Library of Medicine Clifford Goldsmith, MD Microsoft Robert A. Greenes, MD, PhD University of Arizona Peter Haug, MD Intermountain Health Care Erin Holve, PhD, MPH, MPP Department of Health Care Finance, Government of the District of Columbia, (formerly with Academy Health) Tonya Hongsermeier, MD Lahey Health (permission pending) Kensaku Kawamoto, MHS, MD, PhD University of Utah School of Medicine Thomas Kuhn, MA American College of Physicians Lisa Lang, MPP National Library of Medicine Kathy Lohr, PhD RTI International Edwin Lomotan, MD

Agency for Healthcare Research and Quality (AHRQ) Erin Mackay, MPH National Partnership for Women & Families Blackford Middleton, MD MPH MSc Apervita, Inc. David McCallie, MD Cerner Corporation Lauren Murray National Partnership for Women & Families Kara Odom Walker, MD, MPH, MSHS Patient-Centered Outcomes Research Institute (PCORI) Jerome A. Osheroff, MD TMIT Consulting, LLC Eric Poon, MD, MPH Duke University Fred Rachman, MD Alliance of Chicago Trent Rosenbloom, MD, MPH, FACMI Vanderbilt University Mark Savage, JD National Partnership for Women & Families Marcus Thygeson, MD, MPH Blue Shield of California Craig Umscheid, MD, MSCE University of Pennsylvania

iv

Danny van Leeuwen, MPH, RN, CPHQ Health Hats/ Patient-Centered Outcomes Research Institute (PCORI) Advisory Panel on Communication and Dissemination Research

Nate Weiner Avhana Health, Inc. Noah Weiner Avhana Health, Inc.

Jon White, MD The Office of the National Coordinator for Health Information Technology (ONC) Adam Wilcox, PhD University of Washington/ Patient-Centered Outcomes Research Institute (PCORI) Methods Committee

Marc Williams, MD Genomic Medicine Institute Geisinger Health System Adam Wright, PhD Harvard Medical School

v

Table of Contents

Section Page

1. Executive Summary 1

2. Background 4

3. Methods 63.1 Team .................................................................................. 63.2 Environmental Scan Theoretical Basis ...................................... 63.3 Timeline .............................................................................. 73.4 Data Collection ..................................................................... 8

3.4.1 Senior Research Team Discussion Group ......................... 83.4.2 Individual Expert Interviews .......................................... 93.4.3 Feedback on the PCOR CDS-LN Analytic Framework for

Action ........................................................................ 93.5 Data Collection—Peer-Reviewed Literature and Gray Literature . 103.6 Data Analysis ..................................................................... 103.7 Limitations ......................................................................... 10

4. Results 124.1 PCOR CDS-LN Analytic Framework for Action .......................... 124.2 Potential Barriers and Facilitators to PCOR-Based CDS ............. 13

4.2.1 Addressing External Factors ........................................ 134.2.2 Prioritizing PCOR Findings for Dissemination via CDS ...... 164.2.3 Authoring CDS Interventions ....................................... 184.2.4 Implementing CDS Interventions ................................. 234.2.5 Measuring Decisions and Outcomes .............................. 244.2.6 Learning from PCOR-Based CDS Experience .................. 25

5. Discussion 26

6. References 28

Appendices

A: Environmental Scan Timeline B: PubMed Search Strategy C: Expert Suggestions for Evaluating PCOR D: Experts’ Suggested Conditions for PCOR-based CDS E: PCOR-based CDS Bibliography F: CDS Current State and Needs

vi

Table of Exhibits

Exhibit Page

Exhibit 1-1. The PCOR CDS-LN Analytic Framework for Action ............ 2Exhibit 2-1. Intended Outcomes from the PCOR CDS-LN Environmental

Scan ......................................................................... 5Exhibit 3-1. Environmental Scan Research Team .............................. 6Exhibit 3-2. Environmental Scan Methodological Framework .............. 7Exhibit 3-3. Senior Research Team Discussion Group Questions ......... 8Exhibit 3-4. Semi-Structured Interview Questions ............................ 9Exhibit 4-1. PCOR CDS Analytic Framework for Action ..................... 12Exhibit 4-2. PCOR CDS-LN Online Bibliography .............................. 14Exhibit 4-3. Sample of Approaches to Evaluating PCOR by Stakeholder

Type ....................................................................... 17Exhibit 4-4. PCORI Standard Contract Language for Funded Research 19Exhibit 4-5. Sample of CDS Current State and Needs ...................... 22

PCOR CDS-LN RTI International

1

1. Executive Summary Translating knowledge gained from Patient-Centered Outcomes Research (PCOR) into clinical practice is key to healthcare quality improvement. A promising way to ensure that PCOR informs clinical care is through clinical decision support (CDS) interventions, which are technical and non-technical approaches that make it easier for care teams – including patients – to make decisions and take actions known to enhance outcomes. Yet best practices are only beginning to emerge and numerous barriers exist: poor coordination among stakeholders, lack of interoperability, sub-optimal implementations, and poor usability to state but a few. Experts agree that more rapidly incorporating PCOR findings into clinical workflow via CDS will require a coordinated effort among multiple stakeholders who jointly promote best practices, collaborate on innovative means for knowledge dissemination and updates, and develop groundbreaking applications that ultimately improve healthcare quality.

The Agency for Healthcare Research and Quality awarded RTI International a U18 cooperative agreement to develop The Patient-Centered Outcomes Research Clinical Decision Support Learning Network (PCOR CDS-LN) to bring together stakeholders to promote a sustainable community around developing, disseminating, and applying PCOR-enabled CDS. The PCOR CDS-LN is designed to accelerate collaborative learning, overcome barriers and reinforce facilitators for effective CDS use, and evaluate the impact of its efforts. As part of the effort to accelerate learning and address the barriers and facilitators to PCOR-based CDS, the research team was to produce an environmental scan that lends insight into the current state of PCOR-based CDS and offer strategic vision for the PCOR CDS-LN in years ahead. This environmental scan is a result of numerous discussions with members of the PCOR CDS-LN Senior Research Team, experts from across multiple sectors, and literature searches. The findings are meant to provide a “springboard for action” to help the PCOR CDS-LN carry out its vision: to create an ecosystem that allows all stakeholders reduce the friction of turning knowledge from PCOR findings into CDS-enabled actions that produce better care and outcomes.

A critical artifact from this environmental scan is the Analytic Framework for Action (AFA), which graphically displays and defines areas of focus around the prioritization, implementation, and evaluation of PCOR-based CDS (see Exhibit 1-1). The AFA, which was developed and vetted by many of the experts we engaged over the course of this effort, provides the means by which we organized the findings and recommendations.


2

Exhibit 1-1. The PCOR CDS-LN Analytic Framework for Action

We generated key takeaways from each of the AFA’s categories:

§ Addressing External Factors: Providing a working definition “PCOR-based CDS” so to promote effective discussions that lead to action: “PCOR-based CDS helps patients and their care teams apply evidence from patient-centered outcomes research to enhance care processes and their results. Approaches include promoting shared decision-making, incorporating patient reported outcomes, factoring in patient preferences to generate patient-specific recommendations for care and others.”

§ Prioritizing PCOR Findings for Dissemination via CDS: Involving multiple stakeholders in use cases to develop a mechanism for prioritizing PCOR for CDS.

§ Authoring CDS Interventions: Engaging stakeholders to promote standards and access to data that promote effective PCOR-based CDS.

§ Implementing CDS Interventions: Promoting research and knowledge-sharing around ways PCOR-based CDS can be, and is being, operationalized for people and their caregivers.

§ Measuring Decisions and Outcomes: Disseminating the various ways that PCOR-based CDS measurably impacts health and healthcare outcomes.

§ Learning from PCOR-based CDS Experience: Partnering with relevant organizations such as the Patient-Centered Outcomes Research Institute (PCORI) and to feed learning from the implementation and use of PCOR-based CDS back into the research and deployment cycle.


3

This environmental scan identifies numerous barriers and highlights multiple actionable opportunities for the PCOR CDS-LN. Stakeholders are demonstrably engaged through the PCOR CDS-LN’s governance structures including its Steering Committee and Advisory Council. Additional activity is now underway in the form of a Barriers and Facilitators Work Group, which is engaged in case development to expand upon the findings in this environmental scan. The environmental scan findings will also be reformatted and shared with several stakeholder groups as appropriate: articles, webinars, and conference presentations. Tailoring the results will make more visible the challenges and recommendations to authoring, developing, implementing, and evaluating PCOR-based CDS.

We would like to thank the many people who offered their time, insights, and expertise to the development of this environmental scan. Their contributions are what made this report possible.


4

2. Background The Agency for Healthcare Research and Quality (AHRQ) references the Affordable Care Act (ACA) in its definition of Patient-Centered Outcomes Research (PCOR) as, “comparative clinical effectiveness research on the impact of patient health outcomes of two or more preventive, diagnostic, treatment, or health care delivery approaches.”1 Other definitions include AHRQ itself, which tacks on the phrase, “including those that are meaningful to patients,”2 and PCORI, which defines PCOR as, “[PCOR] helps people and their caregivers communicate and make informed healthcare decisions, allowing their voices to be heard in assessing the value of healthcare options.”3

There is great potential for disseminating PCOR findings through clinical decision support (CDS), in that it can enable providers and patients to make evidence-based decisions that ultimately improve the quality of care within a learning health system. CDS is now broadly available within EHRs, yet there still is a lack of despite that widespread adoption for CDS that promotes patient-centered outcomes research. To promote advances in this area, AHRQ funded a U18 cooperative agreement “to build a community of stakeholders” with four official aims4:

1. To establish the PCOR CDS-LN to convene, charter, implement, and initiate operations of a learning network of multiple stakeholders with an interest in disseminating and enabling PCOR findings though CDS

2. To identify barriers and facilitators and formulate recommendations for enabling PCOR findings dissemination into clinical workflows using CDS

3. To monitor, evaluate, and adapt the PCOR CDS-LN to optimally support the dissemination of PCOR findings into clinical workflows via CDS

4. To develop a sustainability plan for ongoing operations

Effort began in April 2016 to achieve the second aim above: conduct and report an environmental scan of PCOR-based CDS to generate action-oriented activities that build value to the community; the public; and to the client, AHRQ. This report compiles results from researchers’ investigations into the activities and thinking in the emerging field of PCOR-based CDS.

The following purpose statement has been reviewed and approved by all members of the PCOR CDS-LN Senior Research Team (Drs. Blumenfeld, Osheroff, Middleton, Kawamoto, and Greenes):

The purpose of the environmental scan is to provide an updated picture of writings and beliefs regarding PCOR and CDS, the purpose of which is to examine the barriers and facilitators to the use of CDS as a vehicle for putting PCOR findings into practice.


5

In carrying out its purpose, PCOR CDS-LN co-chairs (Drs. Osheroff and Middleton) stressed that this environmental scan should be a “springboard for action” that helps fulfill the organization’s vision: to create an ecosystem that allows all stakeholders reduce the friction of turning knowledge from PCOR findings into CDS-enabled actions that produce better care and outcomes.

The intended goal of this environmental scan is to ensure that the PCOR CDS-LN begins with a comprehensive review of the current state, which includes: technologies and architectures; user needs; policy; and more. Intended outcomes of the environmental scan, which include what was originally proposed to AHRQ, are listed in Exhibit 2-1.

Exhibit 2-1. Intended Outcomes from the PCOR CDS-LN Environmental Scan

§ Propose a definition of PCOR-based CDS

§ Support efforts to assess the quality of PCOR evidence for CDS, such as assessments based on the reliability and quality of evidence-generating funding sources

§ Support efforts to provide a basis for assessing the implementability of CDS for disseminating PCOR

§ Inform and serve as the basis to formulate efforts prioritizing disseminating of PCOR

§ Provide a base of understanding with which to develop discussion guide(s) and offer listening sessions with PCOR CDS-LN members

§ Encourage and support the dissemination of materials including manuscripts, popular press, presentations, and posters

§ Identify approaches to evaluating PCOR-based CDS initiatives

§ Build a model of PCOR-based CDS that visually represents the field as well as where tacit and explicit knowledge reside

§ Identify findings that can inform how PCOR-based CDS developments and implementations that promote a learning health system

It is with the purpose, goal, and intended outcomes in mind that the environmental scan research team carried out its investigation: to make its findings and recommendations a “springboard for action” to disseminating PCOR evidence through CDS into clinical care.


6

3. Methods The effort for carrying out the environmental scan was highly iterative and collaborative.

3.1 Team The project Principal Investigator, Dr. Barry Blumenfeld, convened a research team (See Exhibit 3-1) comprised of RTI researchers and Senior Research Team members to plan, collect data, analyze data, and disseminate findings to multiple stakeholders: AHRQ, the PCOR CDS-LN Steering Committee, the PCOR CDS-LN Council, and general members and non-members of the PCOR CDS-LN. The research team was approved by the Senior Research Team, which existed before the PCOR CDS-LN Steering Committee. The team met bi-weekly to discuss preliminary findings and ongoing analyses.

Exhibit 3-1. Environmental Scan Research Team

Name Organization Role

Joshua Richardson, PhD, MS, MLIS RTI Lead Researcher

Melissa Callaham, MS RTI Researcher

Laura Marcial, PhD, MLIS RTI Researcher

Barry Blumenfeld, MD, MS RTI/PCOR CDS-LN Primary Investigator

Senior Advisor

Blackford Middleton, MD, MPH, MSc Apervita/PCOR CDS-LN Co-Chair Senior Advisor

Jerry Osheroff, MD TMIT Consulting, LLD/ PCOR CDS-LN Co-Chair

Senior Advisor

3.2 Environmental Scan Theoretical Basis In guiding the methodological approach, the team referred to an environmental scanning matrix developed by Choo.5 On the basis of a historical analysis of environmental scans, Choo built a 2 x 2 model for recommending how to best focus efforts (see Exhibit 3-2. The model represents how teams may have assumptions about the environment being studied, (Y axis) and the degree of organizational intrusiveness to be conducted (X axis). “Assumptions about the Environment” on the Y axis shows whether the team determines the environment being studied to be analyzable or analyzable. Analyzable areas are best explored through “conditioned viewing” of explicit sources such as literature databases and reports, whereas unanalyzable areas are best explored through “undirected viewing” of tacit


7

knowledge by way of interviews and discussions with personal connections. “Organizational Intrusiveness” on the X axis shows the degree of interaction with the external environment, labeled as passive or active (although “passive” should not be mistaken for an absence of action).

Exhibit 3-2. Environmental Scan Methodological Framework5

The research team determined that the most fruitful approach to data gathering and analysis for this project would be to engage in “Undirected Viewing” (by way of interviews) and “Conditioned Viewing” (by way of literature searches). These approaches would allow the research team to gather the tacit knowledge and explicit knowledge necessary given available time and resources. Future activities may entail “Enacting” and “Searching” as more PCOR-based CDS evidence and resources become available. The Senior Research Team approved both of the proposed approaches.

3.3 Timeline Over 6 months the research team carried out planning, data collection, analysis, and multiple stakeholder vetting; as well as a title-level literature scan. After delivery to AHRQ, the research team will disseminate tailored findings to various stakeholder groups by way of webinars, publications, and conference presentations.


8

3.4 Data Collection

3.4.1 Senior Research Team Discussion Group One area of tacit knowledge was gathered by way of an audio recorded 1-hour discussion with all of the PCOR CDS-LN Senior Research Team members on June 17, 2016, and on October 5, 2016. The first discussion addressed pre-planned questions including the ACA definition for PCOR in relation to PCOR-based CDS, any known past efforts to conceptualize the barriers and facilitators to PCOR-based CDS, and ultimately how the research team could use the environmental scan to lay the groundwork for PCOR CDS-LN strategies (see Exhibit 3-3). The second discussion addressed what the Senior Research Team determined were still open questions about relevant standards for PCOR-based CDS. Researchers took notes in Microsoft Word and analyzed text in qualitative analysis software (NVivo v. 11, QSI International, Inc.), and directly imported results from a group discussion about standards (see Section 4.2.2.3).

Exhibit 3-3. Senior Research Team Discussion Group Questions

§ What are the big problems in healthcare that need to be solved?

- Patients?

- Providers?

- Healthcare organizations and businesses?

§ What can PCOR implemented as CDS do to address those problems (short-term to long-term)?

§ What are the barriers and facilitators to disseminating PCOR via CDS?

§ Is PCOR within CDS any different than other clinical content within CDS?

§ Who are the people or groups with important perspectives on PCOR, CDS, and PCOR-based CDS that we want to engage for the environmental scan and beyond?

§ What can the PCOR CDS-LN do to help stakeholders disseminate PCOR via CDS?

A key outcome from the group discussion was a consensus agreement that there was a need for an analytical framework that could both bind the scope of the environmental scan and provide a common vocabulary that the PCOR CDS-LN community could use to better discuss PCOR-based CDS. The group also provided recommended citations of PCOR, CDS, and PCOR-based CDS that were to be added to a publicly available reading list. The research team then developed an analytical framework (see Section 4.1) and an online bibliography as discussed (see Section 4.2.1.3). Both are described in further detail below.


9

3.4.2 Individual Expert Interviews In addition to the Senior Research Team discussion, the environmental scan research team conducted 27 telephone interviews with 34 experts from late June 2016 to early August 2016. Experts came from multiple areas including PCOR, patient and family advocacy, CDS and electronic health record (EHR) development, provider care, and more. Interviews ranged in duration from 25 minutes to 75 minutes. Researchers asked questions from the Senior Research Team’s approved semi-structured interview guide (see Exhibit 3-4).

To frame interviews, we defined PCOR using AHRQ’s preferred definition from the ACA: “PCOR is comparative clinical effectiveness research of the impact on health outcomes of two or more preventive, diagnostic, treatment, or health care delivery approaches.” Researchers documented the answers in using Microsoft Word and then exported the text into qualitative analysis software (NVivo v. 11, QSI International, Inc.).

Exhibit 3-4. Semi-Structured Interview Questions

§ What is PCOR (reference ACA’s definition) and PCOR-based CDS?

§ How do you think CDS could best be used to disseminate PCOR findings to clinicians and patients?

§ What barriers do you think may make it difficult to disseminate PCOR through CDS?

§ What do you think would help make it easier to disseminate PCOR through CDS?

§ What criteria would you use for evaluating the quality of PCOR that goes into CDS?

§ In your opinion, what is an important PCOR finding (individual OR population) that should or needs to be disseminated via CDS?

§ Who, if anyone, is doing notable work in PCOR for CDS, for example people, companies, institutions?

3.4.3 Feedback on the PCOR CDS-LN Analytic Framework for Action Of particular note, all interviewees were asked if they would review and comment on a draft version of the PCOR CDS-LN Analytic Framework for Action (AFA), which had been under development since the initial Senior Research Team group discussion (June 17, 2016). All interviewees answered that they would be willing to review and comment upon the draft. Of 34 interviewees, 7 provided feedback (those providing feedback included an informatics researcher, a government representative, a provider, and a patients and families representative). Using their feedback, the research team made significant updates to the AFA and its associated definitions.


10

3.5 Data Collection—Peer-Reviewed Literature and Gray Literature

Explicit knowledge was gathered by accessing artifacts made available to the team from a variety of sources: Senior Research Team members’ personal citation libraries, PubMed, citations posted the website of the Patient-Centered Outcomes Research Institute (PCORI), Google and Google Scholar, white papers and various other documents. In all, one researcher (Richardson) independently collected, vetted, and organized over 300 citations he deemed relevant for the PCOR CDS-LN by carrying out a title review. Citations were organized according to the AFA (see Section 4.1) and have been made publicly available on the internet (https://www.zotero.org/groups/pcor_cds-ln_envscan/items). While in the process of collecting peer-reviewed literature and gray literature, the team engaged an expert literature searcher from the National Library of Medicine to refine a search strategy for PCOR-based CDS (given no such strategy yet existed, to the team’s knowledge). See Appendix A for the vetted search strategy.

3.6 Data Analysis The lead researcher conducted a thematic analysis of the group discussion and individual interviews. First, notes of individual answers to questions were imported into qualitative analysis software (NVivo v. 11, QSR International, Inc.). Second, after an initial review and discussion, answers were interpreted as themes relating to barriers and facilitators as according to facets in the AFA.

As previously stated, citations were vetted and organized by a single researcher (Richardson) according to the AFA. Time and and resources did not allow exhaustive review or coding of abstracts and full-text. However, many of the abstracts and some full-text were reviewed by the principal author and research team. In the case of citations originating from the Senior Researchers, many of the papers were familiar to them.

3.7 Limitations As noted above, literature review consisted of article reviews at the title level. This was done in part because of time and effort it took to develop a search strategy for PCOR-based CDS (see Appendix A). Future work through work groups will involve further vetting and updating of the bibliography as existing and new materials on PCOR-based CDS become available, and analysis at at the abstract and full-text levels. Second, time with interviewees was limited and we may not have delved as deep into PCOR-based CDS barriers and facilitators as the researchers may have liked. For that reason, we limited the number of questions asked of each interviewee. However, we believe the experts were engaged in all discussions and


11

provided valuable insights during the time they so graciously donated. We thank all who participated and are immensely grateful to them.


12

4. Results Through data collection and analysis of tacit knowledge (Senior Research Team discussion group and interviews) as well as explicit knowledge (published materials) we have generated results to guide the PCOR CDS-LN:

§ AFA § PCOR-based CDS Barriers and Facilitators organized by the AFA

4.1 PCOR CDS-LN Analytic Framework for Action The AFA (see Exhibit 4-1) was developed and iteratively refined as an outgrowth of developing this environmental scan and of work establishing the PCOR CDS-LN with stakeholders such as the Steering Committee. It is designed to provide a framework for understanding the current state of applying PCOR-based CDS to improving care processes and outcomes (e.g., by organizing pertinent insights and literature as summarized in this environmental scan) and outlining action areas for achieving the PCOR CDS-LN’s goal of facilitating evolution of this current state to one where PCOR findings drive CDS-enabled care improvements much more efficiently and effectively.

Exhibit 4-1. PCOR CDS Analytic Framework for Action

We provide definitions for each of the factors in Exhibit 4-1 above:

1. Addressing External Factors: External factors including the marketplace, policy, legal, and governance issues that impact development, dissemination, and implementation processes for PCOR-based CDS


13

2. Prioritizing PCOR Findings for Dissemination via CDS: Applying objective measures of evidence for identifying and prioritizing PCOR findings that are to be transformed and disseminated via PCOR-based CDS, assessing or defining their implementability, and defining stewardship and governance requirements.

3. Authoring CDS Interventions: Applying accepted data and knowledge standards for translating PCOR findings into one or more CDS intervention types that support key decisions, actions, and communications that are essential to ensuring that the finding improves care and outcomes.

4. Implementing CDS Interventions: Applying standardized, best practice methods and architectures for operationalizing PCOR-based CDS interventions into clinical workflows that deliver the right information to the right user in the right format through the right channel at the right time ("CDS Five Rights").

5. Measuring Decisions and Outcomes: Ensuring that PCOR-based CDS interventions measurably improve clinician and patient decision-making, care processes, and outcomes.

6. Learning from PCOR-Based CDS Experience: Aggregating local PCOR-based CDS-related outcomes and effectiveness measures to facilitate both local and system-level learning from identified gaps in PCOR knowledge, and lessons learned from authoring, implementing, and using PCOR-enabled CDS in clinical practice to enhance care and outcomes.

4.2 Potential Barriers and Facilitators to PCOR-Based CDS

4.2.1 Addressing External Factors

Definition: External factors include the marketplace, policy, legal, and governance factors that impact development, dissemination, and implementation processes for PCOR-based CDS.

4.2.1.1 Potential Barrier: Defining and Describing PCOR-Based CDS Based on our discussions, we found the term, “PCOR-based CDS,” was a barrier in itself. Of note, our question about how CDS could best disseminate PCOR received the fewest numbers of documented answers (14 out of 26 interviews) and no recorded comments from patients and families stakeholders. Multiple interviewees from different fields (providers, informatics researchers, and government representatives) explicitly stated they didn’t know what PCOR-based CDS would look like, and that if PCOR-based CDS was another term for CER then there would be no difference from past CDS. Other discussants posited that PCOR-based CDS


14

would do well to encompass, if not encourage, patient participation in the care they receive (such as patient-reported outcomes), and focus on patient data types that may not be a primary focus of existing data and vocabulary standards (such as recording social determinants data). Without conceptual tools such as specific examples or use cases, people seem to lack a mental model of PCOR-based CDS with which to robustly engage on the topic. We offer the following as a working definition of PCOR-based CDS:

“PCOR-based CDS helps patients and their care teams apply evidence from patient-centered outcomes research to enhance care processes and their results. Approaches include promoting shared decision-making, incorporating patient reported outcomes, factoring in patient preferences to generate patient-specific recommendations for care and others.”

Recommendation: Implement a use case aligned with the AFA to provide an example for putting PCOR into action through CDS.

4.2.1.2 Potential Barrier: Access to Literature Around PCOR-Based CDS Many interviewees explained that they did not know of any specific sources for learning more about PCOR-based CDS. To that end, we have posted an online bibliography of literature composed of citations from the PCORI publications website and recommended citations from the PCOR CDS-LN Senior Research Team (https://www.zotero.org/groups/pcor_cds-ln_envscan/items). As of October 10, 2016, the online bibliography contains 322 citations organized according to the AFA (see Appendix D).

Exhibit 4-2. PCOR CDS-LN Online Bibliography


15

Discussion with a National Library of Medicine (NLM) search expert confirmed that there is not a PCOR-based CDS subject heading (e.g., a Medical Subject Heading) to help identify relevant literature, and the term “patient-centered outcomes research” may not adequately capture relevant citations in this area. Therefore, in collaboration with an NLM searcher, we developed a search term for identifying relevant PCOR-based CDS literature in PubMed (see Appendix A).

Recommendation: Allocate resources for fully curating the online bibliography organized by the AFA and routinely distribute relevant citations to subscribing members.

4.2.1.3 Potential Barrier: Costs and Competing Priorities Some interviewees noted that implementing PCOR-based CDS could be difficult at this time given that the marketplace may not be able to support development and deployment costs, particularly in light of customers who are adjusting to demands in the regulatory environment. Two interviewees highlighted the challenges healthcare organizations are face in implementing PCMHs, ACOs, Meaningful Use or MACRA. For them, PCOR, let alone PCOR-based CDS, is not a priority nor a focus in their organizations at this time. The challenge is heightened when considering the economics of CDS development in general. One interviewee argued that the financial model is such that developing content in-house is neither feasible nor affordable, and if content were to be made available externally, hospitals and practices still have limited means to afford the content and expertise to integrate decision support artifacts into local systems.

Recommendation: Conduct a high-priority use case that aligns development and implementation costs with healthcare organizations’ priorities and means for implementing standardized PCOR-based CDS interventions.

4.2.1.4 Potential Barrier: Levels of Patient Health Literacy for PCOR and PCOR-Based CDS

Some interviewees pointed out that poor health literacy among patients could be a barrier to PCOR-based CDS. A patient and family advocate explained that outreach materials come from researchers and providers who sometimes inadequately capture the patient perspective. For providers, there may be unfamiliarity with effective ways to integrate PCOR findings into their decision-making processes and patient engagement. For researchers, it is not being able to communicate to stakeholders beyond academic technical jargon. The takeaway is that improving patient-centered health literacy among providers and researchers is a potential factor for effectively implementing effective PCOR-based CDS.


16

Recommendation: Disseminate research that addresses any ways low health literacy may have on patients’ ability to successfully engage providers within the context of PCOR-based CDS.

4.2.2 Prioritizing PCOR Findings for Dissemination via CDS

Definition: Applying objective measures of evidence for identifying and prioritizing PCOR findings that are to be transformed and disseminated via PCOR-based CDS, assessing or defining their implementability, and defining stewardship and governance requirements.

4.2.2.1 Potential Barrier: Identifying PCOR to Be Disseminated via CDS Interviewees suggested areas of focus for PCOR-based CDS including mental health, medication management, and genetic-based conditions such as a family history of breast cancer (see Appendix C for the complete list). But most seemed to find it difficult to specify any one PCOR finding that should be a high priority for dissemination through CDS.

Discussions with PCOR experts revealed that PCORI recognizes that disseminating PCOR evidence is a challenge, and for that, PCORI is organizing a committee to focus on that effort. In a similar vein, AHRQ is supporting efforts to make available an online repository of PCOR for dissemination through CDS. A PCOR expert stated that it would be useful for AHRQ to distill comparative effectiveness reviews (screening, diagnostic, and treating) and grade them so to prioritize areas. For example, Baylor’s Eisenberg Center has been working on consumer and physician and sometimes policymaker pamphlets. These are paper-based but could potentially be useful in electronic CDS.

Recommendation: Establish official cross-collaborations with AHRQ and PCORI to leverage their initiatives for advancing PCOR-based CDS.

4.2.2.2 Potential Barrier: Access to PCOR for PCOR-Based CDS Discussants directed the research team to seven online PCOR repositories but most appeared uncertain as to where or how to access PCOR. The online resources mentioned were the following:

§ Institute of Medicine (IOM) § United States Preventative Services Task Force (USPSTF) § PCORI


17

§ AHRQ Guidelines.gov § Cochrane Collaboration § NLM's National Information Center for Health Services Research and

Health Care Technology (NICHSR) § Individual Professional societies

As previously mentioned, AHRQ is actively developing an online resource to make available PCOR evidence for dissemination via PCOR-based CDS.

Recommendation: Establish and maintain cross-collaborations with AHRQ and MITRE in the development of a PCOR-based CDS repository.

4.2.2.3 Potential Barrier: Effective Means for Evaluating PCOR for PCOR-Based CDS

Discussants did not come to agreement around any one tool or method for evaluating the appropriateness of PCOR to be disseminated through CDS. A patients and families Representative suggested it was due to the limitation of PCOR’s definition itself, which rarely incorporates social determinants (see Exhibit 4-3). A PCOR expert stated evaluating would be very hard because there is rarely a single definitive study and would be hard to sell nationally. Adding to the challenge, one EHR vendor representative stated that implementing PCOR findings is not a problem that clients are trying to solve. Depending on their perspective, discussants commented on evaluating the quality of the PCOR and others commented on evaluating the quality of CDS (see Appendix B for the full list).

Recommendation: Task a Barriers and Facilitators Work Group with developing the means for evaluating the implementability of PCOR-based CDS.

Exhibit 4-3. Sample of Approaches to Evaluating PCOR by Stakeholder Type

Stakeholder Approach Patients and Families Representatives

Did the research apply characteristics from Arnstein’s Ladder of Citizen Participation6 (i.e. informing, placating, or partnership)?

PCOR Experts What measures were used and were they based on contemporary measurement theory (i.e. item-response, rational/non-rational)?7

EHR Vendors To what degree did a patient’s voice get represented in the evaluation process: qualitative approaches, focus groups, interviews, surveys?

4.2.2.4 Potential Barrier: Effective Means for Determining the Level of Readiness for PCOR to Be Incorporated into CDS (“Implementability”)

Not all PCOR findings will be amenable to being implemented as CDS yet communities lack assessment tools for knowing which findings may be easier to


18

implement than others. This is a critical gap that will likely hinder future attempts at integrating PCOR-based CDS into clinical workflows.

Recommendation: Work with the PCOR CDS-LN community and partnering stakeholders to determine how implementation readiness can be addressed.

4.2.2.5 Potential Barrier: PCOR-Based CDS Not Aligned with Payer Priorities Another finding may be that PCOR does not align with what payers are will to finance. Payer decisions are not based on the evidence alone; the evidence is one of many factors (cost being a core factor) taken into account in payers’ overall decision-making processes.

Recommendation: Recruit a payer to provide his or her perspective toward a use case in a Barriers and Facilitators Work Group.

4.2.3 Authoring CDS Interventions

Definition: Applying consensus-based data and knowledge standards for translating PCOR findings into CDS interventions that support comparative or patient-centered decision-making (i.e. risk calculators, cognitive aides).

4.2.3.1 Potential Barrier: Previously Known CDS Barriers to Authoring Experts pointed to well-known CDS-related challenges to authoring such as gaining access to well-formatted data and applying standards-based terminologies, particularly if PCOR-based CDS means supporting patient-specific decisions. For example, one provider noted that ICD codes lack the necessary data elements to effectively capture social determinants to health (i.e. housing status or housing stability) for treating patients in community health settings. For that reason, community health centers are taking it upon themselves to develop their own data standards and vocabularies (see the PRAPARE Project at http://nachc.org/research-and-data/prapare/). No single terminology standard exists that is generally agreed upon for representing patient level findings and outcomes; in fact, several competing alternatives exist. Another challenge is that if PCOR-based CDS is to support patient-specific decisions including incorporating data around individual values and preferences, then that can raise the specter of increased numbers of inappropriate reminders or alerts firing due to faulty or a lack of data. One informatics researcher noted this is an unsolved challenge, and likened ideal PCOR-


19

based CDS it to an “80/20” balance: 80% best available evidence and 20% patient values and preferences.

Recommendation: Task a Technical Standards Work Group for documenting and disseminating solutions for authoring PCOR-based CDS.

4.2.3.2 Potential Barrier: Access to PCOR-Related Data As a patient and family representative commented, patient-generated data are key to achieving patient-centered care, and could be key to achieving meaningful PCOR-based CDS. Experts in PCOR, EHRs, and informatics research the field should build CDS tools from there, starting with the data and building out.

Experts described activities directed around data issues that will be of interest to PCOR CDS-LN stakeholders. PCORI, for example, requires researchers make their data sharable and then share their data has data management and data sharing written into its contracts8 (see Exhibit 4-4). Similarly, the expert noted that PCORnet and Patient-Powered Research Networks (PPRNs) are working to rapidly cycle data from collection to dissemination to potentially answer real-world clinical questions. As more funding agencies mandate researchers make their project data available to the public, more data will likely become available to PCOR-based CDS developers. However, issues of data stewardship too will be of likely concern yet none of the experts had ready answers for addressing the issue.

Exhibit 4-4. PCORI Standard Contract Language for Funded Research

“PCORI encourages openness in research and making research data available for purposes of replication and reproducibility. Recipient shall develop and maintain a plan that addresses data management and data sharing of Research Project data in a manner that is appropriate for the nature of the Research Project and the types of Research Project data, and that is consistent with applicable privacy, confidentiality, and other legal requirements.”

Some interviewees pointed to patient registries as effective data sources and tools because of how they organize data for primarily two different purposes: research and CDS. However, the field still needs to better delineate the differences between research and disease or CDS registries like the one being developed by the American College of Radiology. Better definitions and explanations of what registries are and how they are to be used has also been an area of PCORI’s work. One PCOR expert was encouraged by the development of registries and noted there is a proliferation of specialty-specific registries underway.


20

Gaining access to comprehensive data from EHRs, and data subsets such as those in C-CDAs is key and depends on systems being interoperable and being able to access, cleanse, and normalize relevant data. Artificial intelligence is at a stage that it can make interesting discoveries out of the data. Access to more data, particularly from patients about what patients want, means novel approaches to calculating risk.

Recommendation: Post relevant data sources for use in PCOR-based CDS implementations.

4.2.3.3 Potential Barrier: Facilitating Provider-Entered and Patient-Entered Data for PCOR-Based CDS

Since PCOR-based CDS depends on data, it could be more effective if the CDS was developed in ways that supported providers in systematically collecting condition-specific data. For example, one payer has reportedly been experimenting with collecting patient survey data around functional status to determine utilization decisions for knee and hip replacement. This “collaborative survey” approach is bringing the patient more effectively into the care process by actively tracking and sharing their status with providers. The result could ultimately lead to clinical decisions that are more salient to an individual’s patient needs, preferences, and values. The payer can also weigh those factors into to the extent to which it can and will authorize potential procedures. However, a technology vendor representative commented on the difficulty of getting patients to share their health, location, and activity data with healthcare providers and technology companies despite consumers’ seeming comfort with sharing many other details about their personal lives.

Recommendation: Publish best practices that address data privacy and security issues for PCOR-based CDS.

4.2.3.4 Potential Barrier: Access to Reliable PCOR Measures When Authoring PCOR-Based CDS

One interviewee with expertise in quality measurement noted the importance of, yet difficulty with, operationalizing reliable patient-reported measures and integrating any resulting data into EHRs. For example, Epic is incorporating PROMIS, which is a set of person-centered measures for evaluating patient health, to capture highly precise patient measurements.7 Similarly, a vendor is working with care delivery organizations, payers, and quality organizations to incorporate PHQ-9. Effort needs to be made to identify ways that patient-centeredness can be measured so that implementers can judge what PCOR is appropriate or not appropriate for CDS.


21

Recommendation: Publish a library of approved patient-reported data sets and measures.

4.2.3.5 Potential Barrier: Authoring PCOR-Based CDS Tools That Are Patient-Specific and Incorporate Shared Decision-Making, Patient Perspectives, and Patient-Reported Outcomes

Interviewees pointed to areas of work that could be beneficial PCOR-enabled CDS, and shifting focus from disease to health could drive new ways of integrating PCOR into CDS. One area of research that may grow as a result is predictive algorithms to guide decision-making (although PCOR-based CDS would not be the only driver for predictive analytics). However, it provides some focus around the need for granular data to open up a range of decision possibilities, particularly if patients were given means to electronically weigh in on the course of their own care.

One example provided was current work on hospital-acquired pulmonary embolisms (PEs). The work entails identifying a subset of patients who normally do not show signs and have low risk of PE, so much so that the subset would not require inpatient monitoring and instead may be able to go home for a watchful waiting approach. The interviewee noted that more patient-specific data is necessary for them to accurately identify the subset at low risk for PE and therefore eligible for home monitoring.

Another area pointed to was that of patient-facing and provider-facing cognitive aides, which empower patients and providers to visually compare potential outcomes based on data-driven risk calculations. A third area is engaging patients via mobile apps. Patient and family representatives cautioned, however, that apps should be designed in ways or at levels that support health literacy levels and language (including multi-language) needs. Visualizations are important means to communicating health concepts to patients, and collaborating with health coaches to author patient-centered interventions would be one way to inform designs for PCOR-enabled CDS.

Recommendation: Highlight novel areas of work, such as methods for assessing patient preferences in PCOR-based CDS, within a use case.

4.2.3.6 Potential Barrier: Need for Further Development, Validation, and Adoption of Standards Required for Interoperable PCOR-Based CDS

For PCOR-based CDS to be widely implemented at a reasonable cost, interoperability standards are required for sharing CDS interventions efficiently across institutions and health information technology (IT) platforms. One important problem is that standards required for such interoperable PCOR-based CDS will require development, validation, and adoption among stakeholders (e.g., vendors,


22

healthcare organizations) who have competing business requirements. Below is a sample summary of the current state of standards in areas required for interoperable PCOR-based CDS (see Appendix D for full table).

Exhibit 4-5. Sample of CDS Current State and Needs

Standards Category Current State

Development and Validation Need

Foundation—Orderable Catalog

There is no standard orderable catalog, limiting interoperability potential for CDS utilizing orders.

Resources to develop national orderable catalog

Foundation—Logical Expression Language

HL7 Clinical Quality Language (CQL)—developed based on requirements from various earlier standards, including Arden Syntax, GELLO, etc. CMS is adopting for quality measurement programs. Standard itself is fairly mature.

Vendor validation and adoption for CDS

Recommendation: Task a standards work group to address data standards, data and knowledge representation, and vendor adoption (e.g., simplified profiles with functionality that can be supported without the need for vendors to implement new capabilities).


23

4.2.4 Implementing CDS Interventions

Definition: Applying standardized methods and architectures for operationalizing PCOR-based CDS interventions into clinical workflows, which deliver the right information to the right people in the right formats through the right channels at the right times ("CDS Five Rights").

Guidance on improving care processes and outcomes with CDS have been published in journal articles,9 guidebooks,10,11 and more recently in federally sponsored websites from ONC12 and HRSA.13 While organizations increasingly turn to such resources for help in successfully developing implementing CDS-enabled quality improvement strategies, many obstacles remain.

4.2.4.1 Potential Barrier: Previously Known Barriers to Implementing CDS Interventions

Experts pointed to well-known CDS-related challenges to potentially implementing PCOR-enabled CDS:

§ Lack of time § Provider resistance to health IT adoption broadly, perhaps because of

extra work, altered workflows, and other unintended consequences of health IT

§ Too many reminders and alerts § Disruptive to clinical workflows and context without effective means to

customize § Insufficient means to presenting data to providers and patients that are

relevant and actionable § Difficult authoring tools in host EHR systems, and limited knowledge

sharing capabilities § Limited host EMR functionality for advanced CDS methods § Limited ability to use external services providing CDS § Difficult to link some CDS interventions to actions in the clinical workflow

(i.e., actionable CDS)

Recommendation: Identify known implementation strategies and tools for improving care delivery that are transferable to PCOR-based CDS.


24

4.2.4.2 Potential Barrier: Lack of User-Centered Research on PCOR-Enabled CDS

Interviewees noted that it could be difficult making PCOR actionable through CDS because of a persistent lack of understanding around patient and clinician needs and the lack of an infrastructure to turn PCOR into something actionable. This issue could be particularly daunting for patient-facing PCOR-based CDS meant to help a patient not only understand their health status but then to take action (or no action) based on best available evidence. That support should ideally be readily available to patients in and away from clinical settings, yet relatively little is still known how to implement in either arena.

Recommendation: Call for e-journal articles and webinars on usability issues as they pertain to PCOR-based CDS.

4.2.5 Measuring Decisions and Outcomes

Definition: Ensuring that PCOR-based CDS interventions measurably improve clinician and patient decision-making, care processes, and outcomes.

4.2.5.1 Potential Barrier: Determining Optimal Metrics for the Ways PCOR-Based CDS Impacts Decision-Making and Patient Care

Prioritizing, authoring, and implementing PCOR-based CDS alone is not the goal nor the vision of the PCOR CDS-LN. The goal is to measurably improve care and outcomes by way of PCOR-based CDS. Therefore, the issue is how can PCOR-based CDS measurably demonstrate effects (whether positive or negative) on patient outcomes.

First, an infrastructure needs to be in place that enables synthesizing PCOR evidence into CDS tools. Second, those tools need to be designed and implemented in ways that support workflows including provider-patient discussion strategies (ways to discuss what the evidence means and the kinds of decisions that could result from that). Third, PCOR-based CDS tools need to be tied to organizational quality strategies. And fourth, the effects of PCOR-based CDS need to be measured so that the evidence and performance of the tools can be modified as needed.

Recommendation: Publish a use case that identifies barriers and opportunities for measuring the effects of PCOR-based CDS.


25

4.2.6 Learning from PCOR-Based CDS Experience

Definition: Aggregating local PCOR-based CDS-related outcomes and effectiveness measures to facilitate system level learning from identified gaps in PCOR knowledge, CDS-enabled clinical practice, and patient outcomes.

4.2.6.1 Potential Barrier: Integrating PCOR-Based CDS into a Learning Health System

The PCOR CDS-LN Senior Research Team sees it as an imperative that data and evidence from and related to PCOR-based CDS be fed back into the research process to create a true learning health system. One interviewee, however, wondered if medical culture is yet at a point where research (including PCOR) and clinical care can co-exist to drive systems-level learning. Therefore, making PCOR-based CDS actionable within a Learning Health System may be a very ambitious goal.

Toward this end, the PCOR CDS-LN could begin and sustain dialogs among stakeholders within its growing membership. A provider commented that they hoped to see substantive engagement between leaders in PCOR and CDS. And a payer commented he would like to learn more about PCOR-enabled CDS and could see value in the PCOR CDS-LN if it helped him learn how payers could share lessons learned and ultimately promote PCOR-based CDS uptake among providers. Opportunities to collaborate abound. For example, the research team engaged in conversations with a representative from the The Learning Health Community (http://www.learninghealth.org/). Also, NLM is eager to engage and has resources, interest, and people focused on CDS and CER. For example, it makes available materials directly relevant to CER and other tools through NICHSR (https://www.nlm.nih.gov/hsrinfo/).

Recommendation: Engage stakeholders in investigating ways that actions and outcomes from PCOR-based CDS can contribute to a learning health system ecosystem.


26

5. Discussion The findings provide the PCOR CDS-LN in specific, and the larger field of PCOR-based CDS in general, with areas for action as outlined below.

Interviews and literature searches revealed a need for generating a definition of PCOR-based CDS to help stakeholders frame discussions around developing and disseminating PCOR through CDS. In addition, feedback indicated a need for a definition that incorporates elements of CER as well as factors such as patient values and preferences into CDS. We have proposed the following working definition:

PCOR-based CDS helps patients and their care teams apply evidence from patient-centered outcomes research to enhance care processes and their results. Approaches include promoting shared decision-making, incorporating patient reported outcomes, factoring in patient preferences to generate patient-specific recommendations for care and others.

A second action that resulted from this environmental scan is the development of the AFA and accompanying definitions. The AFA provides a means for conceptualizing a generalizable approach to investigating and analyzing critical areas in the identification, development, implementation, and evaluation of PCOR-based CDS. We believe the AFA will be a critical tool for communicating to various stakeholder groups – within and beyond the PCOR CDS-LN – where challenges and opportunities lie for collaboration and common action.

Exhibit 5-1: The PCOR CDS-LN Analytic Framework for Action


27

Interviewees suggested and supported the PCOR CDS-LN developing a compelling use case for helping make the case more concrete. Areas of particular interest (and challenge) would be to estimate development and implementation costs of PCOR-based CDS within that use case. Yet another challenge would be to select a use case that matches or outweighs existing priorities that healthcare organizations currently face (e.g., Meaningful Use, MACRA, PCMHs). Here, involving the payer perspective from the outset would be extremely valuable.

Another area of challenge and opportunity is developing ways that results from applying PCOR findings via CDS interventions can be fed back to support a learning health system. Results show that there are promising connections to be made between the PCOR CDS-LN through its AFA and organizations specifically focused on PCOR (e.g. PCORI) and a learning health (e.g., the Learning Health Community). By fostering dialog among these groups and others, the PCOR CDS-LN can be a valuable partner in developing the structures and processes for translating PCOR evidence into practice in a manner that supports better outcomes and more efficient learning.

One area of particular area of interest across communities will be how PCOR data are generated, formatted, and plugged in to provider-facing and patient-facing CDS. In this respect, the stakeholders that the PCOR CDS-LN are to represent will undoubtedly have valuable input to share.

An online knowledge management platform with the ability to effectively document, track, and disseminate PCOR-based CDS knowledge will be critical for success. The PCOR CDS-LN will do well to continue the efforts already underway for establishing its online presence so to gather input from numerous stakeholders, share knowledge developed through its efforts (like this environmental scan), and then be able to make explicit connections to external efforts in PCOR CDS-LN development: AHRQ repository, PCORI’s efforts based on its Evaluation Framework 2.0, and others. Linking relevant data and knowledge resources to promote CDS authoring environments in business, research, and even individual patients and citizen scientists, would be a valuable resource to multiple communities.

The results of this environmental scan have led to the development of a working definition of PCOR-based CDS, the AFA, and recommended actions that the PCOR CDS-LN can take to engage stakeholders within and beyond its growing community.


28

6. References 1. AHRQ Patient-Centered Outcomes Research Clinical Decision Support Learning

Network (U18) | Agency for Healthcare Research & Quality [Internet]. 2015 [cited 2016 Oct 13]. Available from: http://www.ahrq.gov/funding/fund-opps/rfahs15003faq.html

2. Frequently Asked Questions About the Selection Process for AHRQ Dissemination and Implementation Initiative | Agency for Healthcare Research & Quality [Internet]. 2016 [cited 2016 Oct 13]. Available from: http://www.ahrq.gov/cpi/initiatives/pcortf/pcortf-faq.html

3. Patient-Centered Outcomes Research [Internet]. Patient-Centered Outcomes Research Institute. 2013 [cited 2016 Oct 27]. Available from: http://www.pcori.org/research-results/patient-centered-outcomes-research

4. Optimizing strategies for clinical decision support (Meeting no. 1): Meeting of the Leadership Consortium for Value & Science-Driven Health Care sponsored by the Office of the National Coordinator for Health IT [Internet]. Washington, D.C.: The National Academy of Sciences; 2016 Mar. Available from: https://nam.edu/event/namonc-meeting-optimizing-strategies-for-clinical-decision-support-meeting-no-1/

5. Environmental scanning as information seeking and organizational learning [Internet]. [cited 2016 May 3]. Available from: http://www.informationr.net/ir/7-1/paper112.html

6. Arnstein SR. A Ladder Of Citizen Participation. J Am Inst Plann. 1969 Jul;35(4):216–24.

7. PROMIS [Internet]. [cited 2016 Sep 3]. Available from: http://www.healthmeasures.net/explore-measurement-systems/promis

8. Patient-Centered Outcomes Research Institute: Contract for Funded Research Projects Standard [Internet]. Patient-Centered Outcomes Research Institute; 2016 Apr [cited 2016 Aug 19]. Report No.: CR8. Available from: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwjCvP3opfTOAhVMOiYKHewKB8MQFgghMAA&url=http%3A%2F%2Fwww.pcori.org%2Fsites%2Fdefault%2Ffiles%2FPCORI-PFA-Standard-Contract-for-Funded-Research-Projects.pdf&usg=AFQjCNGZ7v0GHMdWJKWk_REZgkZ99iTORA&sig2=2O9aemhE4aM_quulLhg1qg

9. Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, et al. Ten Commandments for Effective Clinical Decision Support: Making the Practice of Evidence-based Medicine a Reality. J Am Med Inform Assoc. 2003 Nov;10(6):523–30.


29

10. Greenes RA, editor. Clinical decision support: the road to broad adoption. 2nd ed. Amsterdam : Boston: Academic Press; 2014. 887 p.

11. Osheroff JA, Teich J, Levick D, Saldana L, Velasco F, Sittig DF, et al. Improving outcomes with clinical decision support: an implementer’s guide. 2nd ed. Chicago, IL: HIMSS; 2012. 323 p.

12. Osheroff JA. Planning and Implementing Improved Care Processes | Providers & Professionals | HealthIT.gov [Internet]. 2015 [cited 2016 Oct 7]. Available from: https://www.healthit.gov/providers-professionals/planning-and-implementing-improved-care-processes

13. Osheroff JA. HITEQ Center - Guide to Improving Care Processes and Outcomes in Health Centers [Internet]. 2016 [cited 2016 Oct 7]. Available from: http://hiteqcenter.org/Resources/HealthITEnabledQI/ImprovingPerformance/FQHCProcessesandOutcomes/TabId/152/ArtMID/919/ArticleID/108/Guide-to-Improving-Care-Processes-and-Outcomes-in-Health-Centers.aspx

B-1

Appendix A: PubMed Search Strategy (PCOR[tiab] OR "Patient-Centered Care"[Mesh] OR "Patient Centered Care"[tiab] OR "Patient Outcome Assessment"[Mesh] OR "Comparative Effectiveness Research"[Mesh] OR "Precision Medicine"[Mesh]) AND ("Decision Support Systems, Clinical"[Mesh] OR "Decision Making, Computer-Assisted"[Mesh] OR "Decision Support Techniques"[Mesh] OR "clinical decision support"[tiab] OR CDS[tiab])

882 Results on October 27, 2016

C-1

Appendix B: Expert Suggestions to Evaluating PCOR

Stakeholder Means

Patients & Families Identify the degrees to which patients were engaged in developing and conducting the research being disseminated.

Patients & Families

Determine whether the research applied Arnstein’s “Ladder of Citizen Participation” such as: Informing, Placating, or Partnership. http://lithgow-schmidt.dk/sherry-arnstein/ladder-of-citizen-participation.html

Patients & Families Identify whether real patients were used instead of patient proxies or surrogates.

Patients & Families Determine any recommendation’s impact for the specific context of the patient.

Patients & Families Determine the degree to which the research removes patient barriers.

Patients & Families Determine how well patients understood the recommendation.

Patients & Families Develop and determine a way to measure the degree of a recommendation’s actionability.

Patients & Families Determine the feasibility for patients to follow any recommendation.

PCOR Experts Specify the tools that were used to measure any patient actions and make clear whether or not those tools are based on contemporary measurement theory, i.e. item-response, rational/non-rational.

EHR Vendors & Informatics Determine the degree to which patient voice gets represented in an evaluation process: qualitative approaches, focus groups, interviews, surveys

PCOR Experts Determine the degree to which any conditions being studied answer questions that “real” people care about: mortality, cost.

D-1

Appendix C: Experts’ Suggested Conditions for PCOR-Based CDS

Health Care Delivery

End of life therapies, hospice

Highest cost conditions not being optimally treated: asthma, diabetes, hypertension

Mental and physical are the way to go, and align with payer priorities. Go to where the measures are at: pediatrics, OB/GYN, PHQ-9

Nurses doing discharge planning and the education that goes on at that time

Medication Management

Antibiotics use orally vs IV

Geriatrics where there's not a lot of evidence as to meaningful outcomes for older patients, and helping older adults getting off meds over time (reducing polypharmacy)

Wasteful and unnecessary or obsolete care would be high on the list…and is a "huge problem." Overuse of antibiotics would be a good example. Drug replacement would be another

Mental Health

Alzheimer's more screen at point of care

Focus more research for disseminating patient-centered care for pediatric mental health (autism, anxiety, eating disorders, etc.)

Mental health

Mental health based on data from patient-reported outcomes collected from kiosks or PHRs

Physical Conditions

BRCA gene mastectomy

Breast cancer, getting to a point where treatments are customized

Cardiology: CAD, HF


2

Focus more research for disseminating patient-centered care for pediatric mental health (autism, anxiety, eating disorders, etc.)

Hip & Knee arthroplasty: outcomes, provider performance

Incorporating decision-making around high-stakes procedures or screening tests, particularly where there is unclear data, i.e. mammographies in younger women, PSA tests. Also, decisions around elective procedures, i.e. prophylactic mammography for BRCA patients Low Back Pain - watchful waiting vs. prescribing

Macrolides in ophthalmology

Patient-reported outcomes using specific measures from the domains of pain, fatigue, sleep

Sepsis

E-1

Appendix D: PCOR-Based CDS Bibliography

1. Addressing External Factors

1.1 Marketplace Health Level 7. HL7 Virtual Medical Record (vMR) Project Wiki. 2012.

Kale MS. “Top 5” Lists Top $5 Billion. Archives of Internal Medicine. 2011 Nov 14;171(20):1858.

Mandl KD, Kohane IS. Time for a Patient-Driven Health Information Economy? N Engl J Med. 2016 Jan 21;374(3):205–8.

Walker J, Pan E, Johnston D, Adler-Milstein J, Bates DW, Middleton B. The value of health care information exchange and interoperability. Health Aff (Millwood). 2005 Jun;Suppl Web Exclusives:W5-10-W5-18.

1.2 Policy Beal AC. The Patient-Centered Outcomes Research Institute (PCORI) National Priorities for Research and Initial Research Agenda. JAMA. 2012 Apr 18;307(15):1583.

Borden WB, Chiang Y-P, Kronick R. Bringing Patient-Centered Outcomes Research to Life. Value Health. 2015 Jun;18(4):355–7.

Burwell SM. Setting value-based payment goals—HHS efforts to improve U.S. health care. New England Journal of Medicine. 2015 Mar 5;372:897–9.

Casalino L, Gillies RR, Shortell SM, Schmittdiel JA, Bodenheimer T, Robinson JC, et al. External incentives, information technology, and organized processes to improve

Denecke K, Spreckelsen C. Personalized medicine and the need for decision support systems. Stud Health Technol Inform. 2013;186:41–5.

Herr TM, Bielinski SJ, Bottinger E, Brautbar A, Brilliant M, Chute CG, et al. Practical considerations in genomic decision support: The eMERGE experience. J Pathol Inform

Kawamoto K, Del Fiol G, Lobach DF, Jenders RA. Standards for scalable clinical decision support: need, current and emerging standards, gaps, and proposal for progress. Op

Kawamoto K, Del Fiol G, Strasberg HR, Hulse N, Curtis C, Cimino JJ, et al. Multi-national, multi-institutional analysis of clinical decision support data needs to inform

Kawamoto K, Lobach DF. Proposal for fulfilling strategic objectives of the U.S. roadmap for national action on decision support through a service-oriented architecture le

Merino JG, Loder E. PCORI’s ambitious efforts to promote transparency and dissemination of research findings. BMJ. 2014;349:g6261.

National Quality Forum (NQF). Driving quality and performance measurement—a foundation for clinical decision support: a consensus report, Washington, DC: NQF; 2010. 2010.

E-2

OpenCDS. OpenCDS home page. 2014.

Osheroff JA, Teich JM, Middleton B, Steen EB, Wright A, Detmer DE. A roadmap for national action on clinical decision support. Journal of the American Medical Informatics

Scheuermann RH, Milgrom H. Personalized care, comparative effectiveness research and the electronic health record. Curr Opin Allergy Clin Immunol. 2010 Jun;10(3):168–70.

Segal C, Holve E. American Recovery and Reinvestment Act-comparative effectiveness research infrastructure investments: emerging data resources, tools and publications. J

Selby JV, Lipstein SH. PCORI at 3 Years — Progress, Lessons, and Plans. New England Journal of Medicine. 2014;370(7):592–5.

2.0 Prioritizing PCOR Findings for Dissemination via CDS

2.1 PCOR Methods and Research Designs Ahmed SM, Nelson D, Kissack A, Franco Z, Whittle J, Kotchen T, et al. Towards Building a Bridge between Community Engagement in Research (CEnR) and Comparative Effectiveness Research (CER). Clin Transl Sci. 2015 Apr;8(2):160–5.

Darling M, Gonzalez F, Graves K, Sheppard VB, Hurtado-de-Mendoza A, Leventhal K-G, et al. Practical Tips for Establishing Partnerships With Academic Researchers: A Resource Guide for Community-Based Organizations. Prog Community Health Partnersh. 2015;9(2):203–12.

Devine EB, Lee C-J, Overby CL, Abernethy N, McCune J, Smith JW, et al. Usability evaluation of pharmacogenomics clinical decision support aids and clinical knowledge resources in a computerized provider order entry system: a mixed methods approach. Int J Med Inform. 2014 Jul;83(7):473–83.

Erwin K, Krishnan JA. Using design methods to provide the care that people want and need. J Comp Eff Res. 2016 Jan;5(1):13–5.

Kenney JT. Managing the evolving complexity of pharmacologic treatment: comparative effectiveness research, pharmacoeconomic data analyses, and other decision support tools. Am J Manag Care. 2012 Nov;18(10 Suppl):S234-239.

Mayo-Wilson E, Hutfless S, Li T, Gresham G, Fusco N, Ehmsen J, et al. Integrating multiple data sources (MUDS) for meta-analysis to improve patient-centered outcomes research: a protocol for a systematic review. Syst Rev. 2015;4:143.

McCarthy IM. Putting the Patient in Patient Reported Outcomes: A Robust Methodology for Health Outcomes Assessment. Health Econ. 2015 Dec;24(12):1588–603.

Navaneethan SD, Boulware LE, Sedor JR. Patients as stakeholders in setting kidney disease research priorities. Am J Kidney Dis. 2015 May;65(5):641–3.

Norman N, Bennett C, Cowart S, Felzien M, Flores M, Flores R, et al. Boot camp translation: a method for building a community of solution. J Am Board Fam Med. 2013 Jun;26(3):254–63.

E-3

Pearson CR, Duran B, Oetzel J, Margarati M, Villegas M, Lucero J, et al. Research for improved health: variability and impact of structural characteristics in federally funded community engaged research. Prog Community Health Partnersh. 2015;9(1):17–29.

Peek CJ, Glasgow RE, Stange KC, Klesges LM, Purcell EP, Kessler RS. The 5 R’s: an emerging bold standard for conducting relevant research in a changing world. Ann Fam Med. 2014 Oct;12(5):447–55.

Shaya FT, Chirikov VV. Decision support tools to optimize economic outcomes for type 2 diabetes. Am J Manag Care. 2011 Nov;17 Suppl 14:S377-383.

Wortley S, Flitcroft K, Howard K. What is the role of community preference information in health technology assessment decision making? A case study of colorectal cancer screening. Int J Technol Assess Health Care. 2015 Jan;31(4):241–8.

2.2 Patient-Reported Outcomes Adogwa O, Elsamadicy AA, Cheng J, Bagley C. Assessing Patient Reported Outcomes Measures via Phone Interviews Versus Patient Self-Survey in the Clinic: Are We Measuring the Same Thing? World Neurosurg. 2016 Mar;87:230–4.

Bartlett SJ, Orbai A-M, Duncan T, DeLeon E, Ruffing V, Clegg-Smith K, et al. Reliability and Validity of Selected PROMIS Measures in People with Rheumatoid Arthritis. PLoS ONE. 2015;10(9):e0138543.

Basch E, Spertus J, Dudley RA, Wu A, Chuahan C, Cohen P, et al. Methods for Developing Patient-Reported Outcome-Based Performance Measures (PRO-PMs). Value Health. 2015 Jun;18(4):493–504.

Hoy MB. Personal Activity Trackers and the Quantified Self. Med Ref Serv Q. 2016;35(1):94–100.

Jensen RE, Rothrock NE, DeWitt EM, Spiegel B, Tucker CA, Crane HM, et al. The role of technical advances in the adoption and integration of patient-reported outcomes in clinical care. Med Care. 2015 Feb;53(2):153–9.

Judson TJ, Bennett AV, Rogak LJ, Sit L, Barz A, Kris MG, et al. Feasibility of long-term patient self-reporting of toxicities from home via the Internet during routine chemotherapy. J Clin Oncol. 2013 Jul 10;31(20):2580–5.

Okelo SO, Eakin MN, Riekert KA, Teodoro AP, Bilderback AL, Thompson DA, et al. Validation of parental reports of asthma trajectory, burden, and risk by using the pediatric asthma control and communication instrument. J Allergy Clin Immunol Pract. 2014 Apr;2(2):186–92.

Van Der Wees PJ, Nijhuis-Van Der Sanden MWG, Ayanian JZ, Black N, Westert GP, Schneider EC. Integrating the use of patient-reported outcomes for both clinical practice and performance measurement: views of experts from 3 countries. Milbank Q. 2014 Dec;92(4):754–75.

Vodicka E, Kim K, Devine EB, Gnanasakthy A, Scoggins JF, Patrick DL. Inclusion of patient-reported outcome measures in registered clinical trials: Evidence from ClinicalTrials.gov (2007-2013). Contemp Clin Trials. 2015 Jul;43:1–9.

E-4

2.3 Patient-Centered Measures Burke JG, Jones J, Yonas M, Guizzetti L, Virata MC, Costlow M, et al. PCOR, CER, and CBPR: alphabet soup or complementary fields of health research? Clin Transl Sci. 2013 Dec;6(6):493–6.

Cavanaugh KL. Prioritizing patient-centered care implementation and research for patients with kidney disease. Semin Dial. 2015 Apr;28(2):131–40.

Frendl DM, Ware JE. Patient-reported functional health and well-being outcomes with drug therapy: a systematic review of randomized trials using the SF-36 health survey. Med Care. 2014 May;52(5):439–45.

Glasgow RE, Riley WT. Pragmatic measures: what they are and why we need them. Am J Prev Med. 2013 Aug;45(2):237–43.

Gupta S, Brenner AT, Ratanawongsa N, Inadomi JM. Patient trust in physician influences colorectal cancer screening in low-income patients. Am J Prev Med. 2014 Oct;47(4):417–23.

Hyder JA, Wakeam E, Habermann EB, Hess EP, Cima RR, Nguyen LL. Derivation and validation of a simple calculator to predict home discharge after surgery. J Am Coll Surg. 2014 Feb;218(2):226–36.

Longo DR, Woolf SH. Rethinking the information priorities of patients. JAMA. 2014 May 14;311(18):1857–8.

Meghani SH, Chittams J, Hanlon AL, Curry J. Measuring preferences for analgesic treatment for cancer pain: how do African-Americans and Whites perform on choice-based conjoint (CBC) analysis experiments? BMC Med Inform Decis Mak. 2013;13:118.

Saczynski JS, Kiefe CI. Dawning of a new era: understanding the functional outcomes of cardiovascular disease. Circ Cardiovasc Qual Outcomes. 2014 Nov;7(6):812–4.

Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;349:g7647.

Sheetz KH, Waits SA, Girotti ME, Campbell DA, Englesbe MJ. Patients’ perspectives of care and surgical outcomes in Michigan: an analysis using the CAHPS hospital survey. Ann Surg. 2014 Jul;260(1):5–9.

Smelt AFH, Louter MA, Kies DA, Blom JW, Terwindt GM, van der Heijden GJMG, et al. What do patients consider to be the most important outcomes for effectiveness studies on migraine treatment? Results of a Delphi study. PLoS ONE. 2014;9(6):e98933.

Starks H, Shaw JL, Hiratsuka V, Dillard DA, Robinson R. Engaging stakeholders to develop a depression management decision support tool in a tribal health system. Qual Life Res. 2015 May;24(5):1097–105.

Volpp KG, Loewenstein G, Asch DA. Choosing wisely: low-value services, utilization, and patient cost sharing. JAMA. 2012 Oct 24;308(16):1635–6.

2.4 PCOR Advocacy Goold SD, Myers CD, Szymecko L, Cunningham Collins C, Martinez S, Ledón C, et al. Priorities for Patient-Centered Outcomes Research: The Views of Minority and Underserved Communities. Health Serv Res. 2016 May 20;

E-5

Li C. Personalized medicine - the promised land: are we there yet? Clin Genet. 2011 May;79(5):403–12.

Lindquist LA, Ramirez-Zohfeld V, Sunkara P, Forcucci C, Campbell D, Mitzen P, et al. Advanced life events (ALEs) that impede aging-in-place among seniors. Arch Gerontol Geriatr. 2016 Jun;64:90–5.

Parekh A, Schreiber R. How Community-Based Organizations Can Support Value-Driven Health Care [Internet]. Health Affairs Blog. 2015 [cited 2016 Jun 16]. Available from: http://healthaffairs.org/blog/2015/07/10/how-community-based-organizations-can-support-value-driven-health-care/

3.0 Authoring CDS Interventions Bailey LC, Mistry KB, Tinoco A, Earls M, Rallins MC, Hanley K, et al. Addressing electronic clinical information in the construction of quality measures. Acad Pediatr. 2014 Oct;14(5 Suppl):S82-89.

Bender D, Sartipi K. HL7 FHIR: An Agile and RESTful approach to healthcare information exchange. In IEEE; 2013 [cited 2016 Aug 9]. p. 326–31. Available from: http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6627810

Boxwala AA, Peleg M, Tu S, Ogunyemi O, Zeng QT, Wang D, et al. GLIF3: a representation format for sharable computer-interpretable clinical practice guidelines. Journal of Biomedical Informatics. 2004;37:147–61.

Boxwala AA, Rocha BH, Maviglia S, Kashyap V, Meltzer S, Kim J, et al. A multi-layered framework for disseminating knowledge for computer-based decision support. Journal of the American Medical Informatics Association. 2011 Dec;18 Suppl 1:i132-9.

Douali N, De Roo J, Jaulent M-C. Decision support system based semantic web for personalized patient care. Stud Health Technol Inform. 2012;180:1203–5.

Eddy DM, Schlessinger L. Methods for Building and Validating Equations for Physiology-Based Mathematical Models: Glucose Metabolism and Type 2 Diabetes in the Archimedes Model. Med Decis Making. 2016 Apr;36(3):410–21.

Ferranti JM, Musser RC, Kawamoto K, Hammond WE. The clinical document architecture and the continuity of care record: a critical analysis. J Am Med Inform Assoc. 2006 Jun;13(3):245–52.

Freimuth RR, Wix K, Zhu Q, Siska M, Chute CG. Evaluation of RxNorm for Medication Clinical Decision Support. AMIA Annu Symp Proc. 2014;2014:554–63.

Goldspiel BR, Flegel WA, DiPatrizio G, Sissung T, Adams SD, Penzak SR, et al. Integrating pharmacogenetic information and clinical decision support into the electronic health record. J Am Med Inform Assoc. 2014 Jun;21(3):522–8.

Greenes R, Bloomrosen M, Brown-Connolly NE, Curtis C, Detmer DE, Enberg R, et al. The morningside initiative: collaborative development of a knowledge repository to accelerate adoption of clinical decision support. Open Med Inform J. 2010;4:278–90.

Health Level 7. HL7 Decision Support Service standard. 2007.

Hoffman JM, Dunnenberger HM, Kevin Hicks J, Caudle KE, Whirl Carrillo M, Freimuth RR, et al. Developing knowledge resources to support precision medicine: principles from the Clinical

E-6

Pharmacogenetics Implementation Consortium (CPIC). J Am Med Inform Assoc. 2016 Jul;23(4):796–801.

Hongsermeier T, Kashyap V, Sordo M. Knowledge management infrastructure: evaluation at Partners HealthCare System. In: Greenes RA, editor. Clinical Decision Support: the Road Ahead. Boston: Elsevier Academic Press; 2007. p. 447–67.

Hulse NC, Rocha RA, Del Fiol G, Bradshaw RL, Hanna TP, Roemer LK. KAT: a flexible XML-based knowledge authoring environment. Journal of the American Medical Informatics Association. 2005;12:418–30.

Ji Y, Skierka JM, Blommel JH, Moore BE, VanCuyk DL, Bruflat JK, et al. Preemptive Pharmacogenomic Testing for Precision Medicine: A Comprehensive Analysis of Five Actionable Pharmacogenomic Genes Using Next-Generation DNA Sequencing and a Customized CYP2D6 Genotyping Cascade. J Mol Diagn. 2016 May;18(3):438–45.

Johnson PD, Tu SW, Musen MA, Purves I. A virtual medical record for guideline-based decision support. Proc AMIA Symp. 2001;294–8.

Kasthurirathne SN, Mamlin B, Grieve G, Biondich P. Towards Standardized Patient Data Exchange: Integrating a FHIR Based API for the Open Medical Record System. Stud Health Technol Inform. 2015;216:932.

Kawamoto K, Del Fiol G, Orton C, Lobach DF. System-agnostic clinical decision support services: benefits and challenges for scalable decision support. Open Med Inform J. 2010;4:245–54.

Kawamoto K, Lobach DF, Willard HF, Ginsburg GS. A national clinical decision support infrastructure to enable the widespread and consistent practice of genomic and personalized medicine. BMC Medical Informatics & Decision Making. 2009;9:17.

Kawamoto K, Lobach DF. Design, implementation, use, and preliminary evaluation of SEBASTIAN, a standards-based Web service for clinical decision support. Proceedings/AMIA Annual Symposium. 2005;380–4.

Kawamoto K. Integration of knowledge resources into applications to enable clinical decision support: architectural considerations. In: Greenes RA, editor. Clinical Decision Support: the Road Ahead. Boston: Elsevier Academic Press; 2007. p. 503–38.

Kawamoto K, Hongsermeier T, Wright A, Lewis J, Bell DS, Middleton B. Key principles for a national clinical decision support knowledge sharing framework: synthesis of insights from leading subject matter experts. J Am Med Inform Assoc. 2013 Jan 1;20(1):199–207.

Leong T-Y. Toward patient-centered, personalized and personal decision support and knowledge management: a survey. Yearb Med Inform. 2012;7:104–12.

Lowe HJ, Ferris TA, Hernandez PM, Weber SC. STRIDE—An integrated standards-based translational research informatics platform. AMIA Annu Symp Proc. 2009;2009:391–5.

Mackay EA. Patients, consumers, and caregivers: the original data stewards. EGEMS (Wash DC). 2015;3(1):1173.

Marcos C, González-Ferrer A, Peleg M, Cavero C. Solving the interoperability challenge of a distributed complex patient guidance system: a data integrator based on HL7’s Virtual Medical Record standard. J Am Med Inform Assoc. 2015 May;22(3):587–99.

E-7

Marlin BM, Adams RJ, Sadasivam R, Houston TK. Towards collaborative filtering recommender systems for tailored health communications. AMIA Annu Symp Proc. 2013;2013:1600–7.

National Quality Forum (NQF). Quality data model. 2011.

Ohmann C, Kuchinke W. Future developments of medical informatics from the viewpoint of networked clinical research. Interoperability and integration. Methods Inf Med. 2009;48(1):45–54.

Overby CL, Tarczy-Hornoch P, Hoath JI, Kalet IJ, Veenstra DL. Feasibility of incorporating genomic knowledge into electronic medical records for pharmacogenomic clinical decision support. BMC Bioinformatics. 2010;11 Suppl 9:S10.

Peleg M, Boxwala AA, Bernstam E, Tu S, Greenes RA, Shortliffe EH. Sharable representation of clinical guidelines in GLIF: relationship to the Arden Syntax. Journal of Biomedical Informatics. 2001;34:170–81.

Peleg M, Boxwala AA, Ogunyemi O, Zeng Q, Tu S, Lacson R, et al. GLIF3: the evolution of a guideline representation format. Proceedings/AMIA Annual Symposium. 2000;645–9.

Peleg M, Tu S, Bury J, Ciccarese P, Fox J, Greenes RA, et al. Comparing computer-interpretable guideline models: a case-study approach. Journal of the American Medical Informatics Association. 2003;10:52–68.

Ram P, Berg D, Tu S, Mansfield G, Ye Q, Abarbanel R, et al. Executing clinical practice guidelines using the SAGE execution engine. Medinfo. 2004;11:251–5.

Rinott R, Carmeli B, Kent C, Landau D, Maman Y, Rubin Y, et al. Prognostic data-driven clinical decision support - formulation and implications. Stud Health Technol Inform. 2011;169:140–4.

Rumsfeld JS, Joynt KE, Maddox TM. Big data analytics to improve cardiovascular care: promise and challenges. Nat Rev Cardiol. 2016 Jun;13(6):350–9.

Samwald M, Fehre K, de Bruin J, Adlassnig K-P. The Arden Syntax standard for clinical decision support: experiences and directions. J Biomed Inform. 2012 Aug;45(4):711–8.

Samwald M, Miñarro Giménez JA, Boyce RD, Freimuth RR, Adlassnig K-P, Dumontier M. Pharmacogenomic knowledge representation, reasoning and genome-based clinical decision support based on OWL 2 DL ontologies. BMC Med Inform Decis Mak. 2015;15:12.

Shiffman RN, Karras BT, Agrawal A, Chen R, Marenco L, Nath S. GEM: a proposal for a more comprehensive guideline document model using XML. Journal o

the pcor cds-ln environmental scan: spurring action by ... · 11/4/2016 · spurring action by...

Documents