1

When Should Observational Studies Be Used For Decision-

Making, or… When Will the Emperor Have

Clothes? Jean R. Slutsky, PA, MSPH - Director, Center for Outcomes and Evidence, Agency

for Healthcare Research and Quality (AHRQ)

Robert W. Dubois, MD, PhD - Chief Science Officer, National Pharmaceutical

Council

Paul Wallace, MD - Senior Vice President and Director, Center for Comparative

Effectiveness Research, The Lewin Group

Marcus Wilson, PharmD – President, HealthCore

Speakers

Robert W. Dubois, MD, PhD - Chief Science Officer, National Pharmaceutical Council

Jean R. Slutsky, PA, MSPH - Director, Center for Outcomes and Evidence, Agency for Healthcare Research and Quality (AHRQ)

Paul Wallace, MD - Senior Vice President and Director, Center for Comparative Effectiveness Research, The Lewin Group

Marcus Wilson, PharmD – President, HealthCore

2

Discussion Topics

• Overview

• Robert W. Dubois, MD, PhD - Chief Science Officer, National Pharmaceutical Council

• AHRQ

• Jean R. Slutsky, PA, MSPH - Director, Center for Outcomes and Evidence, Agency for Healthcare Research and Quality (AHRQ)

• Professional Societies

• Paul Wallace, MD - Senior Vice President and Director, Center for Comparative Effectiveness Research, The Lewin Group

• Payer

• Marcus Wilson, PharmD – President, HealthCore

3

Issues to Explore

• AHRQ produces evidence based summaries

– how and when do they consider RWE?

– would a large and well-designed OS have more weight than an imperfect RCT?

• Professional Societies develop practice guidelines

– are they using RWE in them?

– should they do it more?

• Payment decisions influence care

– when should payers use RWE?

– are they doing so already?

4

RWE Partnerships Continue to Proliferate

PCORI Seeks to Strengthen Quality of Real-World Data and Research Conducted

8

Funding Awards

Ex: Comparative IV vs. Oral Antibiotic

Therapy for Serious Bacterial Infections

Data Infrastructure

Award

$68 M for Clinical or Patient- Powered

Research Networks

PCORI-IOM Roundtable

“Observational Studies in a Learning Health Care System”

5

“Researchers Mine Data From Clinic, Big Insurer”

By Anna Wilde Mathews, January 15, 2013

“UnitedHealth Group and the Mayo Clinic are set to unveil a new research initiative that will draw on millions of health-insurance claims and in-depth clinical patient records…

The initiative will focus on research into "best outcomes for patients at lower costs,"

Wall Street Journal. Researchers Mine Data from Clinic, Big Insurer. January 15, 2013.

ASCO Linking E-Health Data To Promote Real-Time Personalized Cancer Care

The Pink Sheet. ASCO Linking E-Health Data to Promote Real-Time Personalized Carncer Care April 1, 2013. 10

April 1, 2013

For physicians and patients, the network will guide personalized treatment by allowing EHRs to be queried in an effort learn what the best treatment options will be for a given patient based on real-world experiences and real-time analysis of data capture in electronic records.

6

Questions to Ponder

• Are observational studies influencing decision making too much, too little, or just right?

• What barriers need to be addressed so that optimal use occurs?

• What is the role of ISPOR? PCORI? Others?

12

7

IP9: WHEN SHOULD OBSERVATIONAL STUDIES BE USED FOR DECISION-MAKING, OR WHEN

WILL THE EMPEROR HAVE CLOTHES?

Panelist: Paul Wallace MD Paul.Wallace@lewin.com

Collaborators: Anjali Jain*, Robert DuBois +, Jennifer Graff +, Hillary Kleiner*, Sneha Rangarao*, Adam Lustig + and Mariam Siddiqui*

The Lewin Group* and the National Pharmaceutical Council +

Study of the Use of Observational Studies by Guideline Groups

• Study Aim: Qualitative assessment of the use of Observational Studies (OS) among a sample of Guideline Development Groups (GLG)

• Approach

– Key informant interviews with 12 active producers of Clinical Practice Guidelines (GPGs), sampled to represent a range of sponsoring organizations from Government (3),Primary (3) and specialty care (4) professional associations, and Organizations directly involved in delivery of care (2)

– A semi-structured interview guide was developed, with the Transtheoretical Stages of Change (Prochaska) model as a conceptual framework, to explore the state of readiness, current practices, and future plans for use of OS in development of CPGs

8

CPG Development Characteristics • All GLG used a similar hierarchal approach to initial evidence grading, with

RCTs exclusively at the top of the hierarchy for all professional society GLG

• All GLG described use of defined protocols for literature search, assessment of evidence quality, and evidence synthesis (evidence review)

– About half of the interviewed GLG relied on an AHRQ Evidence Practice Center (EPC) after topic nomination or through a specifically contracted evidence review with an EPC or similar organization

– The remaining GLG used internal resources plus volunteers for evidence review

• Most GLG were aware of the recent IOM reports addressing Trusted CPGs and Systematic Reviews, and approximately 2/3 of the interviewed GLG noted changes in their internal processes as a result of the IOM Reports within the last 2 years

• All GLG were aware of GRADE and had made an active choice to employ or not to employ this methodology

Current Patterns of Use of OS

• Almost all GLG noted frequent use of OS to develop CPG recommendations and OS were used in nearly all or most CPGs developed

• Most common uses of OS

– To fill a gap in RCT based evidence for a key question developed during the CPG scoping process; mainly

– Uncontrolled trials and

– Case series

– To identify Safety signals/Evidence of harms

– Case Series and Case Reports (common)

– Registries (uncommon)

9

GRADE: Rating the Quality of Evidence

Source: GRADE Working Group, 2012. See, e.g.: Balshsem H, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 2011(64):401-6.

Patterns of Use of OS (cont’d)

• For GLG using GRADE, only rare examples of OS being judged equal or higher quality than RCTs when RCTs and OS address the same issue

• Several examples of CPG recommendations with a strong strength of recommendation based only on OS in absence of applicable RCTs

– Multiple concordant OS

– Large sample size

• Special cases where OS generally form the evidence base for recommendations

– CPGs addressing use of diagnostic technologies (RCTs linking to health outcomes uncommon) - 2 Interviewees

– CPGs with key questions addressing use of Complementary and Alternative Medications and Therapies - 1 Interviewee

10

19

Rounding out the evidence…

PICO(TS)*

P Patient, Population or Problem

A description of the patient(s) of interest. It includes the condition(s), populations or sub-populations, disease severity or stage, co-morbidities, and other patient characteristics or demographics

I Intervention or Exposure

Refers to the specific treatments or approaches with the patient or population. It includes doses, frequency, methods of administering treatments, etc.

C Comparison Describes what is being compared with the intervention described above. It includes alternatives such as placebo, drugs, surgery, lifestyle changes, etc.

O Outcome Describes the specific results of interest. It refers to short, intermediate, and long-term outcomes, and includes specific areas such as quality of life, complications, mortality, morbidity, etc.

(T) Timing, if applicable

Describes the duration of time that is of interest for the particular patient outcome, benefit, or harm to occur (or not occur)

(S) Setting, of applicable

Describes the setting or context of interest. Setting can be a location (such as primary, specialty, or inpatient care), or health policy that frames or restricts the important questions to be answered

*Source: AHRQ. The Effective Health Care Program Stakeholder Guide

Appendix D: Research Questions & PICO(TS) (July 2011). Available at:

http://www.ahrq.gov/clinic/epcpartner/stakeholderguide/

Current Main Uses of OS

In CPG

Can OS improve the application of CPG to diverse groups of patients?

Patient-centered uses of OS?

• To reflect under-represented sub-populations

– Race and ethnicity

– Socioeconomic status

– Geography

• To estimate impact of co-morbidity on apparent effects reflected by RCTs

• Offer guidance for other rare conditions

• Inform shared decision making - Link guidance to patient preference and values…

11

Use of Registries and “Big Data”

• Little current development of systematic processes by interviewed GLG to incorporate data from:

–Registries based on single or multiple condition prevalence

• Including several GLG where the sponsoring organization has extensive investment in condition and procedure oriented registries

–Current large population databases (mainly claims data, but also EMR)

Future GLG directions • Key Trends self- identified by GLGs

– Desire to ‘harmonize’ recommendations across multiple CPGs addressing a common topic

• Driving adoption of common CPG processes, such as GRADE

– Desire to more directly involve patients and care givers in CPG process

• Substantial experimentation- little agreement on preferred approach

• Differentiation between patients, patient advocates, and public comment

• Impact on evidence review and role of OS undefined; most groups expect patient involvement to lead to re-examination of “evidence rules”

12

Patient centered or personalized?

Empowering Healthcare Decision-makers with Real World Evidence

Marcus D Wilson, Pharm.D. President HealthCore a WellPoint Company

Panel: When Should Observational Studies Be Used for Decision-Making or, When Will The Emperor Have Clothes

13

Payer Perspective on Observational Research

Promise: Payers seeks real world evidence in key areas including:

• Adherence1

• Cost1, 2

• Safety1, 3

• Outcomes4

Limitations: By and large where payers see limitations it is the areas of :

• Transparency/integrity of process1

• Data (quality, environment, paucity of economic data)1, 5, 6

• Motives of sponsors1

In addition, many payers would like to be involved in informing CER evidence development efforts1

1. Wang A, Halbert RJ, Baerwaldt T, Nordyke RJ. Us Payer Perspectives on Evidence for Formulary Decision Making. J Oncol Pract. 2012;8(3S):22S-27S. 2. Meyer CM, Phipps R, Cooper D, Wright A. PharmacyLowry SJ, Loggers ET, Bowles EJA, Wagner EH. Evidence Gaps in Advanced Cancer Care: Community-Based Clinicians’ Perspectives and Priorities

for CER. Am J Manag Care. 2012;18(5 Spec No. 2):SP77-SP83. 3. Atkins D. Creating and Synthesizing Evidence with Decision Makers in Mind: Integrating Evidence from Clinical Trials and Other Study Designs. Med Care. 2007;45:S16-S22. 4. Hochman M, McCormick D. Characteristics of Published Comparative Effectiveness Studies of Medications. JAMA. 2010;303(10):951-8. 5. Bourgeois FD, Murthy S, Mandl KD. Comparative Effectiveness Research: an Empirical Study of Trials Registered in Clinical Trials.gov. PLoS One. 2012;7(1):e28820 6. Benefit Forecast for a New Interferon Beta-1a for the Treatment of Multiple Sclerosis: Development of a First-Line [trunc]. J Manag Care Pharm. 2003;9(2):168-74.

Hurdles

Right Questions

Appropriate Methods of Analysis

14

Driver of Poor Clinical & Cost of Care Outcomes: Root of the Real World Evidence Gap

Utilization

Little to No

Evidence

Little to No

Evidence

Evidence Clinical Trials

Conceptualization Regulatory Approval

Unstudied co-morbid conditions

Differing concomitant medications

Varying levels of compliance – i.e. < 80%

Major Challenge: Variances in population characteristics from what was studied

Differing age groups – elderly, pediatrics

Race, ethnicity, gender variances

Varying severity of disease

Evidence Post-Marketing Studies

Hurdles

Right Questions

Appropriate Methods of Analysis

Robust, Reliable and Relevant Data

15

Reliable Evidence Development Starts with Good Data

Question the source as critically as we do the methods of analysis

*Note: ~1500 member population. Select enrichment measures only

Electronic Healthcare Data Proliferation

Health Plan Claims

Inpatient Chart

Information

Outpatient Chart Information

Patient Reported

Information

Value is unlocked with integration

Each Data Source

Contributes Unique

Information

Health Plan Claims provides

a broad view, insight into cost

of care & information such as

actual fill (Rx) history

Outpatient Chart Information

provides rich clinical

information and such

endpoints as prescribed

medications, lab and test

results, OTC advice and

reported use, family history

Inpatient Chart Information

provides rich clinical

information on procedures,

conditions treated, medication

use, lab and test results

The greater the

overlap the deeper

the insight

16

A key step for the RWE Collaboration success: the Longitudinal Patient Record…

Patient Reported & Device

Data

EHR Data

Other Data

Pharmacy

Claims Data

Medical

Claims Data

Eligibility Data

Lab Results

Data

Delivery System Cost

of Care Data

360° view of patient to improve patient health

… and manage total cost of care

RWE: Real World Evidence Collaboration

Hurdles

Right Questions

Appropriate Methods of Analysis

Robust, Reliable and Relevant Data

Available at the Point of Decision

• Ability to Effectively Disseminate findings to Downstream Decision-Makers

17

Everything we do must be done with the intent

of strengthening the bond and improving

decision-making between these two individuals.

Of All the Relationships in Healthcare, This One is the Most Important

Driver of Poor Clinical & Cost of Care Outcomes: Root of the Evidence Gap

Utilization

Little to No

Evidence

Little to No

Evidence

Evidence Clinical Trials

Conceptualization Regulatory Approval

Unstudied co-morbid conditions

Differing concomitant medications

Varying levels of compliance – i.e. < 80%

Major Challenge: Variances in population characteristics from what was studied

Differing age groups – elderly, pediatrics

Race, ethnicity, gender variances

Varying severity of disease

Evidence Post-Marketing Studies

Patient

Patient

Patient Physician

LPR

18

Barriers to Improved Quality & Affordability

Current Complaints

Patient History

Test Results Coverage

Reimbursement Medical Policy

Socioeconomic Issues

New Study Findings

For Complex Patients the time needed to make the most informed set of decisions

1-2 Hours

Amount of time the physician actually spends per patient

13 minutes

Errors &

waste Errors &

waste

Disconnect

Translation of population-based

evidence to the individual in front of

me

Impact of Health Information Technology Improvements

Decision Support

HIT

Research

HIT will facilitate Real World Research to some extent.

More importantly, it will make putting research into action quickly a

reality

Without effective and trusted tools for decision-support the knowledge gap at the point of

care will only widen

Improved quality, safety and affordability

19

Research

&

Analytics

Payer

Policy

Aligned

Payment

Model

Relevant

Information

at POC

Closing the Loop on Evidence Development & Meaningful Impact

Care

Management

Programs