ispor real world data brainstorming session · pdf filethe ispor scientific presentation...

ISPOR

Real World Data

Brainstorming Session

Agenda

Getting started and background material (30 minutes)– Intros

– Meeting agenda and objectives

– Brief review of existing RWD & related TF reports

– Publication trends review

Brainstorming on individual topics (60 minutes)– Review topics – any that should be added or combined?

– For each topic (approx. 10 minutes per topic)

• What are the key issues here?

• What elements could a TF reasonably address with clarity?

Prioritization exercise (20 minutes)– Review of topics and discussion of relevance/importance, again consider need to

combine or distinguish topics

– Voting exercise

Summary and next steps (10 mins)

2

Today’s objectives

Review and discuss issues in RWD analysis where

– Guidance would be useful to ISPOR members and researchers in general

AND

– Best practices can reasonably be identified, or

– Emerging practices are worthy of being called out and reviewed

NOT

- A policy proposal

Prioritize qualifying topics for potential Task Force focus

This is not the initiation of a Task Force per se

– Task Force initiation involves creation of a formal proposal and approval by

the Health Sciences Policy Council

3

What is the remit of a Task Force?

ISPOR Good Practices for Outcomes Research Reports are

consensus guidance recommendations for conducting

outcomes research (clinical, economic, and patient-reported

outcomes) or for using outcomes research in health care

decisions.

4

Original RWD TF report

Garrison Jr. LP, Neumann PJ, Erickson P, et al. Using real-world data for coverage and payment decisions: The ISPOR real-world data task force report. Value Health 2007;10:326-35.

Defined RW data as data used for decision-making that are not collected in conventional randomized controlled trials (RCTs).

Considered several characterizations: – by type of outcome (clinical, economic, and patient-reported),

– by hierarchies of evidence (which rank evidence according to the strength of research design),

– by type of data source (supplementary data collection alongside RCTs, large simple trials, patient registries, administrative claims database, surveys, and medical records).

Discussed eight key issues: 1) the importance of RW data;

2) limitations of RW data;

3) the fact that the level of evidence required depends on the circumstance;

4) the need for good research practices for collecting and reporting RW data;

5) the need for good process in using RW data in coverage and reimbursement decisions;

6) the need to consider costs and benefits of data collection;

7) the ongoing need for modeling; and

8) the need for continued stakeholder dialogue on these topics

5

Previous RWD-related TF reports

Motheral B, Brooks J, Clark MA, et al. A checklist for retroactive database studies –

Report of the ISPOR Task Force on Retrospective Databases. Value in Health 2003;

6:90-7

Berger ML, Mamdani M, Atkins D, Johnson ML. Good research practices for

comparative effectiveness research: defining, reporting and interpreting

nonrandomized studies of treatment effects using secondary data sources: The ISPOR

good research practices for retrospective database analysis task force report—Part

I. Value Health 2009;12:1044-52.

Cox E, Martin BC, Van Staa T, Garbe E, Siebert U, Johnson ML, Good research

practices for comparative effectiveness research: approaches to mitigate bias and

confounding in the design of non-randomized studies of treatment effects using

secondary data sources: The ISPOR good research practices for retrospective

database analysis task force–Part II. Value Health 2009;12:1053-61.

Johnson ML, Crown W, Martin BC, et al. Good research practices for comparative

effectiveness research: analytic methods to improve causal inference from

nonrandomized studies of treatment effects using secondary data sources: The ISPOR

good research practices for retrospective database analysis task force report—Part

III. Value Health 2009;12:1062-73.

6

RWE Literature Review Results

05/23/2016

University of Utah TeamDiana Brixner

Tianze Jiao

Junjie MA

Word Cloud

Fellows, Ian. "wordcloud: Word Clouds. R package version 2.2." (2012).

https://cran.r-project.org/web/packages/wordcloud/wordcloud.pdf

Method

Part One: Several searches were conducted using “real world

evidence” or “real world data” as keywords on PubMed,the

ISPOR scientific presentation database, ISPOR Annual

International Meeting Program and the Internet.

Part Two: A literature review was conducted in PubMed to

investigate the trends around RWE from following areas:

– Study design

– Perspective

– Data sources

– Methodology

Part One Results

Table 1. The number of RWD and RWE publications from different

resources

2001-2005 2006-2010 2011-2015

PubMed:

Real world evidence 4 11 93

Real world data 60 194 602

The ISPOR scientific presentation

database:



ISPOR Annual International Meeting

Program:

Issue Panel 0 1 6

Work Shop 0 19 31

Internet RWE Reports:



Part Two Results:

964 abstracts in Pubmed

After title review: 474

abstracts were excluded

Duplications: 14

Statistics: 179

Non medical: 168

Biomedical informatics: 110

Comment: 3

490 abstracts

After reviewed the full abstract, 162 abstracts were

excluded

No abstract: 62

Duplicates: 2

Irrelevant: 7

RCT: 2

Proposal: 3

Statistics: 17

Non medical: 14

Biomedical informatics: 28

Comment: 27

328 full abstracts

Figure 1. Flow chart

Study design

Figure 2. Number of abstracts by study designs in 5 years interval

There are 9, 53, 266 abstracts in 2001-2005, 2006-2010, 2011-2015

independently.

0

20

40

60

80

100

120

140

2001-2005

2006-2010

2011-2015

Study design

0

0.1

0.2

0.3

0.4

0.5

0.6

2001-2005

2006-2010

2011-2015

Figure 3. Proportion of abstracts by specific study designs in 5 years

interval

• The trend of study design is constant.

• Cohort design is still dominant.

Perspective

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Decisionmaking

Economicstudies

Outcomesresearch

QoL Qualitativeanalysis

Review

2001-2005

2006-2010

2011-2015

Figure 4. Proportion of abstracts by different perspectives in 5 years

interval * The overall proportion is larger than 1 for each 5 years interval.

• Outcomes research is still the main perspective.

• Over time, more studies were conducted from

multiple perspectives.

Perspective

Figure 5. Number of abstracts that have multiple perspectives in 5 years interval

0

2

4

6

8

10

12

14

2001-2005

2006-2010

2011-2015

Data source

Figure 6. Proportion of abstracts by different data source in 5 years interval

• Data sources have become more diverse over the years.

• Researchers started to use multiple data sources after 2010.• Real world data (EMR, RCT) and data in the literature was applied to

economic modeling

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

2001-2005

2006-2010

2011-2015

Methodology

Figure 7. Proportion of abstracts by different methods in 5 years interval

• Over time more advanced methods were applied• Cox regression, PS, MSM, G-computation

• Multiple methods were applied in one study after 2010

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

2001-2005

2006-2010

2011-2015

Methodology

Figure 9. Proportion of abstracts

that applied different propensity

score methods in 2011-2015

27%

4%

5%

23%

9%

32%

Cox regression+PS Cox regression+GLM+PS

Decision tree+PS GLM+PS

MIX+PS PS

100%

Propensity score

Figure 8. Proportion of abstracts

that applied different propensity

score methods in 2006-2010

1

abstract

22

abstract

s

Sample size

Figure 10. Scatter plot of sample size by individual year

• There is an increasing trend on sample size

(insignificant)

• More studies have large sample size The largest

sample size:

2001 - 2005 85,617

2005 - 2010 149,785

2011 - 2015 587,691

Kruskal–Wallis test p=0.387

Summary

Study design and perspective have been consistent in

the past 15 years

Data sources are more diverse than in previous years

There is an increasing trend for researchers to apply

several data sources in one study

– Real world data (RCT, EMR etc.) was integrated with data from

literature to build economic models

More advanced methods were applied after 2010

After 2010, causal inference emerged, which better

investigates causality

– Applying PS matching, MSM, G-computation

Pre-identified topics

1. Implementation issues: Use of RWE (including CER results) in several different, but related, health sector decision-making contexts

2. Better use of causal inference methods in RWE to reduce bias in causal effect estimates.

3. Incorporating patient-reported outcomes into retrospective claims and EMR databases – What’s involved?

4. Addressing the quality of observational data—particularly, how to control for missing variables through data linkage?

5. Regulation and certification of RWE

21

1. Implementation issues

Use of RWE (including CER results) in several different, but

related, health sector decision-making contexts:

– Update regulatory label information.

– Support physician-patient shared decision-making

– Inform and update clinical treatment guidelines

– Update price and access conditions for on-market products

– Potential follow-on topics:

• How is RWE currently used specifically in each of these contexts?

• What the barriers (if any) to more effective use?

• How could its use be improved in each specific context? Both locally and

globally?

22

What are the key issues here?

What elements could a TF reasonably address with clarity?

2. Better use of causal inference methods in RWE

Used to reduce bias in causal effect estimates.

Such methods could be employed to test model-based or

network meta-analysis predictions of comparative cost and

effectiveness used at product launch.

23



3. Incorporating patient-reported outcomes into RWD

What’s involved in introducing more PRO/COA data into

retrospective claims and EMR databases?

24



4. Addressing the quality of observational data

How to control for missing variables through data linkage?

25



5. Regulation and certification of RWE

26



Prioritization exercise

What are your thoughts? Let’s discuss …

Straw poll

You are receiving XX sticky dots – please place one or more of

them on the various topics, depending on your sense of their

priority

27

Summary and next steps

Points from today will be written up and circulated

Further thoughts, expressions of interest are welcome

If a new Task Force seems warranted,

– Co-chairs will be solicited

– A formal proposal must be written

– HSPC must review and approve

– A Leadership Group would then be solicited

28

THANK YOU!

29

ispor real world data brainstorming session · pdf filethe ispor scientific presentation...

Documents