Designs for eHealth impact

studies

Jeremy Wyatt DM FRCP ACMI FellowProfessor of eHealth Innovation & Director

Institute for Digital Healthcare, Warwick University

What is an impact study ?

“A study of health technology in clinical use to determine its effects on the health problem it is designed to solve and on the patients, health professionals and health system”

Includes studies of benefits, side effects and changes in health-related costs attributable to the innovation

Goes beyond measuring feasibility, deployment, attitudes, usage or perceived impact

Quantifies changes caused by the technology in healthcare structures, processes or patient outcomes

Why carry out impact studies ?

Help manufacturer:

– Improve the technology

– Write persuasive marketing material for evidence-aware health professionals and purchasers

Help policy maker: results feed into technology appraisals; if cost effective, leads to reimbursement (EU “111” proposal – 1 website, 1 minute, 1 euro)

Help professional bodies: results feed into systematic reviews & practice guidelines

– Recommendation to use eHealth prompts clinical use

– Will also lend greater support via press, public and courts

How impact studies can promote use

Impact

studies

Practice

guideline

Technology

appraisal

Professionals keen

to use eHealth

Efficiency,

quality & safety

improves

eHealth

gets used

Improved

technology

Lessons learned by

developer / supplier

Press, public &

legal support

Persuasive marketing

materials

Reimbursement

“We know it works” – motorbike paramedics

“… Full advanced life-support did not decrease mortality or

morbidity... mortality was greater among patients with Glasgow Coma Scale scores < 9” Stiell IG et al. CMAJ. 2008; 178: 1141-52

Solution: do a trial - Liu & Wyatt, JAMIA 2011

Plausible eH technologies that failed

Diagnostic decision support (Wyatt, MedInfo ‘89)

Integrated medicines management for a children’s hospital (Koppel, JAMA 2005)

MSN messenger triage (Eminovic, JTT 2006)

Smart home applications (Martin, Cochrane 2008):

“The effects of smart technologies to support people in their homes are not known. Better quality research is needed.”

Possible impact study designs

1. Before-after studies with external or internal controls – or both

2. Interrupted time series with at least 6 data points

3. “Evaluation machine”

4. Mendelian randomisation / instrumental variable methods

5. Data mining:

a) Exploring special cause variation using statistical process control

b) Forum / text mining – “Dar Wiki nism”

6. Randomised controlled trials:

a) Step wedge, switchback designs (multiple on / off system phases)

Controlled before-after design

0

5

10

15

20

25

30

35

40

45

50

Before After

%d

rug

to

xic

ity

lymphoma, site A

leukaemia, site A

lymphoma, site B

Intervention

Controlled before-after designExternal control:

– same practice in one or more matched external groups of practitioners

– subject to same secular trends & confounders

– not exposed to the intervention

Internal control:– similar practice in the same target practitioners

– subject to same secular trends & confounders

– not susceptible to the intervention

Interrupted time series design

0

5

10

15

20

25

30

35

40

45

50

% d

rug

to

xic

ity

Intervention

time

Interrupted time series design

At least 3 pre- and 3 post-intervention measurements

Aim to demonstrate regression discontinuity

Problems:

– Cost of making repeat measurements - use routine data

– Difficulty separating intervention from baseline drift, seasonal effects...

The evaluation machine

What actually happened

with eHealth

What would have

happened without eH

Compare:

Eg. compare actual TB sputum conversion rates with conversion rates

for those patients predicted by a model, based on data at presentation

Now, just give me the model…

www.instructables.com/id/How-to-Build-

a-Time-Machine-Vortex-Distortion-Spa/

Instrumental variable approach

OutcomeInterventionimproves

Instrumental variable

(usually) determines

availability of

also improves, or

worsens

Also called Mendelian Randomisation.

See: Davey Smith G, James Lind Library.

Example of IV approach

Health statuseHealth systemimproves

NGO funding

(usually) determines

availability of

Infrastructure existsLocal interest in eH

also improves, or

worsens

Mining of retrospective data

Great for generating hypotheses

Very tricky for causation:– Data problems: changes in quality, drifting

definitions, recall bias, social response bias…– Simpson’s Paradox [unmeasured change in case mix]– Confounding by indication [drug choice the marker -

not cause of - outcome]; limits of propensity scoring– Immortal Time Bias…

See: Byar D. Why databases should not replace trials,Biometrics 1980

Statistical Process Control

Origin in manufacturing, production lines

Core tool of continuous quality improvement

Focus on detecting real variation in apparently stable processes

If variation is real, search for causes (ask “Why?” 6 times !)

Associated methods: Six sigma, kaizen

Run charts, control limits and signal data points

Plot your data

Calculate mean (m), item-to-item differences

Calculate mean item-to-item difference (mid)

Subtract 2.66 X mid from mean to give lower control limit

Add 2.66 X mid to mean to give upper control limit

Identify signal data points outside limits

Ask why they occurred – and why again – six times

SPC illustration

0

2

4

6

8

10

12

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49

Un

its

gen

era

ted

th

at w

ee

k

Week

Weekly units generated Stirton 2009/10

Upper control limit

Reasons why wind generators rarely deliver on expectations

Power output is proportional to wind speed cubed eg. if 12m/s (25mph), generate 8 times as much as 6m/s

However, wind speed follows Weibull (highly negatively skewed) distribution – ie. zero or low speeds >> high speeds

Forum mining and Dar-Wiki-nism

Wisdom of crowds – but also positive feedback loops, mass hysteria

Media & covert industry influence - Carl, the “community manager”

Mismatch in self vs. clinician reported symptoms

Very little clinical exam data

Who contributes to forums ?

RCTs - some unusual trials

Intervention Measure

Butter vs. margarine Plasma lipids

Service dogs Self esteem, community integration, school attendance

Psychological counselling Anxiety in road accident victims

Prayer on behalf of others Recovery of patients in cardiac care unit

Educational visits Use of evidence in 27 obstetric units

Behavioural therapy Success in job finding in unemployed

Insecticide spraying Control of bed bugs and hepatitis B

Breast self exam Death rates in 266000 Chinese women

Liu JLY, Wyatt JC. The case for randomized controlled

trials to assess the impact of clinical information systems.

JAMIA 2011

Step wedge design

RCT in which each unit is randomly allocated to cross over to intervention early or late (or at random time)

Only fair way to allocate scarce resources – a lottery !

Eg. impact study of HIS in 28 hospitals in Limpopo province, South Africa:

– Half randomised to early implementation [but copper cable linking some to data centre stolen – 6 times !]

– Half randomised to late implementation [but chief execs of some persuaded HIS team to implement earlier]

Littlejohns & Wyatt. Evaluating computerised health information systems: hard lessons still to be learnt. BMJ 2003

Impact study trade-offsStudy dimension Scientifically useful Locally useful, acted on

Stakeholders Not involved Closely engaged

Question Generic, abstract Specific, concrete

Study setting Stropped of local context As is, warts and all

Participants Carefully selected subset As generic as possible

Intervention Generic system, no changes allowed

System as installed and tailored

Co-interventions Not allowed As required by implementers

Outcome data definitions

Internationally recognised Locally used definitions

Data capture Validated instruments Locally used forms

Study design Factorial explanatory trial Pragmatic trial

Study funder Remote, disinterested Local decision maker (ownership)

What makes an impact study “high quality” ?

Reliable: internally valid (free from bias & confounding; large enough for results not due to chance)

Relevant: externally valid (measures interest others, cover structure, processes and outcomes; technology readily available)

Specific: state type of patients & professionals & context (eg. 10 / 20

care; fee for service or salaried) using eg. STARE-HI

Accessible: study design / results intelligible, in time to inform decisions of citizens, professionals, policy makers

Multi facetted: reveals likely impact in other settings; how to implement / improve it; patients /professionals likely to benefit

More rigour in systematic reviews

• There are over 100 systematic reviews covering telehealth & telemedicine

• Only 15% of the SRs were eligible for the Medicaid Evidence Based Decisions reports 2009

• Half of these still fell below acceptable standards of rigour assessed by the AMSTAR reporting quality checklist

Better quality reviews are badly needed…

Conclusions

More, better designed impact studies are needed, to:

1. Satisfy funders, patients, clinicians that time and money spent on eHealth systems was used wisely

2. Build evidence base to allow national health systems & professional bodies to invest in eHealth

3. Learn which eHealth systems work where, for whom– and which do not, and how to improve them

4. Transform eHealth activity to one where clinical, social and economic benefits lead, not technology

Barriers to impact evaluationActor Barrier to impact evaluation

System developers

Unaware of importance of performing impact study

Think opinion survey is enough

Scared that result might be disappointing or negative

Commercial interest in concealing negative results

No access to expertise to design impact study

Insufficient funds to carry out impact study

Evaluators

Worry that a controlled impact study is unethical

Worry that RCT results will only apply to trial setting

Study was done but results negative, so not published

Study was done but design does not allow clear interpretation

Lack of capacity across EU to design and carry out eHealth impact studies

Users of results

Study methods / results poorly written

Unclear about the need for impact studies, what makes a high quality study

Policy actions to encourage impact studies and global eH uptake

More high

quality impact

studies

More interest in

study results

Improve

study quality

Education

Funding

112 (cf. 111)

reimbursement

Register, certify & label eH

services with risks & benefits

Shared US / EU

definitions & metrics

Register evaluators,

competing interests

Evaluation code of practice

More capacity

to do studies

Register studies & results

Global network of eH

innovation centres

Work with OECD, WHO,

WEF, EU, ONC, HTAi, AMIA…

Consequences of impact focus

Need to increase evaluation capacity

More chance of engaging clinicians

Focus on reinventing care pathways, not new technologies

eHealth industry based in healthcare, not technology

Ineffective technologies will disappear

Risks of not doing impact studies

Lack of impact studies means we waste resources on ineffective or harmful eHealth

Deluge of impact studies of usual health technologies swamp the few eHealth studies

Excessive industry pressure and no independent studies to refute them

eHealth backlash (telehealth in heart failure)

Conclusions 2To encourage impact studies, we must:

Work with national and global agencies

Agree language and quality criteria

Develop evaluation capacity

Educate evaluators, users, industry, purchasers

Register systems, studies, evaluators

Agree a code of practice (eg. competing interests)

Towards an evaluation framework

Appropriate evaluation methods: just enough extra data, rigour, validity…

How much rigour is needed to:

– Persuade funders to extend / renew, assure auditors of probity

– Convince users of benefits

– Persuade developers to change system

– Attract evaluators, convince editor to accept article

– Write guideline recommendation on system benefits & risks, when to use it

The evaluation mind set

Knowledgeable about outcomes, study designs

Eclectic – whatever method suits the problem

Flexible, opportunistic

Honest, independent

Respectful of local issues, stakeholder dynamics

Understanding of context, timescales, what matters, the nature of evidence

Challenges: funding and feasibility

Cost of studies in relation to implementation costs

Access to clinical settings, high quality data in time to be useful

Ethics, data protection / information governance issues

Getting attention, input of experienced evaluators

Study workforce / training issues

Challenges: relevance

Matching study aims to real stakeholder questions

Using methods that deliver results within the timescale with which the answers needed

Fitting in homework / study plan / execute / analysis / report before committee meets

Challenges: validity

Sufficiently rigorous for findings to be reliable

Sufficiently generic questions & context for others to learn from study results

Issue of sponsorship and “fear of the clear” *Chuck Friedman]

Who to do studies:

– Formative stage: stakeholders, developer

– Summative stage: funder, problem owner, others wanting the system ?

Who benefits from eHealth systems, and how ?

eHealth system

Better managed

healthcare system

More complete, structured data

Better funded

healthcare system

NGO

System userDirect benefits

Indirect benefits

System generated reports

TeleHealth in diabetes, bronchitis & heart failure

Diabetes (Farmer et al SR, 2005):

– Slight reduction of HbA1C by 0.1% (95% CI -0.4% to 0.04%)

– Use of services no different or increased with telehealth

Bronchitis (Polisena et al SR, 2010):

– Mortality may be greater in telephone-support group (RR = 1.2; 95% CI 0.84 to 1.75)

– Reduced hospitalization and A&E visits, but impact on hospital bed days varied

Heart failure (Inglis et al, CDSR 2010):

– Reduced mortality by 44% (RR 0.66, CI 0.54-0.81, p < 0.001)

– Reduced CHF-related admissions by 23% (RR 0.77)

– However, recent large RCT negative (Chaudry NEJMed, Dec 2010)

Factors promoting eHealth uptake ?

1. High quality eH systems – functionality, flexibility, resilience, interoperability

2. Political will and leadership - funding

3. Incentives for professionals – direct benefits (EM Rogers), reimbursement

4. Transparent market – certification, labelling

Evidence from independent impact studies: clarity about which patients benefit, when

Evaluation studies in system lifecycle

eHealth

system in lab

Information / communication problem

Health system

Stakeholders

System requirements

ContextUser needs

System prototype

Qualitative & quantitative studies

Usability

studies

eHealth

system in field

Function studies

Impact studies

Type of studies found by Eminovic

Current evidence base for eHealth

Number of impact studies varies:

Circa 100 reliable studies: decision support, order communications, telehealth, web for health behaviour change

C. 5-10 reliable studies: electronic health records, virtual reality for training…

1 study or less: serious games, smart homes…

When do decision support systems work ?

Success rates across trials

Target clinical practice Clinical practice Patient outcomes

Diagnosis 40% 4/10 0% 0/5

Disease management 62% 23/37 18% 5/27

Single drug prescribing,

dosing

62% 15/24 11% 2/18

Prevention 76% 16/21 0% 0/1

Multi-drug prescribing 80% 4/5 0% 0/4

Overall 64% 62/97 13% 7/52

Garg et al, JAMA 2005, 293: 1223-38