1

General Practice & Primary Health Care Research Conference

Evaluation of Primary Care Interventions Maximising

Learnings from New Initiatives

Dr Leonie Segal*Deputy Director

Centre for Health Economics Monash University

_________________

Brisbane June 2004 *leonie.segal@buseco.monash.edu.au

2

CondundrumIncreasing expenditure on health care

But

Health care not best practice

Health service mix sub-optimal

Clinical practice and resource allocation not responsive to evidence?

Behaviours influenced by marketing of companies and professional bias

Example - Management of CHF

Ace inhibiters in CHF sign. in deaths (1987)

Also C-E @ <$10,000/LY saved.

But Management in 2002:

33 - 58% patients with CHF on ACE inhibitors

42% referred to cardiologist within last 3 yearsSource: * BEACH data. SAND abstract 38, AIHW GP Stats & Classification Unit, 2003; CASE

3

Other issues

Substantial inequalities in access to health care and in health outcomes

Substantial unrealised opportunities to improve health of the community

If Redirect $1m from services cost $100,000/QALY to $10,000/QALY gain 9 LYs

Given limited resources for evaluation resources

How ensure research contributes to improved health ?

4

How to maximise learnings

1. Ask the right questions

2. Conduct quality evaluation

3. Conduct quality analysis

4. Distribute findings

5. Pursue Mechanism for change

Example of functional system: US VHA

5

I Ask the right questions

Role for pure research

But also Policy relevant research not as highly regarded? constrained by

political agendas sources of funding whether Q easy to answer

Establish research program as part of implementation and evaluation feedback loop

6

Ask right questions

Ensure scope valid does not constrain type of answer

Consider cost-effectiveness as well as effectiveness

Research issues of implementation, and mainstreaming as well as efficacy

7

Example ‘wrong?’ question distort outcomes

Too narrow scope: Eg PBAC mandateDrugs only evaluated against other drugs

+ Open-ended funding Supports new drugs on PBS drug use and costs

But • What of other approaches to management or

prevention?

Alternative approach: Priority setting across modalities and disease stages.

Eg OA - consider Exercise/strength training, Hip replacement Education Prescription drugs Natural

therapies

8

0

50

100

150

200

250

92-3250

95-6

97-8

98-9

99-00

00-01

92-3

95-6

97-8

98-9

99-00

00-01

8

6

4

2

Cost $millions

Scripts (millions)

Cost and scripts for NSAIDs

9

II Conduct Quality evaluation To enable research Q to be

answered.

Select suitable evaluation model RCT

Matched control, random selection

Before/after – ‘own control’ random selection

Before/after – naturalistic

Clinician judgement

Theory driven evaluation

10

I RCT – doubled blindedGold standard to establish program

performance

Controls for: Other influences on outcomes Self selection bias Placebo effect – if no control wrongly

attribute all change to the program.

But:

RCT often not used. Why?

1. ‘know’ intervention works But evidence or marketing,

professional bias ? context transferable ?

11

Why not use RCT?

Can’t deny care that works?

Can’t set up randomised control that represents usual care. Management protocol, participants contrived

Can’t blind participant or clinician source of bias.

How randomise system wide/population-based interventions?

Capacity for long term follow-up? High cost, drop out, retain distinction between arms

12

2. Matched control random selection

Eg by geographic area reduce possible contamination

of service provider offering treatment to control patients,

Increases number intervention patients with provider

But client groups not match on

important parametres confounding

Other factors differ; eg access to services

13

RCT cf matched control Sign. diff. control & intervention groups

Example: National CCTs RCT Area control

Attribute n=6 n=6___

age *** gender ** Australian born ***

ATSI *English spoken at home **marital status * ****needs a carer *****employment status **living arrangements **health care card holder ***receives a pension ***no private health insurance **SF -36 PCS ***SF-36 MCS **___ * Sign. Diff. in populations at baseline x n of trials

14

SHCN CCT: PBS admissions - mean cost $ per intervention & control group participant

$0

$100

$200

$300

$400

$500

$600

$700

Nov97 Feb98 May98 Aug98 Nov98 Feb99 May99 Half Years All

Month

Ann

ualis

ed M

ean

PB

S C

ost

Intervention Control

15

$0

$2,000

$4,000

$6,000

$8,000

$10,000

$12,000

$14,000

$16,000

$18,000

Remaining inTrial

Total Cost

Died Remaining inTrial

Inpatient

Died Remaining inTrialMBS

Died

Intervention Control

SHCN CCT: Mean cost per equiv. participant year. Intervention & control

16

3. Before/after ie ‘own control’ random selection

Confounders – How attribute change

But:

• Combine with qualitative research,

• Good understanding of theory

• Knowledge of natural history

4. Clinician judgement– Unreliable– Lack of quantitative evidence

– Subject to professional bias

17

5. Before after (matched control?) – naturalistic • Major problem with self selection.But • Can achieve extensive follow-up • Can seek matched control• Can make conservative assumptionsExample: Helman et al; diabetes ‘trial’

14 year follow-up. Found 20-40% redn in all-cause mortality with Comprehensive care cf usual care.

Extend opportunity for naturalistic experiments with LT follow-up through single patient identifier

18

6. Theory driven evaluation: Tasks

• How is the program meant to work – What is the underlying theory?

• Does the trial design reflect the theory?

• Was the trial implemented as intended?

• What outcomes were achieved – process and final?

• How did outcomes relate to expectations?

• If program worked/didn’t work Why?

19

Suitable for complex interventions with system wide impacts

Also formative evaluation/action research to improve the intervention

Can be combined with RCT

20

Use of Theory driven evaluation SHCN CCT Was the Trial Implemented as intended?

Time GPs spent developing care plan and Implementing risk assessment tool

High & medium-risk patients

(n=177)

Low-risk patients (n=236)

Less than 15 minutes 3% 46%

15 minutes and up to 30 46% 41%

30 minutes and up to one hour

47% 12%

One hour or more 4% 1%

Total 100% 100%

21

Comment re Evaluation models

Adopt RCT where-ever possible

Ensure sufficient time for planning, Implementation and follow-up

22

Information Collection

Consider Cost-Effectiveness

Health end points Major health events: stroke,

AMI, amputation Quality of life: utility score, SF-

36 Death: life years

Intermediate outcomes: relate to final health endpoints - eg behaviours, clinical parametres

23

Information collection

Costs

of intervention

potential cost savings through disease modification

of side effect profile

on others – eg family members

Extend follow-up

24

Key principles:

• Ensure data collection can answer research question

• Collect data as close to final health end points as possible

• Maximise follow-up period

• Maximise numbers, minimise drop-out.

• Consider direct patient/participant recruitment – be aware of selection bias

25

III/IV Analysis/Input findings to policy process

Consider efficacy, cost-effectiveness, implementation issues, embedding successful experiments

Concerns:

Independence of research?

Constraints on publishing trial results?

Access to data?

Sufficient funding for analysis?

26

Support research policy interface

Through

• Engage stake holders at start – but limit scope?

• Ensure address current policy question – but impose unrealistic time constraints

• Ensure rights to publish results – but constituency make want control?

• Report relevant information Eg ARR

27

Report ARR not simply OR

Absolute risk reduction = end point/100

Scenario A B Deaths

• placebo 5% 20%

• intervention 2% 15%

OR 0.4 0.75

ARR 3% 5%

Number treatto avert 1 death 33 (100/3) 20 (100/5)

28

IV Mechanisms for change

Financing reform: Make system more responsive &

equitable Single fund holder + allocate

health funds to populations Strengthen universal cover Adjust MBS to support certain

services Eg EPC Expand scope of core services Adjust means to pay for health

care Salaried Capitation via enrolled clients

29

How to achieve change

Information Inform and empower

citizens and patients Inform providers,

encourage referral, extend use of IT

etc. Support lobby for change

30

Mechanisms for change

Health services planning Economic evaluation to

determine optimal health service mix

Priority setting at regional level

Manpower planning Determine allied health

requirement to deliver best practice care for chronic diseases

31

Example: USA VHA System

Policy driven research that incorporates all elements for success

Ref Kizer et al 2000

32

US VHA – how to maximise learnings from research

key elements:1. Detailed accountability system – regions

responsible to meet performance targets

2. Comprehensive IT system – for patient care, accountability, research

3. Involvement of stakeholders

4. Single fundholder & LT responsibility

5. Capacity to implement change; via Direct service delivery (Eg fund ’00s of

Ambulatory Care, Drug & alcohol Centres) Control clinician training

33

6. quality assurance program supported by

research outcomes

Disease based quality assurance program Set up disease expert working parties Define best practice care Establish departures from best practice Determine how best to modify practice Implement changes Monitor impact on health

If don’t have answers fund research to get them.

34

National Surgical Quality Improvement Program (NSQIP) Actions

Develop quality indicators/benchmarks,

Collect prospective data on surgical procedures and risk adjusted outcomes

Monitor/feedback on performance to VA hospitals

Develop programs to improve outcomes in facilities that perform poorly.

 Collaboration of heath policy makers, health services researchers, surgeons at VA facilities.

 Results 1994-5 to 1997-8

• 30% 30-day post-surgical morbidity,

• 9% 30-day post-surgical mortality

35

1 year risk adjusted death rates.VA patient cohorts

Disease group 1992-3 1998-9 % change

Renal failure 25.6% 18.6% - 27.3%

CHF 23.3% 16.9% - 27.5%

chronic obstructive pulmonary disease

15.0% 11.5% - 23.3%

Pneumonia 17.8% 10.7% -39.9

diabetes 5.3% 5.2% no change

angina 4.0% 3.2% - 20%

major depression 1.9% 1.7% - 10%

schizophrenia 1.8% 1.8% no change

bipolar disorder 2.0% 1.5% -25%

36

Substantial gains achievable

• If take role of research seriously

• Invest heavily in data collection, analysis & evaluation

• Accountability/monitoring processes to support adoption of best practice