The Implementation of New Zealand’s Health Targets: Generating Traction or Driving to Distraction?

Tim Tenbensel and Esther Willing, University of Auckland

New Zealand’s health targets2009-2012

95 percent of patients will be admitted, discharged, or transferred from an Emergency Department (ED) within six hours.

95 percent of two year olds will be fully immunised by July 2012

Overview

The debate about health targets

Our questions and our ways of researching them

How were the targets implemented?

Assessing the ‘balance of consequences’ of each target

How to explain different consequences?

Implications and Discussion

The Use of Performance Measures

give account (to external

stakeholders)

steer and control learning

Targets (command and control)

Intelligence (collegial)

Rankings (choice,

competition)

Sources: van Dooren et al 2011;Hood 2007

‘High stakes’

NZ’s health targets

Health Targets: The Case For

Generate sustained improvements (when other policy instruments do not)• cf the UK ‘natural experiment’ of the 2000s

Enable clearer focus – prioritising what is deemed important

Promote distributed ownership and collective problem-solving

Encourage creativity and innovation across services

Improve intelligence of organisation(s) and stimulate learning

Enable learning across organisations

Health Targets: The Case Against

Fosters behaviour that ‘hits the target but misses the point’• Gaming (and possibly cheating)

Generate resistance from clinicians• An extrinsic motivation crowding out an intrinsic one• Undermine clinical autonomy

Focus and attention lead to ‘effort substitution’

Promotes ‘myopic’ behaviour• Focus on the ‘parts’ of the service to the detriment of the whole• Jeopardises inter-organisational relationships

Measured target performance 2009-2012

  July 2009 performance

July 2012 performance

Number of DHBs that reached the target (95%)

Immunisation

77% 93% 8 /20

Emergency Department

80% 94% 12 /20

We are not evaluating the targets based on achievement against the targetWe are seeking to understand and learn from implementation processes

Our questions

What conditions are conducive to the application of health targets?

Under what conditions should they be avoided, or used with caution?

Our Research What we can’t answer (at this stage)

o Effects (positive or negative) on health outcomeso The extent, and scope of ‘effort substitution’

What we can answero What implementers did, and how they responded (eg…

buy-in, relationships, systems and processes, gaming)

Immunisation Target Research Methods

Four DHBs as case study sites Quantitative data from the National Immunisation

Register Qualitative data on the implementation process Two phases of interviews

o Phase one – 33 interviews between August and December 2011

o Phase two – 16 interviews between September to October 2012

Key informants across each level of the local health systemo DHB, PHO, National Immunisation Register, Public Health

(Medical Officers)

Identified immunisation champions

Created immunisation health target networks

Improved local systems and processes for immunisation

Data matching between practice records and the National Immunisation Register

Outreach Immunisation Services

What did DHBs and PHOs do in response to the target?

Implementer Experience of the Immunisation Target

Focused attention on

child immunisatio

n

Target Pressure

Target Fatigue

Improved relationships between

local organisation

s

Collegial Competition

Researching the Shorter Stays in ED Target

Stream 4: Integratio

n

Stream 1a: Development of

patient-flow model in conjunction with

sector clinicians and managers

Stream 1b: Survey of target-related

resource use in all NZ hospitals;

Stream 2: utilisation and data for all NZ

hospitals 2006-2012clinical quality

markers in 4 case study hospitals

2006-2012

Stream 3: Semi-structured

interviews with key informants at 4 case

study hospitals 03/04 2011, 06/07

2012, 68 interviews in total

http://www.akhdem.co.nz/ssed-research/

ED Target Research Methods

Four DHBs as case study sites Qualitative data on the implementation process Two phases of interviews

o Phase one – 47 interviews between March to May 2011o Phase two – 21 interviews between July to September

2012 Key informants within case study site hospitals

o ED nurses, doctors, o inpatient specialists, nurseso Allied professionals (hospital)o DHB / hospital managers

Jones, P., E. Sopina, et al. (2014). "Resource implications of a national health target: The New Zealand experience of a Shorter Stays in Emergency Departments target." Emerg Med Australas 26(6): 579-

584.

ED Target: What did DHBs do?

Put more resources into Emergency Departments

Created or beefed up ‘short stay’ wards

More attention to discharge planning from wards

Some changes to referral procedures between EDs and wards

Developed internal information systems about ED LOS

What did hospitals do in response to the target?

Implementer Experience of the Emergency Department Target

Target Focus

Target Pressure

Focus on the Front

of the Hospital

Demands on

Emergency Nursing

and Medicine

Dealing with

Resistance

Push Through

the Hospital

July – September 2011

July – September 2008

Balance of Consequences

Table derived fromo Literature reviews of targets and performance managemento Inductive coding of case study interviewso Comparison across the two targets

  Positive (Gaining Traction) Negative (Driven to Distraction)

1) Focus / pressure

Target enables organisation to focus, clarifies organisational and individual priorities

Implementers experience (undue) pressure related to focus on target

2) Buy-in / Resistance

Widespread buy-in and ownership from key staff involved in implementation

Active and/or passive resistance (e.g target is not my problem”)

3) Relationships Target stimulates formal and informal relationships and networks between implementers, facilitates trust, collective problem-solving

Target exacerbates pre-existing tensions, and/or creates new tensions between individuals/ units/organisations

Balance of Consequences

  Positive (Gaining Traction) Negative (Driven to Distraction)

4) Systems and Processes

Target generates new and improved systems and processes of monitoring, and follow-up,

Failure to generate new systems and processes and/or innovations experienced as negative

5) Learning Across Sites

Case study sites learning from others’ experiences

Absence of learning across sites, or failed attempts at translation

6) Effects on other services

Target implementation has beneficial consequences for other services

Target implementation has adverse consequences for other services

7) Gaming No evidence of gaming (Substantial ) evidence of gaming

Balance of Consequences: Immunisation Target

 

1. Focus / pressure ++/- (significant focus and commitment to target, but pressure leads to target fatigue at end)

2. Buy-in / Resistance

+++ (buy-in from all key staff across all organisations)

3. Relationships +++ (target fosters the development of immunisation networks in each sites leading to collective problem-solving)

4. Systems and Processes

+++ (extensive development of new systems and processes across all sites)

5. Learning Across Sites

+++ (sites actively learning from early achievers – adapting solutions to local contexts)

6. Effects on other services

++ (perceived enhancement of access primary care through engagement with practice nurses)

7. Gaming + (no evidence of gaming)

Balance of Consequences: ED Target

  Emergency Department target1. Focus / pressure ++/-- (strong focus and clarity in EDs, pressure

experienced negatively in EDs and wards)

2. Buy-in / Resistance

++/-- (buy-in from management and ED, resistance from inpatient specialists)

3. Relationships +/-- (mixed effect on relationships – some new relationships strengthened, many tensions exacerbated)

4. Systems and Processes

++/- (development of new systems and processes in all sites, some experienced as micro-management)

5. Learning Across Sites

+ (limited attempts to transfer learning across sites)

6. Effects on other services

- (perception of chaotic environment and churn in inpatient wards)

7. Gaming -- (clear evidence of gaming: clock-stopping, and decanting patients to short stay units)

Similarities and Differences

Similarities Staff experienced

targets as ‘high stakes’

Significant levels of staff ‘buy-in’ (esp when staff agreed with problem definition)

Stimulation of novel systems and processes

Differences Bottom-up ‘networking’,

stronger relationships across implementation ‘units’ (immunisation only)

Resistance (ED target only)

Flow-on effect to other services (+ve for imms, -ve for ED target)

Gaming (ED target only)

Why are the ‘Balances of Consequences’ Different?

Is it something about the

performance measure?

Is the target measuring the

right thing?

What else is the measure

doing?

Is it something about the

implementation context?

Are inter / intra-

organisational relationships

complex?

Does the target encroach on

medical autonomy?

Is it the Performance Measure?

Synecdoche (Bevan and Hood 2006) – how good a proxy is the target for the desired outcome?

Is it the Performance Measure?

Immunisation Output measure –

highly effective proxy for desired outcome (herd immunity)

No scope for gaming o Numerator is clear (and

audited)o Denominator is clear

(and controlled/audited independently)

Emergency Department

Process/efficiency measure – captures one dimension of ED quality

There is scope for gamingo Particularly regarding

the numerator

Is it the Implementation Environment?

Immunisation Service (and service

context) is not complex

Inter-organisational links:o Facilitated by buy-in,

shared professional identity

At DHB & PHO level, requires only a few key implementers

Emergency Department

Complexity of ED demand & supply

Intra-organisational dynamics are complex

Power and autonomyo Solutions required

changes in inpatient specialists’ conditions

Our questions

What conditions are conducive to the application of health targets?o When the measure is an effective proxy

for the desired outcomeo When the implementation context is fairly

simple Under what conditions should they be

avoided, or approached with caution?o Where the measure is only an

‘approximate’ proxyo When implementation context is complex

The Good News

Targets can work, and have an even wider range of benefits• Reducing inequalities• Harnessing ‘synergy’ between ‘hierarchical’

and ‘network’ governance• Can be used for accountability and learning

(across the ‘target use’ spectrum)How common are the circumstances that enable the effective use of targets?

But……

Targets can have adverse consequences

A ‘screening’ process is necessary

• Synecdoche test• How closely does the measure represent the policy objective?• To what extent is variation (over time, between sites) attributable to ‘external’ factors?

• Gaming opportunities• Implementation environment• Anticipating resistance• Mapping the complexity of inter and intra-organisational relationships

Current health sector approach: if something is a priority, then establish a target performance measure

Consider use of targets alongside other instruments