FULL REPORT

IS THE HEALTH QUALITY AND SAFETY COMMISSION

MAKING A DIFFERENCE

IN

NEW ZEALAND HEALTH CARE?

AN EVALUTION REPORT

FOR THE

HEALTH QUALITY AND SAFETY COMMISSION

PREPARED BY

PROFESSOR JACQUELINE CUMMING

DR PHIL HIDER

PROFESSOR ROBIN GAULD

Building on an earlier report by PROFESSOR JACQUELINE CUMMING, DR KIRSTEN LOVELOCK & DR GREG MARTIN,

DR PHIL HIDER, PROFESSOR ROBIN GAULD, DR JANET MCDONALD, DR DAVID MASON

APRIL 2016

1

TABLE OF CONTENTS

Acknowledgements................................................................................................................4Abbreviations..........................................................................................................................4Executive Summary................................................................................................................ 5

1.0 Introduction...................................................................................................................91.1 Report Structure..................................................................................................................10

2.0 The Health Quality and Safety Commission.................................................................11

3.0 Evaluation Questions, Framework and Methods.........................................................133.1 Evaluation Questions...........................................................................................................133.2 Evaluation Framework.........................................................................................................13

3.3 Evaluation Methods.............................................................................................................133.4 Limitations...........................................................................................................................15

4.0 How Successful has the Commission been in Effecting Change?.................................164.1 Introduction.........................................................................................................................164.2. How has the Health Quality and Safety Agenda been Influenced by having an Independent

Commission?..........................................................................................................................164.3 What is the Commission’s Role and How is the Commission Changing the Ways in which Key

Stakeholders think about Health Quality and Safety?............................................................224.4 How is the ‘Open for better care’ Campaign Seen to be Contributing to Improving Health

Quality and Safety in New Zealand?.......................................................................................404.5 Has Quality Improvement Capacity and Capability Improved?............................................46

4.6 Has the Engagement and Empowerment of Consumers Improved?...................................50

5.0 Have the Desired Changes in Safety Practice and Reductions in Harm and Cost Occurred?....................................................................................................................54

5.1 Introduction.........................................................................................................................545.2. Stakeholder Perspectives.....................................................................................................54

5.3 Analyses of Quality and Safety Markers...............................................................................54

6.0 Has the approach to change the Commission has adopted resulted in sustainable improvement?.............................................................................................................63

7.0 Conclusions and Recommendations............................................................................667.1 Introduction.........................................................................................................................66

7.2. The Role of the Commission................................................................................................667.3 The Work of the Commission...............................................................................................67

7.4 The ‘Open for better care’ Campaign...................................................................................69

2

7.5 Capacity and Capability........................................................................................................70

7.6 Consumer Engagement and Empowerment........................................................................707.7 Sustaining Improvement......................................................................................................71

8.0 References...................................................................................................................72

Appendix 1 Case Studies.......................................................................................................78Hutt Valley District Health Board/Wairarapa District Health Board.................................................78Canterbury District Health Board.....................................................................................................79

Waitemata Distric Health Board......................................................................................................80

3

Acknowledgements

This full report draws on an earlier report prepared for the Health Quality and Safety Commission. The research team would like to acknowledge and thank all of the participants who gave freely of their time over the course of these 18 month evaluations. We would also like to thank Health Quality and Safety Commission staff members who assisted in organising interviews, the Executive Assistants at the Hutt/Wairarapa, Canterbury and Waitemata District Health Boards who provided contacts and arranged interview times and rooms. We would also like to thank the District Health Board General Managers Human Resources group and their nominated colleagues for facilitating the distribution and promotion of a national survey of all District Health Board clinical staff. Thank you to Dr Jenny Neale for her help with interviewing. This independent evaluation was funded by the Health Quality and Safety Commission.

Abbreviations

ACC Accident Compensation Corporation

CLAB Central Line Associated Bacteraemia

DHB District Health Board

HQSC Health Quality and Safety Commission

IAD Programme Improvement Advisor Development Programme

Ministry Ministry of Health

QSM Quality and Safety Markers

QSI Quality and Safety Indicators

The Campaign ‘Open for better care’ Campaign

The Commission Health Quality and Safety Commission as a whole

The Commission Board Health Quality and Safety Commission Board

4

Executive Summary The Health Quality and Safety Commission New Zealand (the Commission) was established as a Crown agent in late 2010 to lead and co-ordinate work in quality and safety across the health and disability sector in New Zealand, with the goals of ensuring that fewer patients are harmed, and more patients get care they need and value. In 2013, the Commission engaged and funded researchers at Victoria University of Wellington and the University of Otago to undertake a high level examination and evaluation of key aspects of the Commission’s work programme. This evaluation report aims to answer the following high-level questions:

How successful has the Commission been in effecting change? Did the desired change in safety practice occur? Did a reduction in harm and cost occur? Has the approach to change the Commission has adopted resulted in sustainable

improvement?

The research evaluated aspects of: i) the national ‘Open for better care’ national patient safety campaign; ii) the improvement advisor development programme; and iii) the overall impact of the Commission’s work. These evaluations took place concurrently between October 2013 and June 2015. They involved several streams of work and a mix of qualitative and quantitative methods, including interviews with a range of stakeholders at all levels undertaken in 2013, 2014, and 2015; a Health Professionals’ Perceptions of Quality Survey undertaken within 19 District Health Boards (DHBs) undertaken in early 2015; and an analysis of quantitative quality and safety marker data (with analyses to the end of October 2015 where data were available).

The emphasis in this research is on the work of the Commission through to mid-2015 and, throughout this report, we note where it was felt that progress was being made on issues identified in earlier stages of the evaluations. The evaluations particularly emphasise the Commission’s work with DHBs to improve quality and safety within DHB hospitals, where the Commission’s early work has been focused. This report is structured in order of the high-level questions we investigated.

How successful has the Commission been in effecting change?

This first section explores health sector perspectives on the work of the Commission, relating to how successful the Commission has been seen to be in effecting change.

Overall, the role of the Commission appears to be very well understood as a leadership and facilitation role, providing strategic direction, advice, and resource. The Commission’s work is seen as important and the independence of the Commission is well supported, enabling a focus on improvement instead of compliance. Moreover, many in the sector have been very impressed with the staff of the Commission and have appreciated their work in the sector. Key issues for future consideration include a) the importance of the Commission taking a supportive approach to quality and safety in New Zealand as opposed to a compliance approach; b) the Commission making more of its advocacy role, including

5

raising its visibility as a commentator on health quality and safety; and c) recognising the importance of leadership and culture in improving quality and safety in New Zealand.

In terms of the work of the Commission, many of those we interviewed felt it important that the Commission work with health providers across the entire sector. Many also stressed the importance of the sector working more with consumers, having a patient-centred approach, and gaining a better understanding patient experience. There were some key aspects of the Commission’s work that were identified as needing greater attention, including improved strategic planning, a greater coherence in its work, and a more focused, evidence-based approach to determining its work programme. In later interviews, it was felt by interviewees that progress was being made on these issues as the Commission matured.

Improved prioritisation was seen as particularly important because many of those interviewed were concerned about the resource implications of expansions in the Commission’s work across the entire health and disability sector and through greater consumer engagement. The evaluations also reinforced the importance of ensuring there are sufficient resources available for health providers to work on Commission priorities.

In terms of the ‘Open for better care’ Campaign, not all of those interviewed – especially front- line staff – were clear about its aims or what it involved, although the Health Professionals’ Perceptions of Quality Survey showed a good level of awareness of the Campaign by early 2015. There were mixed views about the Campaign, with some not appreciating the ‘public relations’ elements and concerned about its cost. Some very much appreciated the Campaign. There was support for the programmes/topic areas focused on during the Campaign, as these were topic areas that the DHBs had historically been addressing. Generally, it was considered that the Campaign had raised the profile of health quality and safety concerns in DHBs. The evaluations raise a number of issues relating to the Campaign, however, including the need to: keep the costs reasonable; pay attention to the resource implications for health providers and provide additional resources for implementation; consult more and ensure local context is well understood; ensure there is sufficient time for programmes to bed in before moving on to the next programme; engage more with consumers to take on a role in relation to health quality and safety; and recognise that front-line staff are pragmatists and are more likely to respond to practical improvements.

The evaluations found it is well recognised that there are major gaps in capacity and capability in quality improvement in New Zealand, and a need for initiatives to build leadership, as well as skills and expertise in quality improvement. Participants in the Improvement Advisor Development Programme viewed it very positively and enthusiastically, while recognising the workload it entailed. Some concerns were, however, raised over the cost of the programme, the time involved and the impact on participants’ workload, and the ability of those trained to undertake further quality improvement in their workplaces, in particular, whether they would have the time and resources available to support them.

Throughout the interviews, it appears that the work of the Commission was seen to be 6

contributing to an improved health quality and safety ethos. The Commission’s work, according to some participants, had helped a focus on health quality and safety, and it was thought that this would ultimately result in gains in reduced rates of harm and likely health quality and safety benefits. Some concerns were raised, however, over ensuring that the Commission’s work did not simply lead to a compliance response that might arise from a significant increase in workloads, and that real culture change in the sector did occur.

Did the desired change in safety practice occur? Did a reduction in harm and cost occur?

The main aim of the quality and safety markers (QSMs) is to measure the success of the ‘Open for better care’ Campaign and the programmes that form part of the Campaign, and to determine whether the intended changes in practice and improvements in outcome have occurred. The QSMs are a mix of process and outcome measures: the process measures provide information about whether the desired changes in practice have occurred while the outcome measures assess whether harms and costs have been avoided.

Process markers have tracked changes in relation to falls (2 markers), hand hygiene (1), perioperative harm (1), central line-associated bacteraemia (CLABs) (1) and surgical site infections (3). All of the process markers report improvement, with the largest gain related to the increase in the proportion of operations where all three parts of the surgical safety checklist have been used increasing by 36.6%. Longitudinal data related to the outcome markers were available for all of the key areas: falls (3), hand hygiene (1), perioperative harm (6) and CLABs (3). All of the CLAB outcome markers consistently indicate improvement with decreasing CLABs and CLAB-related costs. Likewise, the markers related to the additional bed days and costs associated with either deep vein thrombosis (DVT)/pulmonary embolism (PE) or falls resulting in fractured neck of femur events also suggest that improvements have occurred. By contrast, Staphylococcus aureus bacteraemia rates appear to be generally stable and sepsis events per 1,000 at risk operations appear to have increased and the additional bed days and costs related to these events have risen. Medication reconciliation was introduced as a new marker related to medication safety in 2014. It is a structural marker and relates to the national rollout of electronic medication reconciliation. By the end of the 2014, four DHBs had implemented electronic reconciliation; a further three were planning to do so in early 2015.

These analyses show that generally there is evidence of strong and sustained improvements in most areas addressed by the Campaign over the period 2012-2014. The process marker results are generally more pronounced than those observed with the outcome data and the results from both types of markers are usually consistent. Process markers would be expected to change before outcome markers and it is perhaps not surprising that some outcome markers have not significantly altered over the relatively short period of assessment.

Note, however, that the quantitative analyses of trends in QSMs have a number of specific limitations, arising from analyses: covering a short period of time; not being able to determine whether it is the Commission’s work that is resulting in change nor which specific activities are responsible; not taking random variation into account; possibly finding ceiling effects; drawing off a 2012 baseline that may or may not be an appropriate baseline; and

7

from the data used being subject to a number of assumptions that may also impact on the data’s validity.

8

Has the approach to change the Commission has adopted resulted in sustainable improvement?

Although it is too early to be certain on this, there are positive signs that the approach adopted by the Commission may result in sustainable improvement. We found strong support for the Commission and no-one questioned the need for an emphasis on health quality and safety in New Zealand. Many of those we interviewed stressed the importance of a sustained a focus on health quality and safety, but a number of difficulties to achieving this were identified. Some reflected that key to sustainability would be the leadership of Boards, CEOs, and senior leaders within health organisations; greater engagement with health quality and safety issues at all levels, including middle managers; an emphasis on ensuring that health provider workplace cultures support a focus on health quality and safety; and ensuring that key leaders and staff gain and sustain skills in quality improvement. A key concern is the ability to allocate staff to health quality and safety roles, and to release staff to work on health quality and safety issues.

In summary, we believe there is strong evidence to assert that the Commission can be confident in, and proud of, what it has achieved in a relatively short time since its creation. Its work has clearly had a wide-ranging impact at many levels and points across the New Zealand health sector and we found considerable positive support for this. As noted by many of our interviewees, and in the international literature and experiences, quality and safety improvement are a journey not an end. This highlights that the Commission should continue to be supported and developed as an independent agency and with its work programme, and be encouraged to embed the way in which it has worked as a promoter of health quality and safety.

9

1.0 IntroductionThe Health Quality and Safety Commission New Zealand (the Commission) was established as a Crown agent in late 2010, ‘to lead and coordinate work in quality and safety across the health and disability sector; to measure, monitor and improve the quality and safety of health and disability support services; and to help providers across the sector improve these services’ (Health Quality and Safety Commission, 2014a p.8). Its goal is that fewer patients are harmed and more patients get care they actually need and value (Health Quality and Safety Commission, 2014a p.8).

In 2013, the Commission engaged and funded researchers at Victoria University of Wellington and the University of Otago to undertake a high level examination and evaluation of key aspects of the Commission’s work programme. In evaluating the Commission’s work we aim to answer the following high level questions:

How successful has the Commission been in effecting change? Did the desired change in safety practice occur? Did a reduction in harm and cost occur? Has the approach to change the Commission has adopted resulted in sustainable

improvement?

The research evaluated aspects of: The national ‘Open for better care’ national patient safety campaign The improvement advisor development programme (IAD Programme), and The overall impact of the Commission’s work.

These evaluations took place concurrently between October 2013 and June 2015. They involved several streams of work and a mix of methods, including document review; interviews with a wide range of stakeholders; case studies; a health professionals’ perceptions of quality survey undertaken nationally within 19 DHBs; an analysis of key quantitative measures of quality and safety, the quality and safety markers (QSMs); and literature reviews in support of the evaluations.

The emphasis in this research is on the work of the Commission through to mid-2015, and, throughout this report, we note where it was felt that progress was being made on issues identified in earlier stages of the evaluations. The evaluations particularly emphasise the Commission’s work with DHBs to improve quality and safety within DHB hospitals, which is where the Commission’s early work has been focused.

In this Report, we present the findings from the three evaluations.

10

1.1 Report Structure

This Report is set out as follows.

In Section 2, we provide an overview of the Commission’s work.

In Section 3, we summarise the evaluation questions, framework and methods.

In Section 4, we explore the perspectives of key stakeholders in terms of how successful the Commission has been in effecting change. It examines issues relating to: the Commission as an independent Crown agent; its role and influence in terms of health quality and safety; perspectives on the ‘Open for better care’ Campaign; capacity and capability and the impact of the IAD programme; and consumer engagement.

In Section 5, we explore the outcomes from the Commission’s work, including examining whether the desired changes in safety practices and reductions in harm and cost are occurring.

In Section 6, we examine issues relating to the likely sustainability of improvements in quality and safety.

In Section 7, we draw overall conclusions and set out a series of recommendations.

In Section 8, we set out references referred to in the report.

11

2.0 The Health Quality and Safety Commission

The Health Quality and Safety Commission was established as a Crown agent in New Zealand in late 2010. The establishment of an independent body to encourage, support and guide improvement in health care practice in New Zealand was an outcome of the 2009 Report of the Ministerial Review Group (‘the Horn report’) (Ministerial Review Group, 2009) which recognised the value of having an agency with a mandate to address quality and safety in health care provision in New Zealand. The Commission reports directly to the Minister of Health; it has its own dedicated staff. The Commission is governed by a board of eight members appointed by the Minister of Health.

The Commission’s vision is that New Zealand will have:

‘a sustainable, world-class, patient-centred health care and disability support system, which will attract and retain its workforce through its commitment to continually improve health quality, and deliver equitable and sustainable care’ (Health Quality and Safety Commission, 2014b).

The Commission’s mission is to ‘work with everyone to ensure New Zealand has the highest quality health care’. To achieve this mission, the Commission pursues the Triple Aim1 of:

1. Improved quality, safety and experience of care for consumers2. Improved health and equity for all populations3. Best value for public health system resources (Health Quality and Safety

Commission, 2014b).

The Commission’s values are centred around its work being patient-centred, evidence-informed, open and transparent, undertaken in partnership with stakeholders, and through strong leadership in setting the direction for health quality and safety in New Zealand (Health Quality and Safety Commission, 2014b).

The Commission has six main areas of interest (sourced from Health Quality and Safety Commission 2014c and Health Quality and Safety Commission 2015a):

1. Preventing and reducing harm, focusing ona. Reducing harm from falls, especially older people receiving care, whether in hospital,

residential care, or in their own home. b. Infection prevention and control, to reduce the number of infections from care; key

projects focus on Hand Hygiene; Central Line Associated Bacteraemia (CLAB) prevention; and Surgical Site Infection Improvement (SSII) to reduce the harm from surgical site infections.

c. Reducing perioperative harm following surgery (‘Safe Surgery NZ’), including through improved teamwork and communication.

d. Improving medication safety, to reduce the number of New Zealanders harmed each year by medication errors.

1 The Commission’s Triple Aim is based on the Institute for Healthcare Improvement Triple Aim (Institute for Healthcare Improvement, 2009).

12

2. Learning from harma. Four permanent, statutory mortality review committees review the deaths of

children and young people; deaths of babies and mothers where death is caused by pregnancy or childbirth; deaths resulting from family violence; and deaths associated with surgery. One temporary committee is ascertaining the feasibility of suicide mortality review. The focus of the reviews is on preventing future deaths from the above causes.

b. Serious adverse events reporting, to encourage the reporting of all adverse events so that the events can be learnt from and improvements made in service delivery. The Commission supports the analysis, reporting and management of adverse events.

3. Learning from dataa. The Atlas of Healthcare Variation: an interactive tool drawing on analyses of

variations by geographic area in the provision and use of specific health services and health outcomes. This work is designed to encourage discussion about why any differences exist, and to stimulate improvement.

b. Quality and safety indicators: summary indicators covering an internationally recognised range of aspects of quality – safety, patient experience, effectiveness, access/timeliness, efficiency and equity; designed to track change over time in the performance of the New Zealand health system.

c. Quality and safety markers: a mix of process and outcome measures focused on driving improvement for the four key safety priorities in 1 above. The markers set expected levels of improvement and enable public reporting of progress.

d. Quality accounts: prepared by DHBs to give an account for the quality of their services and to tell each district health board's (DHB’s) story – ie their commitment to evidence-based quality improvement, where improvements are needed and how local communities play a part in making health services better and more responsive. The Commission supports DHBs through workshops and guidance.

4. Involving consumers a. Partners in Care programme. The Commission’s four-year plan (2012–2016) includes

work-streams designed to increase health literacy, improve consumer participation, and develop leadership capability for providers and consumers (for further details, see Section 4.6 below).

b. Patient experience surveys. The hospital survey previously used by DHBs has been revamped, and a new survey designed to capture experiences in primary health care.

5. Improving systems a. Improving capability and leadership for quality improvement, including the

Improvement Advisor Development Programme (IAD Programme) (for further details, see Section 4.5 below).

b. ‘Open for better care’ Campaign, covering the four topics set out in 1 above (for further details, see Section 4.4 below).

6. Addressing disparities 13

a. This emphasis runs across all programme areas, taking into account the wide-ranging determinants of health).

3.0 Evaluation Questions, Framework and Methods

3.1 Evaluation Questions

In evaluating the Commission’s work we aim to answer the following high level questions:1. How successful has the Commission been in effecting change? 2. Did the desired change in safety practice occur? 3. Did a reduction in harm and cost occur?4. Has the approach to change the Commission has adopted resulted in sustainable

improvement?

Three separate evaluations were undertaken at the same time; ie evaluations of a) the national ‘Open for better care’ patient safety campaign; b) the improvement advisor development programme; and c) the overall impact of the Commission’s work. In practice, the three evaluations are tied together, given the fieldwork explored each of the three topics at the same time using the same data collection sources and processes.

3.2 Evaluation Framework

The Commission’s work involves a large number of programmes and projects, and, in order to be successful, the Commission must also work well with a wide range of stakeholders. Not all programmes and projects are likely to be successful in all parts of the country, and it is likely that the different contexts in which the Commission is operating will influence success.

As a result of this, the evaluation framework we are using is a simplified, high-level ‘realist evaluation’ (Pawson and Tilley, 1997), enabling the evaluations to examine the differing contexts in which the Commission’s work takes place and how this influences the achievement of the desired impacts from the Commission’s work. This evaluation approach is particularly relevant to initiatives where the local context is very much likely to influence the achievement of outcomes.

3.3 Evaluation Methods

The evaluations are pragmatic mixed methods (quantitative and qualitative) evaluations (Cresswell, 2009). By using both qualitative and quantitative data, the evaluations are examining ‘why’ certain things happen (or do not happen) as well as ‘what happens’. Qualitative interview data enables the evaluation to explore in depth the reasons why (and why not) quality and safety is being improved. Quantitative data includes both administrative data relating to health care outcomes and health care processes (eg, health quality and safety markers and indicators) and survey data to examine a large number of people’s views on how the Commission is influencing quality and safety in their workplaces, and which provides an assessment of safety culture in New Zealand DHBs.

The evaluations include both process and outcome measures. The process aspects focus on how the Commission is going about its work, and includes documentary data; qualitative,

14

interview data; quantitative data from surveys; and analyses of health quality and safety markers. The outcome measures examine the Commission’s impact in relation to its goals and objectives, and any unanticipated effects, and also include both qualitative and quantitative data, including quantitative data on key quality and safety markers.

Data collection included:a) Review of Commission documentsb) Interviews with Commission Board and staff membersc) Interviews with District Health Board (DHB) Chief Executive Officers (CEOs)2 d) Interviews with DHB managers, health quality and safety leaders, clinical leaders, and

clinical staff through three case studiese) A health professionals’ perceptions of quality survey, undertaken nationally with clinical

staff in 19 DHBs34

f) An analysis of quantitative measures, the Quality and Safety Markers (QSMs)g) Interviews with a range of key national stakeholder organisation senior leaders (Ministry

of Health, ACC, Treasury)h) Literature reviews in support of the evaluations.

Note that there are three case studies, but the case studies in practice cover the views of those working across four DHBs. One case study involves the Hutt and Wairarapa DHBs. (which were working together during these evaluations, alongside Capital and Coast DHB). The second case study involves Canterbury DHB. The third case study is Waitemata DHB. Details on each of the case study sites are included in Appendix 1.

Table 1: Participants in the Evaluations and Timing of Data Collection

Participants No. of Participants Timing

HQSC Staff and Board Members 15 Late 2014; Early 2015

DHB CEOs 19 Mid 2014

Improvement Advisor Development Programme participants 16 Mid 2014

Other Senior DHB staff 48 Hutt/Wairarapa – Late 2014Canterbury – Late 2014/Early 2015Waitemata – Early 2015

Other Key Stakeholders (Senior MoH, ACC, and Treasury officials) 6 Early 2015

National Health Professionals’ Perceptions of Quality Survey 4101 in 19 DHBs Early 2015

The data collection occurred over an 18 month period. The interviews resulted in more than 80 hours of transcribed interview data. Document and interview data were analysed

2 Nineteen of twenty DHB CEOs were interviewed; one declined as he is on the HQSC Board.3 One DHB did not participate (MidCentral DHB) as it had recently completed its own survey.4 A separate report sets out the full findings from the survey.

15

thematically. The Health Professionals’ Perceptions of Quality Survey was designed to explore the work of the Commission, and to provide baseline information about the health quality and safety culture in DHBs. It was developed by the research team, drawing on a number of existing questionnaires, and distributed to all clinical staff via email by the General Managers Human Resources at 19 DHBs. We estimate a response rate of around 10%. Those responding are representative, however, of the health professional workforce in New Zealand (see Gauld and Horsburgh, 2012). Survey data were analysed descriptively. Ethics approval for the evaluation was granted by the Victoria University Human Ethics Committee.

3.4 Limitations First, the emphasis is on the work the Commission is undertaking with DHBs, particularly emphasising its work to improve the quality and safety of DHB hospital services. We have not assessed the impact of the Commission’s work with the primary health care, disability, or aged care sub-sectors.

Second, the case studies provide more in-depth data on what is happening in individual DHBs, but the three case studies involve three very different contexts and the choice of sites may have influenced our findings.

Third, the three separate evaluations were all completed at the same time, but the work of the Commission is not so easily compartmentalised in practice, particularly in the minds of those working in DHBs.

Fourth, the evaluations took place during the third and fourth years following the Commission’s establishment and, in analysing our data, we took into account that the Commission is still a young organisation.

Finally, the quantitative analyses of trends in QSMs also have a number of specific limitations, which are germane to evaluations such as this one. These arise from analyses: covering a short period of time; not being able to determine whether it is the Commission’s work that is resulting in change nor which programmes or projects are responsible; not taking random variation into account; possibly finding ceiling effects; drawing off a 2012 baseline that may or may not be an appropriate baseline; and from the data used subject to a number of assumptions that may also impact on the data’s validity. (For more detail, see Section 5.2.4 below.)

16

4.0 How Successful has the Commission been in Effecting Change?

4.1 Introduction

This section of the report explores health sector perspectives on the work of the Commission, relating to how successful the Commission has been in effecting change. Section 4.2 focuses on how the health quality and safety agenda has been influenced by having an independent Commission. Section 4.3 examines the Commission’s role and how the Commission is changing the ways in which key stakeholders think about health quality and safety. Section 4.4 explores perspectives on the ‘Open for better care’ Campaign. Section 4.5 examines issues relating to capacity and capability and perspectives on the Improvement Advisor Development Programme (IAD Programme) introduced by the Commission to enhance capability. Section 4.6 focuses on issues relating to consumer engagement. In some cases, we separately identify the general source of the findings (Commission Board and staff members; sector leaders which include DHB CEOs, DHB senior leaders, and national stakeholders; and case study participants), in order for the Commission to identify commonalities and differences in views across the sector.

4.2. How has the Health Quality and Safety Agenda been Influenced by having an Independent Commission?

4.2.1 Background

The Commission is established as a Crown agent. Crown agents are a form of Crown entity, which are legal entities in their own right. A decision to assign a government activity or function to a Crown entity indicates that the function should be carried out at ‘arm's length’ from the government (State Services Commission, undated). A Crown agent must, however, give effect to government policy when directed by the responsible Minister. Crown agents are governed by Boards appointed by the relevant Minister (in the case of the Commission, the Minister of Health) and their performance is monitored by a central government department (in this case, the Ministry of Health). Here, we consider the establishment of the Commission as a Crown agent and explore the benefits and downsides of this form of entity.

4.2.2 Brief Review of Existing Literature on the Effectiveness of Independent Agencies

Various studies highlight the different reasons why an independent agency may be an important alternative arrangement for delivering on government policy goals. First, there may be potential reductions in overall costs in the broader policy delivery area if the agency has the expected impact on improvement in its field of responsibility (Gormley et al., 2013). Second, creation of an independent agency may garner increased attention to the policy area and, given the location outside of the executive branch of government, may also have greater potential for ensuring that policy deliberations are transparent, fair and reasonable. Following from this, the focus may be more tilted toward engaging with the service provider community and public, rather than servicing political leaders and the government of the day which is a core function of executive agencies (Nunes, Rego and Brandao, 2009). In this regard, the impact of partisan political preferences is also potentially reduced (Khemani,

17

2007). Third, and importantly, this ‘removal from politics’ and potential for sector and public engagement can mean an independent agency is ultimately more likely to be effective than an executive agency. This can be particularly important in terms of policy sustainability, impact and the development of partnerships with the sector. Therefore, an independent agency created and maintained for specific purposes may improve government credibility in the policy area (Elgie, 2006). Creation may also be as the particular ‘task characteristics’ in question are better suited to an independent agency (Bach, 2012).

There are various advantages and disadvantages of agency independence. The primary advantages appear to be in the potential to develop closer and longer-term relationships with service provider communities, and to build inter-subjective understanding and ultimately trust (Koop, 2011). This is especially important in terms of delivering sustainable policy and practice solutions that are supported by service providers in a context of high administrative devolution, as is the case in New Zealand’s health system. The focus on the longer term and sustainability would be more difficult to achieve by an executive agency which is more vulnerable to politicisation and subject to periodic shifts in focus when governments and political leadership change (Christensen & Laegreid, 2006). Health care quality improvement and patient safety is, arguably, an area where the focus on longer-term planning and relationships is paramount as only limited gains are likely to be made with short-term and one-off projects (Bisognano and Kenney, 2012; James and Savitz, 2011).

A further advantage is the singular focus that often sits behind independent agency creation. This means that the work of the agency is unlikely to be ‘crowded out’ by other activities that can often take higher priority in a larger executive agency that has multiple and often evolving responsibilities (Christensen & Laegreid, 2006).

Yet another advantage is in the legal status of the independent agency. Executive agencies may be viewed with suspicion for the fact that they are often directive and possess the legal authority to demand compliance from the policy sector and service providers. This can undermine good will and relationships. An independent agency may function in a manner more akin to an ombudsman, where the role is to analyse, provide advice and work with key actors to reach an agreeable outcome (Palmer, 1987). Decisions and policy suggestions are not binding but tend to be adhered to due to the particular largesse of the office and respect within the community with which it is held.

The potential disadvantages include the contribution to ‘agencification’ (Christensen and Laegreid, 2006). This describes the creation of many new small agencies to take on various functions which might be either new or in keeping with a policy of devolution. With agencification can come challenges with policy co-ordination, responsibility and accountability. In an area such as health quality and safety, where it is ‘everyone’s business’ to have such a focus, these challenges may be greater compared to those of, for example, pharmaceutical purchasing where a designated agency perhaps has a more specific mandate and scope of responsibility (Gauld, 2014). With agencification can come considerable transaction-costs, these being the time and costs in coordinating the activities of the various agencies involved in the policy area.

18

Thus, there can be strong arguments for agency independence, although this can also contribute to policy and administrative complexity. Independence is contingent to a degree on the particular policy area and sector structure, and the goals of government. Independent agencies can be more effective than executive agencies for the fact that they sit at arms-length from government and have a designated responsibility; they also demonstrate a commitment to the specific policy area which can send an important message to the sector.

4.2.3 Commission Board and Staff Members’ Perspectives

Amongst the Commission Board members and staff, the establishment of the Commission as a Crown agent was viewed positively. Some noted that this was in line with developments internationally where a number of independent entities had been established to address health quality and safety. Some also noted that the establishment of a new, separate agency – at a time when there was reluctance to establish new organisations in New Zealand – also reflected concern that there were barriers to improving quality and safety through the Ministry of Health. There was a view that the Ministry had not been actively pushing a health quality and safety agenda and that indeed it was difficult for the Ministry to do so, given that any issues raised around health quality and safety would quickly become issues relating to accountability, making it difficult to bring an improvement focus to the topic.

Most agreed that it was an advantage to be a separate organisation:

‘I had to come from the position of being quite agnostic really about whether or not you should have a separate agency and all. But having...gone into a separate agency, being a Crown entity, does give you greater flexibility to engage faster with the sector and you are closer to the sector, they don’t perceive you with the same suspicion that you’re perceived with if you’re with the Ministry. And you don’t have to wear the double edged thing… “I’m from the Ministry but I am here to help you”… [it helps you get] over that barrier first.’

‘I think having the ability to be independent is really important and we have a really clear message from the providers that we work with, because we do a lot of work in the private sector and aged residential care – we get a clear message that being at arms-length from say the Ministry [is good] … in the early days we were described as the Switzerland of Health…the neutral party that people trusted to share information with and it is the information piece that is very important…the information would be used for improvement reasons…we have also been able to focus on patient safety and improvement in a way that other agencies haven’t been able to because they have got multiple demands.’

19

Some did note, however, some disadvantages and these included spending a lot of time ‘managing up’ and that ‘government still thinks of us as an “agent” that can do their bidding’. This requires managing these tensions, keeping Ministers on board with what the Commission is doing at all times, and managing the relationship with the Ministry of Health. The Ministry is considered a regulatory body and the Commission staff think it is to the advantage of safety and quality improvement for the Commission to be perceived as separate from this and to engage in the health quality and safety agenda as collegial peers.

However, independence does not really mean completely separate, and a number of staff addressed the ambiguous nature of being a Crown agent:

‘It [has] had its ups and downs … The whole business of independence in the health care sector is kind of a bad word, because nothing can be done independently … If we want to improve the quality and safety of health care in New Zealand in a sustainable kind of way...that involves a lot of groups…working collaboratively, working around [an] agreed agenda and…bringing together all the different things that are going to nudge the system along over time to get it where we want it to be….having an independent Crown entity is good...a little removed from the Ministry…and the Minister. But at the same time, you know, it’s got to work collaboratively with the Ministry and the National Health Board and other agencies if it is going to be successful in what it wants to do…’.

‘…we are still seen by some as being from the Ministry... I do a lot of visits to district health boards and other providers and I am frequently introduced to groups of staff as [being] from The Ministry…and sometimes it feels like we are a mini Ministry – too bureaucratic…it is good having an “independent” organisation, just so long as we are listening to people, the wider sector, not just to our political masters, I don’t think we do listen to what the sector wants…what happens if the sector wants something…and the Minister wants something different?’

One interviewee further noted that the Commission has had to use accountability processes through the Ministry of Health to support its work, for example including health quality and safety within the Minister of Health’s annual Letter of Expectations and including some reporting requirements in the Crown Funding Agreement accountability documents. They felt, however, that on balance the Commission saw its role as an improvement agency as opposed to an accountability agency.

4.2.4 Sector Leaders’ Perspectives

The majority of sector leaders also thought there was value in the Commission being an independent crown entity outside of the Ministry of Health. The benefits were that independence enabled a ‘helicopter view’ or a ‘third eye’ and was more efficient, not so bureaucratic, and able to completely focus on quality improvement). It was also considered that from the point of view of the public it was very important for the Commission to be perceived to be independent, rather than under the direct control of the Ministry of Health. The role of the Commission and its focus on quality improvement

20

was contrasted with the role of the Ministry of Health, which was commonly perceived to be a funding, regulatory, and compliance body, focused on accountability.

As one participant observed:

‘…in terms of a strength of the Commission is the degree of independence that they have. … I think that should be really exploited because … they are not within the grasp or the control of anyone either. They are slightly removed from the Ministry, they’re not accountable to the DHBs, they’re not accountable to providers … they’re not, you know, they’re not captured by anybody. I would really celebrate the maintenance of that independence. I have seen overseas independent bodies that kind of just provide all the information and the data out there in a safe, transparent way. I think the Quality and Safety Commission is the best vehicle that we have at the moment … it encourages us to utilise the information for continuous improvement, rather than using it to beat us up about policy, or funding or whatever.’

A number of sector leaders thought that the degree of independence should be exploited and increased. They also considered that the Commission was the best vehicle or means of addressing quality and safety and that the Commission’s approach differed from the Ministry: it was seen that the dominant culture of the Ministry was one of reporting and compliance; in contrast, the Commission persuaded and provided support for addressing quality and safety concerns within the DHBs. Many described the Commission as being at ‘arms-length’ from the Ministry and this was seen by most as a positive attribute, although some thought that there was a politic connected to this ‘arms-length’ where the Commission was required to report to the Minister and, therefore, could be vulnerable to political manipulation.

In addition, many thought that independence supports the work that is needed to be done in New Zealand, for example in terms of developing national standards to manage issues highlighted recently in the United Kingdom, specifically the Mid Staffordshire problems with quality and safety of patient care, and to challenge the status quo:

‘…need national standards and the HQSC is best placed to do this and exercise a degree of independence…’

‘…they can challenge the status quo without feeling they are speaking out of turn, to identify areas and be a voice that can be heard…not captured by accountability.’

With respect to independence, many referred to the Commission staff and there was a consensus that the Commission staff were easy to deal with and well respected. At the same time, some queried whether the Commission had the mandate it needed. As one observed:

‘I think they made some very good appointments… And I think they have been wonderful champions from the start… I don’t think they have been zealous or dogmatic… I do think they have been very respectful in the way they have worked with DHBs and from my observation other agencies…They have established very

21

respectful relationships and they haven’t come in with a too dictatorial attitude. Having said that, I think at times they could have been, or maybe feel they can’t as they don’t have the mandate. But I would like to see them being a bit more directive, I think particularly in relation to adverse events and as to how we investigate and report and the follow-up on adverse events’.

A number of sector leaders connected adverse events with independence, but also questioned the degree of power the Commission has and whether persuasion is enough. Some felt the Commission ‘could have a stronger voice’ and fulfil the role of ‘watch dog’ over the sector (Hardy, Gauld and Holmes, 2015).

Others recognised the importance of independence but felt that the Commission has some way to go in recognising the benefits of such independence:

‘The theoretical advantage of it [the Commission], independence, is that it is able to speak clearly and independently, without being sucked into the politics of the Ministry. The Ministry exists to serve the policies of the Minister, a Commission is free to give independent advice….it hasn’t really happened as yet. I don’t think the Commission is speaking independently about anything…’

Some sector leaders felt that the Commission must exert influence not only over the DHBs but also over the Ministry and policy if real improvements in health quality and safety are to be made.

A small number of downsides from having an independent Commission were mentioned in the interviews with sector leaders. These included that the Commission was not, in the early days, seen to be taking a sector-wide view. Having an independent Commission also meant that directives were coming from a number of sector leaders or organisations, such as the Minister of Health, the Ministry of Health, ACC, the Health and Disability Commissioner, and the Commission, with DHBs not always clear if such leaders or organisations had similar perspectives on key issues. It was felt that greater clarity and timeliness in key messages would assist DHBs in planning their work. It was also pointed out that there is a need for the lead sector organisations to work through how the various roles fit together, to overcome the fragmentation that can occur where independent agency roles overlap with those of other organisations.

4.2.5 Case Study Participants’ Perspectives

Many of those interviewed in the case studies also considered it was important that the Commission be independent of the Ministry of Health. Further, the Commission’s focus on quality improvement as opposed to targets, accountability, regulation, and compliance was appreciated. The Commission was seen as somewhere for organisations to go to where discussion about problems and solutions can be more open. Several interviewees noted that if it were the Ministry leading, then Ministry advice would likely be ignored or resented.

Several participants questioned the actual degree of independence that the Commission had in practice, however. They felt that independence was a moot point where an

22

organisation was still reporting directly to the Minister of Health, who could still make or overturn decisions. This point was often linked with a view that the Commission should speak out more about the issues that it saw as compromising health quality and safety in New Zealand (see Section 4.3).

A few case study participants could, however, see no problems with health quality and safety issues being part of the Ministry of Health’s role, with some pointing to the potential benefits from an integrated approach (eg, a whole-of-system approach, fewer organisations leading initiatives, and improved consistency).

Overall, at all three case studies there was strong support for the Commission being independent of the Ministry and this relative independence was seen to be a strength that would ultimately assist in a positive response to improving quality and safety in the sector. This independence was viewed as setting the Commission outside of the compliance role associated with the Ministry.

4.3 What is the Commission’s Role and How is the Commission Changing the Ways in which Key Stakeholders think about Health Quality and Safety?

4.3.1 Commission Board and Staff Member Perspectives

4.3.1.1 Commission Role and Goals

Amongst the Board members and staff of the Commission, the core roles of the Commission were seen to be about leadership, communication, and enthusing people about the importance of quality and safety; facilitating change; and especially bedding in a quality improvement culture across the sector. The ultimate goal is to improve the quality and safety of health care in New Zealand.

In addition, staff saw a role for the Commission as being a strategic catalyst for change and to create tension and debate in the sector, speaking at the system level. The Commission was also seen to have a role as an information broker bringing different agencies together (Ministry of Health, ACC and the Health and Disability Commission) to leverage capability and capacity, share intelligence and knowledge, and ensure that inter- agency goals are aligned to realise improved health quality and safety outcomes. The Commission staff engaged with international equivalent organisations and their programmes to inform their work programme. In addition, international experts are invited to present and engage with the sector on regular basis to share their expertise and knowledge and to promote dialogue.

Interviewees also noted that the role of the Commission was focused on quality improvement as opposed to quality assurance. The latter is viewed as a process of checking processes against standards to ensure that organisations comply; examples include building standards or hospitals having the right equipment in various clinical departments. Instead, as one interviewee noted:

23

‘Quality improvement is a different process. It’s a hearts and minds game where you try to get people to live perpetually in a process of examining what they’re doing, understanding where it has capacity for improvement, doing things to improve it, measuring that improvement … and then carrying on…’.

This interviewee also noted that in practice there are compliance elements to the work of the Commission, for example where there are targets and auditing of practices (such as with the hand hygiene programme), but that the Commission emphasised ‘trying to engage them [hospital staff] in ways of doing this better that are actually affirmative actions’.

The early work of the Commission focused on DHB hospital services and was seen to be ‘hospital centric’. It was recognised, however, that it would be important for the Commission to expand its work more widely, and to be seen to be a key part of the wider health and disability sector. Thus, more recently the Commission has responded to their mandate of having a broader sector wide focus by developing initiatives focused on aged residential care, disability services, and primary health care. This has involved developing networks with the key stakeholders in these areas and there is recognition that these are high risk areas for a range of quality and safety concerns, all with significant economic implications.

Several interviewees suggested that the Commission was perhaps established as it was thought by some that improved quality and safety might lead to savings within the health and disability sector. One interviewee noted that although reducing harm and improving health quality and safety does lead to savings at times, this may not always be the case. For example, they suggested that ‘it’s really cheap to kill people’ and in some instances, additional resources may be required to support improved health quality and safety in order to save lives. Another interviewee noted that savings from improved quality and safety do not necessarily lead to lower expenditures, but do allow resources to be reallocated and hence contribute to achieving better value-for-money, ie spending resources positively to improve health as opposed to having to spend resources to rectify errors.

4.3.1.2 Facilitating Quality Improvement

The Commission Board and staff members recognised that the Commission plays a facilitative role, both regionally and nationally. Holding national and regional meetings allows the sharing of knowledge and experience between health professionals and, ultimately, is seen as assisting with achievement of the more successful interventions. The role of facilitator and efforts to address capability within the sector were considered important parts of their work; indeed, for some, it was felt that this was where the Commission should focus its work, leaving specific programmes and projects to be undertaken at a regional or district level. But others disagreed, noting the importance of programmes and projects, including providing practical support, as key means of improving quality and safety in services as well as building capability.

24

The emphases on facilitation and capability were also linked with issues around measuring progress. As one participant observed:

‘We have to be careful not to default into specific programme areas, you know, the harm avoided as the measure of success…some of those important areas around consumer engagement and partnership, building capability is a really important focus, but much harder to measure…’

Following these observations, many also talked about making change sustainable and how leverage could occur from following people involved in an initiative and seeing what they may contribute to another different initiative, that is:

‘…what has happened to the people, the key people that were involved in that particular project, at a regional or local level and what have they gone on to apply those skills too for ...other particular initiatives.’

The Commission staff also thought that they had worked hard at acknowledging that not all in the sector are in the same place with respect to quality improvement and that success will occur at a different pace for some. It was their role to ensure that improvements were being made and that ultimately all had the same destination in mind. A regulatory body would typically be more critical, but the Commission’s role was one of encouraging people to move on the quality improvement agenda and in part this also involved acknowledging that New Zealand has a good health system with good people who were trying to do well and achieve cost effective medical practice. This was considered key in the view of Commission staff to ‘winning hearts and minds’ and bringing about change in the sector.

Most Commission staff were of the view that they should work toward changing the culture in health care providers. One interviewee suggested, however, that ‘top down edicts to say you’ve got to improve your culture, are very poorly received’, and that New Zealand has quite a good culture. They felt it important to recognise that New Zealand has a good system, ‘with good people that are trying to do well’.

To achieve their goals, all staff realised that they needed to get buy-in from the sector. Thus, leadership was seen to be essential. At times, getting buy-in was a challenge, however. Some staff reflected on how the Commission had to encourage participation rather than mandate it but that at times some mandate would be useful.

For example, a number of staff noted that there was no legislative imperative for DHBs to report serious adverse events and that when the Commission was established less than half of the DHBs were reporting serious adverse events. Since the establishment of the Commission, reporting had improved considerably with all DHBs now reporting serious adverse events, and it appears that the Commission has been successful in encouraging DHBs to be more open in their reporting of serious adverse events. It was noted that other areas of the health system, including primary health care and midwifery, do not report serious adverse events, and, as a consequence, the New Zealand public have no

25

knowledge of harm in these settings:

‘…but there are 1500 GPs, 3500 independent midwives and all the disability services…there are huge areas where we and therefore the public have no idea what is going on and I think it is unacceptable.’

It was considered a challenge for the Commission to promote the reporting of serious adverse events across the entire health sector, not just within DHBs, and having some mandate in this area would assist.

26

The work of the mortality review committees, reporting of adverse events, and the quality accounts were all seen as key ways of creating a more open culture. This more open culture was also reflected in the ‘Open for better care’ branding for the national health quality and safety Campaign (see below). Each project was aimed at ‘team work and discussing things openly and … open disclosure…’. It was noted that there was still some nervousness around this, particularly amongst the medical profession, and that there was still some way to go to create a more open culture. It was also noted that, at present, the number of adverse events being reported was increasing, but it was felt that this was due largely to the better reporting that was occurring in the sector. It was recognised, however, that it was expected that the number of adverse events reported should eventually begin to fall.

There was a consensus amongst staff that their work needed to be evidence-based, and that they needed to take a lead across the sector and be a central repository of knowledge and resources with regard to health quality and safety improvement. A number of staff stressed that the Commission was an improvement agency not an accountability body. It was noted that this was a ‘fine line to tread’ but an important one. Central to their operational work was the need to build, maintain and sustain relationships across the sector and all noted that this was a challenge given the breadth of the sector and the size and resourcing of the Commission.

Helping providers to make improvement changes was not simply about providing material resources but also, according to some Commission staff, about ‘shining the light’ onto key issues, providing measurement data and providing tools, knowledge, and capability to affect change across the sector.

4.3.1.3 The Work of the Commission

A number of Commission staff reflected on the establishment of the Commission and the early work of the Commission. They stressed that the Commission did not commence with a clean slate; that it inherited a number of projects from (i) the former New Zealand Quality Improvement Committee, which had been formed in 2007 within the Ministry of Health; and (ii) and the Mortality Review Committees, which are statutory committees that review particular deaths in order to learn how best to prevent such deaths in future (Health Quality and Safety Commission, 2015a). An early decision was taken to continue with key programmes and projects, in order to build from there, and not simply unpick the investment that had already been made in particular programmes and projects. For some, however, there was a view that it was necessary to take stock and ensure programmes and projects were evidence-based and that programme and project work did not dominate the focus or prevent strategic planning. Similarly, in the early days, while the organisation was still relatively young, many thought that the Board should become more strategic rather than focused on operational issues. One interviewee reflected that they felt that the Board had made significant progress over time, particularly in better understanding its governance role.

27

Some stressed, however, that it was important not to become too ‘blue sky’ and to remember what improving quality and safety is about:

‘I see large amounts of preventable harm taking place…I see people dying and being harmed as a result of serious harm, so I think that we should exist in order to stop preventable harm taking place and I think that’s most people’s view…our job is to improve the quality…all this harm just shouldn’t be happening.’

‘The key role for us is to be…championing quality and safety across the health and disability sectors, around how to improve quality and safety, so the ultimate aim is that patients are safer.’

Many of the staff spoke about the speed with which they had been working, the long hours and small teams who carry responsibility for a range of initiatives and how they were aware that it was necessary to be seen to be ‘getting runs on the board’ and to demonstrate their effectiveness as an improvement agency. It was clear to the researchers that the staff at the Commission were dedicated to improving quality and safety across the health sector.

It was also recognised that given its resource and capacity limitations there was only so much the Commission could do. One interviewee noted that the Commission has a budget of around $13 million per annum, a quarter of which goes on the Mortality Review Committees. This leaves around $9-$10 million for all the other work. This, in turn, increased the importance of leveraging the work of the Commission with that of other organisations, such as DHBs. This interviewee felt that as the Commission’s work gained traction, it would be necessary for the level of investment in the Commission to increase, in order to achieve more over time.

Many Commission staff addressed the issue of resourcing. As another interviewee noted:

‘...we have to come from a place of knowing what we have, who we are and what we can offer and be realistic and we can’t … raise false hopes with people. We can’t raise people’s hopes and that’s the balancing kind of stuff that you have to do – it is very challenging and quite hard work’.

Commission staff particularly pointed out that the budgets they had for programmes were constrained and that this ultimately impacted on scale, scope, pace and spread. With greater resourcing they felt they would be able to go faster, develop new initiatives and move toward addressing quality and safety across the whole health system. At present, they have to stop an initiative if they wished to proceed with a new one. In addition, while they all considered that they had become skilled at working within their resourcing and that they were ‘nimble and quick’, the resource constraints impeded their ability to stay relevant to the sector while simultaneously ensuring what gains that had been made were sustained.

28

Key strengths of the Commission were seen by the Board and staff to be its focus purely on health quality and safety; its ability to relate directly and quickly to DHBs; the engagement of nationally-respected clinical leaders in the work of the Commission, providing a direct line into the work of clinicians; a commitment to collaboration and partnership; a strong emphasis on good communication; and a Campaign that has been seen to work extraordinarily well (see below). It was also felt that the Commission was being flexible and ‘nimble’ in its work, and that being ‘outside the bureaucracy’ assisted in this. It was recognised, however, that New Zealand is well behind other countries in emphasising quality improvement and that the Commission still had a way to go to build its influence, particularly beyond the DHB hospital sector.

4.3.1.4 Achieving and Assessing Success

Commission Board members and staff members noted that there would be a number of ways of assessing the success of the Commission. One interviewee noted:

‘I think we will have been a success … when people jump out of bed excited about using the information … that they see about their patients and they think, maybe not every day, but they think at least once a week about how to do it better.’

They also noted:

‘I think we will be a success when … there is a, somebody called a consumer worker working in a service or in wards. We’ve got consumers working on Boards being involved in planning and evaluating services then I think we will be a success and we’ve got more staff able to think about quality improvement and measure it and know what they’re talking about and know how to interpret basic graphs and so on then I think we’ve been a success. And we’ve got a General Practitioner who, who thinks about how to improve the whole general practice all of the time.’

One key measure was the recognition that harm can successfully be reduced, as demonstrated in the CLAB programme: As one interviewee noted:

‘… you can be almost assured now when you go into intensive [care] we really need to know that you won’t get an infection in a central line …. Whereas it was almost accepted two years ago that actually a certain number of people always got infection. So what, that was life you know you kind of expect that sort of thing, actually no that you can actually do better.’

Other measures included the increased recording of adverse events followed by a reduction in such events, improvements in the QSMs, decreased admission times and admissions in hospitals from reduced harm, and savings able to be reallocated to more highly valued health and disability services.

29

4.3.2 Sector Leaders’ Perspectives

4.3.2.1 Commission Role and Goals

The majority of sector leaders thought the role of the Commission involved having a leadership role, setting the health quality and safety agenda and direction for New Zealand as their primary purpose, and utilising expertise and the local and international evidence base to define what the sector should be doing. As one participant observed:

‘The Commission’s role is driving and setting a quality improvement framework across DHBs, across the health system…It needs to be strengthened.’

Another added:

‘The role of the HQSC is championing quality in the health system and providing strategic direction for health quality and safety in New Zealand.’

A third saw the role of the Commission as absolutely ‘pivotal’ with good health care quality and safety a fundamental requirement for a health system.

Further, a key role for the Commission was in supporting health care providers, through knowledge, information and advice. Some felt that key to the Commission’s success would be its ability to listen to those in the sector about what they see as key health quality and safety issues and ask how the Commission can support this. It was felt that such an approach was more likely to get ownership from health care providers such as DHBs.

Developing nationally consistent processes and indicators where national comparisons could be made were seen as key roles for a national organisation such as the Commission (eg, patient satisfaction surveys, monitoring of key indicators, quality and safety reporting). In addition, it was felt that the Commission should be involved in sharing successful innovations from around New Zealand and the world. Supporting monitoring and evaluation of quality improvement initiatives was also seen as a key role; this was viewed as particularly important for smaller DHBs that may not have resources themselves to undertake significant monitoring and evaluation.

A number of sector leaders also suggested that alongside a strategic, advisory and information provision role, the Commission may have a place in aligning lack of performance with consequences. For example:

‘…making strong recommendations and if the sector doesn’t pick these up there are consequences.’

‘HQSC should be a watchdog sitting on top with consequences if you don’t perform.’

Not all of those interviewed agreed with this approach, however. In particular, it was felt to be important for the Commission to work in partnership with provider organisations to help them overcome challenges, rather than simply pointing to poor performance (eg, ‘rank and spank’).

30

The observation was also made by sector leaders that the Commission was not working across the health sector as a whole and they felt it was important to do so. This was most obviously the case in relation to reducing falls, as so many occur outside of DHB hospitals. (There are now more recent Commission initiatives in the primary health care, aged residential care, and disability arenas, and this was recognised in our later interviews.)

There was recognition that the Commission was relatively recently established and that their scope would widen and evolve over time, although ensuring coherence was also seen as important:

‘Three to four years ago the Commission needed to get some runs on the board and now they have done quite a lot of things. The challenge now is not more and more new initiatives, but coherence – a clear coherent strategy that the health system is behind and fully engaged in how it all hangs together.’

4.3.2.2 The Work of the Commission

There was overall good support amongst the sector leaders for the Commission and praise for its work. One interviewee noted that prior to the establishment of the Commission, health quality and safety issues were ‘ghettoised’, but that this had changed. Connected to assertions about what their role should be were a number of observations about the extent of the Commission’s involvement in participating in and or generating public debate and playing a role of advocacy. Generally it was thought that the Commission could improve their profile in this regard.

It was also suggested, however, by one interviewee, that the Commission was underfunded for the role it had been provided with. Potential additional work might focus on included greater patient information so patients better understood the risks of treatment; more resources for improved data to support the work of the Commission; promoting best clinical practice, including through clinical guidelines (particularly where there is a major change in clinical practice desired or significant variation in practice), and through the sharing of successful initiatives within New Zealand’s DHBs; and a stronger academic and research arm to explore health quality and safety in depth.

A key issue raised by several interviewees was the need for the Commission to align its work with other key agencies, both nationally (eg, Ministry of Health, ACC, Health and Disability Commissioner) and internationally, such as the Health Roundtable, in order not to duplicate effort and waste health quality and safety resources, which can come, in DHBs, at the expense of clinical services. Working closely with DHB experts on key issues – such as key performance indicators – was also seen as important, as it enabled the Commission to draw on existing expertise from around New Zealand.

With respect to the operational aspects of the role of the Commission, there were conflicting views. For some, the role of the Commission was strategic, data collection, monitoring, and feedback and that their role should not be operational – rather this should be left to the DHBs. It was thought that these two roles had become blurred.

31

Conversely, it was also noted by some that the Commission needs to work more closely with those directly involved in health service provision. Thus, one participant, who represented many comments made, stated:

‘I think there needs to be more engagement with people on the shop floor.’

It was acknowledged, however, that such engagement was difficult to achieve. Some interviewees suggested that nursing and allied health staff were reasonably well engaged with health quality and safety and the Commission’s work, but that engagement was harder to achieve amongst doctors. Some stressed the importance of an evidence base to support the quality and safety improvement agenda as a key means of engaging with doctors.

The quality of the Commission’s staff was commented on by a number of participants, some of whom named various individuals as examples of excellence in the provision of knowledge, feedback, and encouragement, for the enthusiastic manner with which they engaged with the sector:

‘There is a lot of respect for the Commission staff around the DHB…supply good support if you ask for it…the right people are in the job; approachable, which is the key.’

‘Staff at the Commission work really hard to get things to us and make sure that we are kept in the loop – doing a grand job.’

‘Excellent individuals in the Commission and really good to work with and they see us as customers, so treated well and valued. I’m very impressed…They’re very committed and passionate about what they are championing.’

The Commission, in the view of most participants, had a range of strengths, and was widely praised for its work on sharing expertise and experience, and establishing networks to support health quality and safety improvement in New Zealand:

‘HQSC is quite low key, gives credit back to those doing the work – medication errors, falls etc. and health providers take ownership with HQSC quietly behind it. HQSC are providing systems and structure to support changing the culture. They came up and supported staff on the ground, didn’t take the credit, but had enough leadership to drive it a bit.’

A commonly expressed opinion was that along with the enthusiasm of the staff came perceived difficulties related to the speed at which things happened and the rapidity with which various initiatives were rolled out and or moved on from. It was thought that alongside an increased volume of reporting requirements and other pressures faced by DHBs, at times the Commission’s work added to the workload of DHBs which put pressure on existing initiatives and ‘ business as usual’ activities within the DHBs. This was particularly the case for smaller DHBs that had limited if any equivalent full time

32

positions devoted to quality and safety initiatives. It was also noted that when this was the case that there was fine balance that the Commission needed to negotiate to prevent some of their work being viewed as yet another compliance exercise:

‘The volume of stuff coming out of the Commission is hard to deal with because of all the other pressures we are under… The Commission needs to look at how their work intersects with other initiatives in the sector.’

‘I think they have been relatively effective in terms of, you know, setting that sort of quality indicator framework and continuance improvement there.’

For the smaller DHBs, it was challenging for them to respond to some of the work programmes as they simply did not have the capacity to complete them to the same level as larger DHBs (for example, the hand hygiene audits).

Overall, there was also support for the Commission’s programme work:

‘Auckland has taken a lead for surgical site infections and hand hygiene for the HQSC and doing the training. It is quite time consuming and different places have different levels of ability but it is good to have a goal. We actively support the HQSC’s work-streams.’

‘The topics they have chosen I am very supportive of. They have chosen good topics and they have limited work [needed] within the topics has been good. I am thinking of the medicines programme in particular… they have been very open to input from the sector. So even though this has been a Commission led campaign, it has been the sector’s campaign.’

One particular aspect of the Commission’s work that was mentioned by sector leaders included the quality accounts. A number of sector leaders were highly supportive of these as a way of engaging on health quality and safety, and celebrating achievements within DHBs. A number of interviewees felt that the purpose of the accounts was not always clear at the beginning, while some concerns were also raised over having quality account reporting that was separate to the reporting already in place (to the Ministry). Others felt that there needed to be a better balance between evidence and stories within the accounts themselves. One interviewee felt that it was important to include material from other organisations in their districts, such as Primary Health Organisations. Getting doctors on board was seen to be a challenge, as was getting community interest in the work.

Others strongly supported the work on adverse events and the importance of transparency on such events. One interviewee noted that the reporting made the events more visible, and gave a greater impetus to make changes. There was also a view, however, that adverse events reporting drew perhaps too much attention to deficiencies within the sector, which might alarm the general population. The Atlas was also mentioned by some as important work, but some concerns were expressed about the lack of context and risk adjustment in the analyses, and about the potential overlap with the work of, for example, the National Health Committee. Others wondered about the extent to which the Atlas was actually being

33

used by clinicians, and what work was going on to use the Atlas information to change clinical practice.

34

In terms of assessing the health quality and safety performance of DHBs, it was noted by sector leaders that a challenge for the Commission was to try to take a more holistic view of the performance of DHBs, ie, not just emphasising how well a DHB might be doing with respect to a particular programme or project, but how that programme or project was contributing to the overall performance of a DHB and signalling the range of performance that might be occurring within a particular DHB. Linking financial experts in with quality improvement initiatives was seen as a good way of doing this.

The importance of having the right pre-conditions in place to successfully achieve improvements in quality and safety was also noted by one interviewee, ie good leadership throughout a provider organisation or a culture that would support health quality and safety initiatives and that is able to take on board the substance of health quality and safety across all the work the provider organisation does. (These are factors identified in the existing literature as important in achieving success with quality improvement initiatives.) This points to the need for the Commission to work to improve leadership and culture and ensure that its work can flow through to a wide range of quality and safety, and overall, performance gains. A key aspect of this would be following through the implications of successful quality improvement initiatives on actual resource utilisation, with engagement of clinicians in understanding the resource implications of improvements an important component of such follow-through. The Commission’s emphasis on achieving the Triple Aim goals (improved quality, safety and experience of care for consumers; improved health and equity for all populations; and best value for public health system resources) supports this approach, but it was felt by sector leaders that there was some way for the Commission to go to achieve all of the Triple Aim goals.

While there was support for the work programmes, many suggested they needed more notice about them so they could plan; one suggestion a number of participants made was that the Commission provide them with a set year calendar noting what was planned and when so adequate preparation could be made for up-coming initiatives. (Since our interviews, this suggestion has been taken on board by the Commission.)

There was also significant praise for the communications from the Commission. A number of sector leaders commented on the focus on visual stories and on the website, which it was felt was easy to use.

4.3.3 Case Study Participants’ Perspectives

4.3.3.1 Commission Role and Goals

Interviews with case study participants also showed that there is a great deal of support, at all levels, for the role of the Commission in pursuing the health quality and safety agenda. They saw the role of the Commission in the same general terms as the Commission Board and staff members and sector leaders, emphasising leadership; the building of capability; developing an understanding of national issues through data analysis; and providing resources and tools, including sharing experiences from around New Zealand and internationally. (One interviewee wondered if the Commission might be a lead conduit for this rather than each DHB spending resources on local and international

35

connections.) One interviewee felt that the Commission should be the ‘face of patient safety’.

The tension between developing a national approach to quality improvement and enabling individual regions or organisations to develop their own local approaches to health quality and safety issues was well recognised by case study participants. Case study participants also recognised the importance of local leadership and champions and that without local leadership and engagement, improvements in quality and safety were less likely to occur.

Interviewees understood the importance of working in the first instance with DHBs, but would like to see an extension of the Commission’s work into primary health care, private hospitals, and the aged care sector. The last of these was seen as particularly important in the case of falls, where so many occur outside public hospitals. Interviewees also expressed concern, however, that a greater level of consumer engagement and empowerment was necessary. A number of interviewees suggested that the Commission was currently under-resourced to expand their work beyond DHBs and to support greater consumer engagement.

Some considered there should be a national standard for reporting for all DHBs, and perhaps minimum standards below which no DHB should fall, but there was some disagreement about whether the Commission should have regulatory power. Front-line staff members were more inclined to be supportive of the idea of the Commission having regulatory power. Some of those interviewed noted the importance of the Commission having the courage to challenge and tackle institutions and being more assertive with Chairs and CEOs; eg, where service delivery not based on evidence, leading to variations across the country.

4.3.3.2 The Work of the Commission

Interviews in the case studies suggested that the Commission has achieved a strong profile at the governance and management levels, among key senior clinicians and those directly responsible for health quality and safety programmes. Their profile was thought to be weaker amongst front-line staff, some of whom were unaware of the Commission and some of whom conflated the Commission’s work with that of other organisations (eg, ACC, the Health and Disability Commissioner). In terms of the challenge of getting information to front-line staff, one participant observed:

‘Well the overarching aim, if I take it back to patient safety is good, I think they struggle to get messages that they are trying to deliver to the front line and what does that mean to the... I call it people who put hands on patients?’

Even where front-line staff were not aware of the Commission itself, many were aware of a number of health quality and safety initiatives occurring in their DHBs, particularly the falls and hand hygiene programmes, and at times, medication safety programmes. In some cases, this was because these initiatives pre-dated the setting up of the Commission. Overall, there was support for the priorities that the Commission has focused on, with positive comments made about the focus on falls, medications, hand hygiene, and adverse

36

events reporting. Several participants noted that falls has been a major issue in New Zealand for many years, and some expressed weariness about the number of attempts there had been at reducing falls. Some participants expressed reservations, however, about the likely accuracy of the hand hygiene audits, and one participant reflected that the programme needed good follow-up as to why audit scores might be low and what to do about it, and that the Commission needed to encourage such follow-up within DHBs if the programme were to be a success. Another suggested that New Zealand really needed to expand the audit programme across more wards if we were to be really sure of making a difference from the hand hygiene programme.

A number of interviewees also commented on having good relationships with the Commission staff. They appreciated the work done by the Commission to keep DHBs ‘in the loop’ and a number commented on how the Commission ‘really listens’, ‘has its ear to the ground’, ‘is prepared to be flexible’, and is easy to contact if help is needed. Staff members of the Commission are well respected. One interviewee noted that, at times, key health sector agencies that fund services can be quite demanding, but the observation that the Commission has worked flexibly was appreciated by a range of participants. Another interviewee noted that the Commission has recognised DHBs are subject to different environmental circumstances and that the Commission has done a good job recognising this. Yet another noted that, at times, some initiatives have not cost a lot and this too was appreciated, given the constant pressure on DHB budgets.

There were a few dissenting views on this, however, with some participants suggesting a greater need for the Commission to actively seek DHB staff ideas and input early on. One interviewee also noted that there may be some risks if the Commission became too big and disconnected.

Those interviewed through the case studies also agreed with the comments from sector leaders that there was a significant amount of activity from the Commission and that this put pressure on the DHBs, many with only a small number of (and limited full-time-equivalent hours for) health quality and safety leaders. Some work had had to be done within extremely limited timeframes (eg, the quality accounts), and it was felt at times that it was not well recognised how long it can take for a DHB to work through initiatives within their DHB, identify any local issues that may be relevant, and determine how best to manage such local issues. Local leadership and governance were also seen to be an important means of engaging local staff. Several interviewees particularly noted that they felt the main burden from the additional work required for the Commission was being felt by the nursing staff. These issues, along with the fact that projects often came with no additional funding to support them, led to some interviewees suggesting that a small number of key priorities would work best for the DHBs, with ideas and input from the DHBs and funding to support their work at a district level. Concerns were also expressed over programmes where a standardised auditing methodology across DHBs meant all DHBs had to do the same amount of work, which was harder for the smaller DHBs.

A number of interviewees commented about the extensive reporting required at times, with concerns raised that too much time spent on ‘ticking boxes’ detracts from the real

37

health quality and safety work that needs to be done. This was made more difficult where information technology systems were not up to scratch, where data had to be collated manually, and where data were of poor quality. Indeed, problems with data quality and information technology systems were mentioned quite frequently as barriers to improving health quality and safety in New Zealand (eg, it was well recognised that information needed to be more readily available in DHB hospitals, but also shared with all those working in the health and disability sector.)

Specific programmes and projects supported by case study participants included the serious events reporting which was felt to be important in improving transparency and enabling people to learn from their mistakes, although in one DHB there was a concern that more could be done to truly learn from the reports. The Atlas of Variation was seen as an important means of providing the ability to compare across similar DHBs, although several interviewees would have liked more flexibility to add in their own information, one interviewee felt that not quite as much attention as it might do, and another reflected on the need for guidelines to reduce variation, with such work falling between the interests of the Commission and the National Health Committee. One interviewee, however, found it difficult to understand the data within the Atlas and compare their service with that in other DHBs, and commented that the United Kingdom has gone a lot further than New Zealand in terms of releasing finely granulated data on performance. They felt New Zealand needed to be braver about this. The highly successful CLAB initiative and Improvement Advisor Development Programme (see below) were also mentioned as excellent initiatives from the Commission.

The quality accounts were well supported, with some participants commenting on the importance of assessing performance in relation to health quality and safety in balancing out the focus that is placed on financial performance. Interviewees commented on the transparency provided by the accounts and the importance of this in being open with their communities about health quality and safety in their DHB. A number of interviewees commented on how they very much enjoyed pulling the quality accounts together and being able to lay out the positive work that was going on in their DHB. Focusing on particular consumer experiences through stories was seen as a great way of discussing key issues relating to health quality and safety. One interviewee commented on how important it is for each DHB to use its own stories, suggesting that these are far more powerful than stories from other DHBs, which are often viewed as working with different populations in different contexts and with different priorities.

Some participants reflected that they would have liked more time initially to work with their communities to identify community priorities in relation to the quality accounts. Another participant felt that the accounts could more clearly lay out what each DHB was doing in terms of quality improvement, track progress in more detail, and set out an action plan for each year, but they were highly supportive of the accounts and the approach the Commission had taken in not being too prescriptive. There was support, at one case study DHB in particular (Waitemata), for the accounts to be on-line as opposed to being in printed form, as the former allowed for more content and regular up-dating at a lower overall cost.

38

Several interviewees, however, who supported the development of the quality accounts, were unclear about what their impact might be. One interviewee compared the low key approach in New Zealand with that of the United Kingdom and felt that a lot more needed to be done with the accounts, including ensuring that the public was well aware of quality of care issues in their DHBs. This interviewee recognised that a more mandatory approach might be needed from the Ministry of Health to achieve this, recognising the importance of the Commission’s work in supporting DHBs and other health care providers in improving quality of care.

A number of participants noted the huge volume of emails they receive from a range of organisations, making it difficult for them to feel that they are top of the many requests and initiatives occurring. A number did specifically mention the Commission newsletters, however, commenting on their value in keeping in them informed.

4.3.3.4 Achieving and Assessing Success

Those in the case studies also reflected on the challenges faced in New Zealand in relation to achieving better health quality and safety. The current culture in DHBs was mentioned; one interviewee noting, when asked if they felt the ‘Open for better care’ Campaign would promote a focus on health quality, suggested that it would, but not as much as it could do:

‘I think it’s more to do with the cultures within the DHBs at the moment. People are just head down, bum up. It’s the financial issues. They see it as nice to have, not essential.’

Some participants reflected on cultures where complacency and acceptance of less-than-ideal quality care was evident at times in DHBs; for example, where falls or central line infections were seen as inevitable. It was this culture that they felt needed to change. Those who had worked in the United Kingdom noted how stringent requirements around health quality and safety had become there in recent years, whereas they felt that in New Zealand there was a view that nothing would happen if there were issues with the quality of care provided.

Individual and team workload issues were another factor that would influence success. Many participants noted how busy they are, making it difficult to focus on new initiatives. One interviewee commented:

‘And… and I’m torn, because I actually believe it, they’ve [the Commission] every right to be demanding. I think they’re very right to be pushing quality. … But the other side of me, which is the reality, which is running round, spinning all the plates, is saying “is it sustainable?”, “are we losing good people?”, “can we do it better?”. … Is it right to focus on all of this stuff all at once? Because it sometimes just feels … that we’re not doing any one of it any justice.’

Others noted that a major barrier to progressing health quality and safety initiatives was the

39

difficulty in getting staff release time from their main roles. This included time for leadership and time from clinicians to provide expertise and knowledge around key initiatives, and the funding to back-fill clinical roles. One interviewee commented:

‘… it’s what blocks engagement at a frontline staff level because they come up with ideas but ideas can’t be progressed so therefore they get a little bit of fatigue around trying to contribute to those sorts of things and get frustrated around well we’re trying to do things but we can’t progress them.’

This same interviewee noted:

‘I think that’s the contradiction, you’re working on quality to improve patient care but if you pull people out of patient care to do that, you’re compromising patient care…’

‘…it’s really hard to push quality and engagement with all of these sorts of things when we’re actually under resourced to do the work, the actual physical work … and I think that’s, until we can address that balance, it’s a really hard sell … when you’ve got a bunch of clinicians that are sitting with vacancies and can’t take leave and are over worked and it’s sort of, the message you’ll get from that level is well if they’ve got all that money to put there [into quality improvement], why can’t they put it into clinical staff to do the work before so it feels, from a front line perspective it feels a little bit that the emphasis is in the wrong place…’

A lack of training and support around projects and how to progress them, particularly for front-line staff, was also seen to be a barrier to progressing a health quality and safety agenda.

Some of those interviewed noted the importance of the work of nurses in relation to health quality and safety, and the difficulties of getting medical doctors on board, as has been found in other research in New Zealand (Gauld and Horsburgh, 2012). Some cultures in DHBs were seen to place significant power in the hands of doctors, to the point where nursing staff and others did not feel comfortable in speaking out. Indeed, one participant suggested that a key issue for the Commission to focus on would be asking DHBs how they engaged with their clinicians on a day-to-day basis on quality initiatives.

Key recommendations included recognising the need for flexibility at DHB level (‘one size doesn’t always fit all’); plans or schedules setting out the objectives for each year; fewer objectives; pick-and-mix toolkits; and additional financial resources and other tools (including those collated from other DHBs) to support the work on health quality and safety. One participant felt that it would be important for the Commission to increasingly focus on patient experience as well as outcomes from treatment, including at department level, but recognised the difficulties involved in getting people on board to achieve this.

In terms of outcomes, it was noted by some participants that success would ‘look like’ successful uptake of Commission programmes and projects at DHB level, as well as manifest change in attitudes and behaviours towards health quality and safety that were

40

sustainable as a result of cultural change in the health and disability sector. Others simply noted that success would be measured by a reduction in harm. It was well recognised that there was a difference in measuring inputs and outputs as a metric of success, eg, uptake and use of the CLAB tool kit versus a measurable reduction in actual CLAB rates, and a number of participants worried that at times improvements in processes (such as increases in falls assessments) may not always lead to reductions in harm (if, for example, no-one was able to respond when a patient rang on a call bell, and the patient then tried to get up and fell). That is, participants were very clear about the importance of measuring both processes and outcomes in order to identify whether or not programmes were having an effect, but they were equally aware of the complexities and difficulties in such measurement.

One interviewee similarly noted the importance of emphasising the outcomes that were desired from each service:

‘… what’s really on our minds is that if you went to a service and you asked them, how do you know you’re doing any good, right? What are your Clinical outcomes? No matter what that service is we’ve got very poor information or data to support what we actually are doing. … if you go down to a service and ask them, how do you know you’re seeing the right patients and they’re have good quality outcomes. … so we’ve charged all the services to come up with two or three matrix that they would like to track and follow that would be of obvious significance to that service.’

This interviewee felt that it would be helpful for the Commission to support such work through providing a good understanding of the range of measures that might be used to assess clinical outcomes, and to widen the range of indicators being measured in New Zealand.

It was generally felt that the work of the Commission has raised the profile and importance of health quality and safety in the DHB hospital sector. Some noted that the Commission’s work had reinvigorated their prior health quality and safety initiatives and that the national attention being paid to health quality and safety as a result of the Commission’s work has made it easier for those in the sector, who see this as an important area, to promote improvements. Others noted that the Commission was resource constrained, but did play an important role as a knowledge repository, which, as a national organisation could draw on a wide range of expertise from around New Zealand.

Many interviewees commented very positively on the achievements of the Commission. One interviewee felt that the Commission had moved away from its early years focusing on providing advice, to a much greater leadership role. They felt that DHBs were increasingly now looking to leadership from the Commission on key topics, and that this was important as individual DHBs would find it hard to not become defensive if they alone were, for example, identifying shortcomings in the care they provided, while the Commission was doing a great job of putting key information into a broader context. They also felt that the Commission had raised its visibility significantly over time. Another commented that the development of the QSMs and their public visibility has made a significant difference to the

41

attention paid to them in their DHB. The fact that the markers were well supported by international evidence was seen as important, but this interviewee also suggested that the markers themselves could be seen as a form of accountability. They were seen as important as the Minister of Health’s Health Targets, reported by each department, through to the senior executive, and to the Board. This had in turn raised the importance of health quality and safety in their organisation.

Several interviewees commented that health, quality, and safety data show improvements over time, although it will be difficult to show that this was due to the Commission’s work alone. Others commented that such improvements come about as a result of the work of many in the sector and suggested more recognition of this was important.

Throughout the interviews, it appears that the work of the Commission is seen to be contributing to improved health quality and safety ethos, focus, and outcomes. The various programmes, projects, and activities according to some participants have sharpened focus, and concerted efforts to engage with areas of concern: it was felt that ultimately these gains will be evident in reduced rates – of medication error, surgical site infection, falls, CLAB rates – and there would likely be health quality and safety benefits resulting from the Commission’s work.

4.3.4 Health Professionals’ Perceptions of Quality Survey

Through The Health Professionals’ Perceptions of Quality Survey, we can assess awareness of the Commission and its work. As can be seen in Figure 1 below, the Commission, as an organisation, has achieved considerable profile amongst clinical staff within a relatively short time. Only 16% of respondents endorsed the ‘have never heard of it’ response option for awareness of the Commission while 27% were quite or very familiar with the Commission’s work. There was also a good level of awareness of the work of the Commission with, again, only 16% of respondents having never heard of the Commission’s work; 22% were quite or very familiar with its work. Note, however, the much higher rates of not having heard of the ‘Open for better care’ Campaign (see below, Section 4.4.3).

Figure 1: How familiar are you with the work of the HQSC?

42

The Health Professionals’ Perceptions of Quality Survey largely assesses issues at a point in time and will need to be repeated to assess changes in culture over time, but one question provides a sense of change in recent years. Figure 2 asks if [your] DHB is doing more in quality improvement than a year ago. Forty-two percent agree or strongly agree with this statement, while only 8% disagree or strongly disagree. It is impossible to know how much of this is due to the Commission and its work, but on the basis of our findings throughout this evaluation, it is likely some proportion of this new attention to quality improvement results from the work of the Commission.

Figure 2: Is your DHB doing more in quality improvement than a year ago?

In addition, several questions in the survey repeated those asked in research on clinical governance undertaken by Gauld and Horsburgh (2012). They found 57% agreement in 2012 that staff work together in a co-ordinated way compared to the 61% in the 2015 survey; that 69% agreed in 2012 it was easy to speak up if there was a perceived problem with care compared to 71% in the 2015 survey; that 70% agreed in 2012 that they involve patients and their families in efforts to improve patient care compared with 77% in the 2015 survey. Assuming there is no bias in who responded to each survey, while all these differences are statistically significant, the first and last differences show an increase that are of practical importance, suggesting small gains are being made in New Zealand with respect to these key quality and safety culture features.

4.4 How is the ‘Open for better care’ Campaign Seen to be Contributing to Improving Health Quality and Safety in New Zealand?

4.4.1 Introduction

In this section, we provide our overview findings exploring how the ‘Open for better care’ Campaign is seen to be contributing to improving health quality and safety in New Zealand. We first provide a brief overview of the Campaign (Section 4.4.2) and then examine perspectives on the Campaign and its contribution to health quality and safety in New

43

Zealand (Section 4.4.3).

4.4.2 A Brief Overview of the Campaign

The ‘Open for better care’ Campaign is a national initiative that aims:

‘To inform and mobilise the New Zealand population to sustainably reduce harm, and ensure safety and quality improvement in health care by preventing harm, avoiding waste and getting better value from resources.’ (Health Quality and Safety Commission, 2013, p.2).

The Campaign was launched in May 2013. The Campaign has focused sequentially on reducing harm through four priority topic areas (programmes), where evidence suggests change in health professional practice can reduce patient harm: falls (May-September 2013); healthcare associated infections (surgical site infections) (October 2013-March 2014); perioperative harm (surgery) (April-September 2014); medication safety (October 2014-March 2015); and back to falls (April-September 2015). A number of these programmes had been priorities for the Commission prior to the development of the Campaign, with the Campaign ‘wrapped around’ the topic areas as a brand and an approach to improvement. The Campaign involves a wide range of collaborative activities with DHBs at national, regional and local levels, alongside the development of best practice resources, guidelines and tools. Success was to be measured in part through this evaluation, as well as through the tracking of trends in quantitative Quality and Safety Markers (QSMs) (see Section 5 below).

4.4.3 A Brief Review of the Existing Evidence of the Effectiveness of Patient Safety Campaigns

Very little international literature exists examining the effectiveness of large-scale, system-level initiatives such as the ‘Open for better care’ Campaign (Ovretveit and Gustafson, 2002; Schouten, Hulscher, Everdingen, Huijsman, and Grol, 2008; Benn, Burnett, Parand, Pinto, Iskander, and Vincent, 2009; Benning, Dixon-Woods, Nwulu, Ghaleb, Dawson, Barber, and Lilford, 2011). Schouten et al. (2008) point to major differences in initiative and evaluation design that make it difficult to draw overall conclusions about the likelihood of major campaigns succeeding in their goals, while Benn et al. (2009) point to the key methodological challenges involved in evaluating complex such complex initiatives, the complexity deriving from the challenging nature of health care delivery and the ways in which health care systems are organised. Trial designs may increase the generalisability of findings but at the expense of understanding how, in the real world, those involved will develop their own local variations of key initiatives and adapt their responses over time (Benn et al., 2009). This points to the need for an evaluation design that can explore differences in context; something that we set out to do at a high-level in this evaluation, through the use of a realist approach.

From the existing literature it is possible, however, to identify some factors that are key to achieving success, including: the commitment of senior management and governing boards; ensuring there is a good understanding of the need for change; engaging clinicians; putting reporting processes in place; developing a good safety culture; and encouraging staff-driven

44

processes that engages with front-line staff (Gollop, Whitby, Buchanan, and Ketley, 2004; Vaughn et al., 2006; Walley, Rayment, and Cooke, 2006). These are all issues that are explored in this evaluation of the ‘Open for better care’ Campaign.

4.4.4 Commission Board and Staff Member Perspectives

The Campaign was seen by Board members and staff of the Commission as a way of accelerating and advancing the Commission’s programmes, giving them ‘zest and energy’. ‘What the Campaign is there to do is push [the programmes], give [them] energy and get people to engage in [them]…’ and it ‘gives people permission to have a real focus on patient safety’. A key measure of success was seen to be the increased visibility of the Commission and its work, and it was felt that the Campaign had helped achieve this. Others pointed to the importance of the ‘Open’ branding in encouraging people to think about how they improve the care delivered to consumers and in being open, transparent and compassionate, and noted that success would be achieved if the emphasis on such transparency and focus on health quality and safety continued long after the Campaign itself had finished. In this way, the Campaign was designed to support long-term cultural change.

It was recognised by Commission Board and staff members that the Campaign had extremely limited resources available to it – nowhere near, for example, major international or national campaigns using national media to support key messages. This meant prioritising the programmes to focus on, and identifying when an issue had been tackled and the Commission could move onto another topic, were key issues for the Commission to consider.

Strong leadership at a number of levels was seen as key to the design and implementation of the Campaign and the associated programmes. This included setting out a strong case for why a particular issue was a problem, to convince DHB Boards and CEOs to become engaged. The Commission Board was seen to play a key role in supporting the Campaign and resolving issues, while a Campaign Advisory Group, clinical leads, and DHB champions all played important roles in the Campaign and the programmes.

A key aspect of the Campaign was the building of key ‘infrastructure’ to support such work, such as regional quality and safety networks which did not always exist or work well at a regional level prior to the roll out of the Campaign. The regional approach was seen as an important way of not duplicating effort and of sharing knowledge and experience. Over time, Commission staff found that a number of the regional networks began to work well independently of the Commission itself. This was seen positively by the Commission staff, who hoped that the networks and their focus on health quality and safety would continue well beyond the Campaign itself. Increasingly, it was being recognised that some regions and DHBs were well ahead in terms of focusing on health quality and safety, with good leadership, operational ability, capacity, and enthusiasm to bring about change.

Working with the Northern region was found to be more of a challenge, due to the pre-existence of their own quality and safety campaign. It was also noted, however, that perhaps it did not matter if the networks themselves did not work as effectively as they might, and it was felt that with individual DHBs involved in key activities, this was less of an

45

issue at times: although things might be slow, it was felt that the DHBs would get onto each key programme in the end. A key debate arising from this experience at the Commission was whether or not to focus its efforts on those regions and DHBs needing more support.

Commission staff worked closely with a range of clinical leaders to design the key messages and new, innovative activities involved with each programme. One interviewee noted that they had learnt it was helpful to give DHBs a ‘smorgasbord of activities to choose from because that way they could do what worked best in their DHB’. It was recognised that providing advice and support on simple actions that could be taken were important at the programme level as part of such campaigns.

Additional funding was seen as a way of potentially strengthening the Campaign, including through more intensive work by Commission staff at a regional level, funding of regional leaders with sufficient time to put into supporting the Campaign, and the use of a collaborative methodology to strengthen engagement throughout DHBs. One interviewee noted, however, that not all agreed it was the Commission’s role to be so hands-on at a regional level, but they felt the need for such intensive work, because ‘where we haven’t had good engagement in the Campaign is when we’ve just put stuff out’.

A number of Commission Board and staff members had ideas about potential improvements to better support the Campaign. It was felt that there was a need for more internal communication within the Commission so that all staff and those engaged with the wide range of Commission programmes and projects understood the Campaign and were able to discuss it in the wider community. It was also felt that perhaps fewer topics might be focused on over a longer period of time (eg, nine months) to give those in the sector more time to build on the initial work required to focus on each topic.

4.4.5 Sector Leaders’ Perspectives

Sector leaders were generally well aware of the Campaign, and were generally supportive of it. As with other groups of participants, there were, however, some dissenting views around the value of the Campaign. Thus, it did not resonate with all sector leaders, and not all felt that it had been a success.

It was recognised by sector leaders that the Campaign was supporting moves towards greater transparency. It was noted that one key strength of the Campaign was that it had used a range of people from across the sector.

Sector leaders also pointed to the balance required between having a nationally consistent Campaign, and one that encouraged local creativity and encouraged the development of local resources and tools. In the northern region, there was initially duplication with and confusion over the ‘Open for better care’ Commission Campaign and the ‘First do no harm’ regional campaign, but by the time of our interviews these issues appeared to have been sorted out to sector leaders’ satisfaction. Sector leaders appreciated the flexibility that the Commission allowed at regional and district levels, recognising that there is ‘no one size fits all’ approach that would work to achieve change. Several noted, however, that a key barrier to implementation was a lack of resources within DHBs.

46

4.4.6 Case Study Participants’ Perspectives

In the case study interviews, there was not always good awareness amongst front-line staff of the Campaign goals and aims, nor of how it related to their everyday work. There was also little demonstrated understanding of how the Campaign was being implemented, although there was awareness of various projects and activities, perhaps more important for those delivering services directly to consumers.

Amongst those aware of the Campaign and its programmes and activities, there were a range of views. On the one hand, some of those interviewed viewed it positively. The project topics chosen were supported, and it was noted that the Campaign had given visibility to the four project topic areas and had driven an improvement focus, including facilitating a collective sharing of results and discussions that otherwise may not have taken place. The funding provided to DHBs to support the Campaign was appreciated. Some appreciated the branding, and the creativity it allowed them to think about new ways to get key messages across (eg, lanyards), but many others were very much more ambivalent. Some appreciated the work that had gone into the Campaign; and some found the targets very useful. In some cases, the Campaign was seen to provide added impetus to initiatives already underway. It was also clear that the Campaign had inspired local responses and buy-in at times, crucial to the implementation of the Campaign in DHBs. It was generally considered that the Campaign had indeed raised the profile of the Commission, its programmes and activities, and promoted an emphasis on health quality and safety in health care.

On the other hand, a number of issues were raised about the Campaign. Some felt it had been put together in a rush, and was not as well conceived or delivered as it might have been. For some, the cost of the Campaign was a concern, given the constraints on resourcing being experienced in all DHBs, and some interviewees suggested they would rather see the funding allocated to bringing people together and networking. (Commission staff noted, however, that key aspects of the Campaign did not actually cost that much in practice, and that part of the reason why so much had been provided through the Campaign was due to the hard work of the Commission communications staff.)

Others pointed to the confusion that arose in their DHBs with the many initiatives underway and they fitted together. For a minority of participants, the concern was over the branding, in that it was not immediately clearly evident what the Campaign was trying to achieve; some suggested that ‘safety’ should be in the branding, or that the northern ‘First do no harm’ was a more effective brand. There were also questions over whether aspects of the Campaign – such as slogans, brands, posters, balloons – would really lead to the changes in culture needed to bring about sustained improvement in care. The Campaign was seen as somewhat less useful in the northern region, where the ‘First do no harm’ initiative had already been put in place.

Time pressures were also raised by some, where sector leaders and front-line staff considered that they had multiple demands placed on them by a range of agencies, which, some thought, could lead to DHB responses being compliance responses rather than being evidence of a change in safety culture. In the view of some participants, engagement with

47

the Campaign, programmes and activities all involved an increase in workload. Some concerns were also raised over the additional reporting required as part of the Campaign.

There was a sense amongst some participants that the Commission had at times been ‘too directive’ and the DHBs would be more receptive if approached about interventions and asked for their views on what these should be. Some concerns were also raised that at times the various programmes and activities were not always compatible with pre-existing initiatives. Consultation to manage such issues was envisaged as a conversation amongst peers and ultimately it was thought that decisions should be based on consensus. This suggests one way in which implementation can be strengthened.

In DHBs, there was no real concern expressed in terms of the programmes (topic areas) chosen by the Commission for the Campaign; rather, concern tended to be focused on having enough time to embed significant and sustainable behavioural change in clinical practice and the embedding of this change into routine practice. There was initial concern within the DHBs that the too rapid shift in focus from one programme of interest to another could result in backsliding in the previous programme and reversion to habitual practice. In response, the Commission has subsequently slowed its pace and this slowing of pace was acknowledged by DHB participants in our most recent case study.

There was also a view that real culture change with respect to quality and safety would be realised if the Campaign engaged the public; that this, in turn, would lead to patients then requesting quality and safe care (eg, ‘Have you washed your hands?’) and that this ultimately would have a powerful influence on the provision of quality and safe care. Front line staff reported that they found practical solutions more appealing than a ‘public relations’ Campaign (eg, the approach taken with CLAB – see below). It was well recognised that achieving reduced harm would result from sustained changes in culture and that one intervention, such as a Campaign, was by itself unlikely to achieve sustainable change in routine practice. It was also well recognised that any impact would take time to be achieved; for example, within the sector there is a widely accepted view that addressing falls has always been a challenge in spite of concerted local efforts to address falls over many years.

One of the key programmes within the Campaign focused on reducing CLAB rates, and this programme has been extremely successful (see Section 5 below). We found widespread satisfaction about the success with respect to CLAB rates. This success is attributed to the efforts of the Commission to ‘shine a light’ on this quality and safety issue and provide resources. Those we interviewed put success down to the evidence being convincing (research, clinical and patient experience) and the CLAB bundle being simple, clear and easy to use. Reduction of rates became evident reasonably quickly and thus there was an affective response to using this intervention and for continuing to apply the bundle and maintain CLAB free days. In addition, all DHB staff involved proudly reported the lowered CLAB rates (eg with notices marking the number of CLAB free days).

4.4.7 Health Professionals’ Perceptions of Quality Survey

The national Health Professionals’ Perceptions of Quality Survey also included questions

48

relating to awareness of the Campaign. In response, 63% had some awareness of the Campaign, with 13% reporting good familiarity and 6% reporting they were very familiar with the Campaign. Thirty-seven percent had never heard of the Campaign. Note that these are higher levels of people having never heard of the Campaign, than of having heard of the Commission itself and its work (see above, Section 4.3).

Questions were also asked with respect to how effective the Campaign has been in (1) Promoting a focus on health quality and safety issues; (2) Influencing workplace practices to improve patient safety and quality of care; (3) Supporting health outcomes in [your] DHB; and (4) Supporting quality and safety outcomes in [your] DHB. A fairly consistent picture emerged (Figure 3). Roughly 30% viewed the Campaign as either effective or very effective, while the majority of respondents (55%–58%) endorsed the ‘neither effective nor ineffective’ option. A consistent 13%-14% reported the Campaign to be ineffective.

Figure 3: How effective has the ‘Open for better care’ Campaign been in:

4.5 Has Quality Improvement Capacity and Capability Improved?

4.5.1 Introduction

In this section, we discuss issues relating to capacity and capability, and provide an overview of our findings relating to the Improvement Advisory Development Programme (IAD Programme), a capability building programme arranged by the Commission and aimed at improving quality improvement skills in DHBs. We first set out perspectives on capacity and capability for quality improvement (Section 4.5.2), followed by a brief overview of the IAD Programme (Section 4.5.3). We then examine perspectives on the IAD Programme (Section 4.5.4), followed by an exploration of its strengths and weaknesses and how can it be improved (Section 4.5.5). Note that issues relating to consumer capacity and capability are considered in the following section (Section 4.6).

49

4.5.2 Perspectives on Capacity and Capability for Quality Improvement in New Zealand

The Commission aims itself to be a leader in quality improvement and it aims also to work closely with regional leaders and local champions to operationalise it programmes and activities. In the first instance, the focus has been on hospital care within 20 DHBs. Concerns were raised in many interviews about the lack of capacity (numbers of people or full-time-equivalents working in quality and safety, the same people being involved in various programmes at the DHBs, and the ability to release people to do more work on quality and safety) and capability (knowledge and skills) available in New Zealand with respect to quality improvement, both within the Commission and at DHB level. Capacity was a key issue, particularly in smaller DHBs. Interviewees suggested that key gaps in capability also needed to be filled, and the Commission emphasis on this through the IAD Programme was seen to be particularly important.

One interviewee reflected the views of a number of those we interviewed when they noted that enhanced capability was about:

‘getting people to be able to assess a problem, assess something that they think needs to fixed in a way that is quantifiable and have the skills to do that properly and then to put in some kind of intervention and then to reassess … ; to … have that … framework.’

Such capability was seen as needing to be a part of the ordinary skill set of those working in the health and disability sector, and hence part of both under-graduate and post-graduate training. Some participants pointed to other countries where there appeared to be much stronger health quality and safety training through the medical, nursing, and allied health core curriculum. Some suggested the need for the Commission to work with Health Workforce New Zealand on such issues.

Through our interviews, we also identified that it was common for people trained in other countries to be working in New Zealand DHBs; as such, it will be important not to solely rely on under-graduate and post-graduate training in New Zealand to deliver a workforce well versed in health quality and safety. Many of those we interviewed were new to their roles; staff turnover in DHBs also suggests the need for on-going programmes in health quality and safety in New Zealand.

Linked with concerns over capacity and capability in quality improvement were more general concerns over gaps in leadership across the sector, including the difficulties in getting people to step up to leadership roles. Commission staff reported that it has been a challenge finding people in all DHBs and across all regions to perform leadership and champion roles with respect to specific programmes. It was noted that some people were reluctant to take a lead because of ‘tall poppy syndrome’ and the fear that they would be seen as promoting themselves as a ‘leader’ rather than the cause.

Those at the larger DHBs thought that they had greater capacity to address quality and

50

safety and considered that the smaller DHBs would most benefit from support and resourcing from the Commission.

The role the Commission has played in providing health quality improvement training was therefore widely acknowledged as a positive contribution towards initiating process and cultural change in the sector. There was broad agreement that this needed to be on-going as retention of trained staff could not be assumed. The IAD Programme was also strongly supported (see below).

Moving out from DHBs and engaging with primary health care, aged residential care, and disability sectors has begun more recently. It was well recognised by those we interviewed that similar issues relating to gaps in leadership more generally, leadership in health quality and safety specifically, and capacity and capability, will arise in these sectors.

With respect to capability building, it was noted that the Commission has started to press for quality improvement to be included in under-graduate and post-graduate health science qualifications.

51

4.5.3 A Brief Overview of the Improvement Advisor Development Programme

The Commission engaged Ko Awatea (based within Counties Manukau DHB) to deliver a nine-month action learning IAD Programme to individuals who were sponsored from within each of the DHBs, supported through scholarships funded by the Commission. The IAD Programme was designed for individuals with an aptitude for turning vision into action and who were looking for an opportunity to develop their quality improvement skills and knowledge of improvement methodology.

The IAD Programme was informed by the Institute for Healthcare Improvement (IHI) model developed in the United States and disseminated widely. The intent was for the Programme to increase capacity in quality improvement, provide expert-level training in processes of transformational change and the skills needed to successfully perform the role of a change agent, and, ultimately, to improve the success rate of improvement initiatives within the organisations of those completing the Programme.

Each of the participants recruited into the Programme was required to bring a real-world change project from their own work environment which they would work on through the use of skills learned during the training period. In the 2013 intake, 16 participants were awarded scholarships. Another intake of participants is scheduled for 2015/2016.

4.5.4 Does the Improvement Advisor Development Programme Result in an Increase in New Zealand Health Organisations’ Capacity for Improvement?

Commission staff commented on the IAD Programme in interviews. Staff noted that the Programme was instituted to develop capability and capacity improvement science in DHBs, with a view to enabling more successful improvement initiatives. At the time of the interviews, most Commission staff were enthusiastic about the value of such a Programme. Some staff, however, did question the resourcing of this Programme and reflected that there may be better ways to invest in sector capability. There was a concern that those that had received the training may leave the sector and or that there would be insufficient diffusion of knowledge after the course and on return to the DHB.

Amongst participants, it was universally reported that participation in the Programme had been a positive and useful experience and that the quality of the Programme content and teaching was excellent. The knowledge and skills learned through the Programme were highly valued by participants. For the participants, the Programme was a clear success.

The project-based component of the IAD Programme (ie, applying the Improvement model to a real-world improvement project) was also seen as valuable, both as an approach to learning and as a means to address an identified area for improvement in the participant’s local area:

‘… a lot of things were baptism by fire because you’d never been exposed to them before, but by working through a project you could see how it worked and could see

52

how it was applicable.’

Another key benefit to emerge from the Programme was the development of networks among the participants, which created both professional support and information sharing networks. In some cases, regional networks were established that have continued beyond the end of the Programme period. This was noted as an important benefit of the Programme by many participants.

It is clear that the participant group came from diverse backgrounds and a range of different roles in their home DHBs. Both the size of the DHB and the type of role in which the participant was employed were seen to have implications for the sustainable use of the improvement methodology once the course was completed. In some cases, participants were already employed in dedicated quality improvement roles; in other cases the quality improvement function was one that was overlaid on an existing full time position as a clinician or manager. In these cases it was seen to be considerably more difficult to continue with quality improvement activities as the demands of their existing role allowed little time to engage with additional quality improvement work. It was stressed that local DHB support was necessary to allow time to be quarantined, to provide opportunity for the implementation of quality improvement initiatives:

‘…it really depends on the DHB embracing this and supporting you to use the skills, and giving you the ability to do it. I think the will is there but …it is very difficult in these times to build cases for anything outside the existing FTE.’

Another theme was that the specifics of the Programme training would impact on participants’ approach to future quality improvement projects and that, in future, they would work differently as a result of participating in the Programme:

‘I don’t think that I’m ever going to do an improvement project again in the way that I used to do in the past; it’s profoundly changed the way I approach improvement projects… it’s kind of put a turbo booster onto my skills and ability.’

In relation to whether the Programme had resulted in a changed way of working from how they had worked previously, one participant commented:

‘Yes, definitely. I mean just having a charter at the beginning which is very clear about what your aim is, how you’re going to measure it, why you are doing it… who is involved, how you are going to report back, the timeframe… it’s just, you know, everyone understands what you’re about and where you are heading.’

There was, however, a suggestion by some that it would be extremely challenging to continue using the methodology in their home DHB without adequate resources locally. Staff from larger DHBs with established specialist quality teams did not share this concern as their quality improvement efforts were supported by local resources.

4.5.5 What are the Strengths and Weaknesses of the Improvement Advisor Development Programme? How can it be Improved?

53

With regard to how the IAD Programme might be improved, there were a variety of pragmatic suggestions, primarily related to software packages and statistics. It was noted that Programme participation presupposed a certain level of familiarity with Microsoft Excel, and that the group members had various levels of experience and competence in the use of this software. A pre-programme course in the use of Excel was suggested as a possible Programme improvement for those with less confidence in using Excel. It was also suggested that having a standardised statistical package that everyone used for the course could be a benefit.

Numerous participants commented that, in terms of benefit to the New Zealand health system, there is an ongoing need to continue to build capacity in quality improvement methodology and that to avoid being a ‘lone voice’ in some DHBs, it would be useful to continue the Programme. This would increase capacity and capability and develop and sustain a cohort that can continue to support ongoing quality improvement in the sector.

Participation in the course was associated with considerably increased workload for some, as they tried to balance existing work commitments with the time requirements for the course. This was true largely for those working in roles other than those directly related to quality improvement. Again, it was noted that local DHB support was required to ensure that there was sufficient ‘ring fenced’ time that could be dedicated to completing the improvement project for the Programme:

‘[there needs to be] a really clear understanding and commitment from the DHB in supporting exactly what it [the programme] entails, as it is very full on.’

When asked to comment on barriers to participating in the Programme, few were mentioned other than, as noted, time commitments to other roles and the cost of the Programme. Some noted that it was an expensive Programme and there were other potential alternative training courses that may be more cost-effective in developing greater capacity (in terms of the number of individuals trained in improvement methodology). This was, however, a minority view.

A number of those interviewed as part of the case studies also reflected on the Programme. Although there was support for the Programme, two key issues arose, which correspond with those noted above. First, there were concerns about the use of a single approach within the Programme (the IHI approach), which was seen as a potential barrier to involvement of some in New Zealand who were already well versed in other approaches (eg, lean thinking, six sigma). Second, was concern over the cost of the Programme – with some participants suggesting that the Programme did not provide great value-for-money and that resources might be better spent on an alternative that enabled more people to be up-skilled for the same cost of the IAD Programme.

4.6 Has the Engagement and Empowerment of Consumers Improved?

4.6.1 Introduction

This section explores issues relating to the engagement and empowerment of consumers through the Commission’s work. Section 4.6.2 describes the Commission’s work in relation

54

to engagement and empowerment of consumers, while Section 4.6.3 explores the perspectives of those interviewed in these evaluations.

55

4.6.2 The Commission’s Work on Engagement and Empowerment of Consumers

In our research, we explored general issues relating to consumer engagement and empowerment. The research, however, took place within the context of a four year plan running from 2012 to 2016, Partners in Care, although we did not explicitly evaluate the plan. Some background information is provided here on the plan, as context for our research.

Partners in Care is aimed at enhancing the engagement and empowerment of consumers, at all levels of the health system - including governance, planning, policy, setting priorities, and highlighting quality issues. A consumer network provides strategic input. The plan involves several work-streams. The first, Co-Design Partners in Care, uses resources from the National Health Service Institute’s Experience-Based Design programme, to support co-design in partnership with consumes. The second, Consumer Engagement, provides resources to support providers in engaging with consumers. The third, Health Literacy, focuses on enhancing health literacy, that is, ‘being able to obtain, understand and use basic health information to navigate health services and make appropriate health decisions’. The Commission stresses that ‘it is the responsibility of the health care professional to present health information in a way that a person can understand’ (Health Quality and Safety Commission, 2015b). The Health Literacy work-stream includes Let’s PLAN, a tool to assist consumers in preparing for primary health care visits, emphasising Preparation for a visit; Listening and sharing; Asking questions; and Noting down what needs to be done next. It also includes a guide to health literacy for health professionals. The fourth work-stream emphasises the development of leadership capability for consumers, with a series of events, provision of resources and establishment of a consumer network key aspects of the work.

Another important stream of work focuses on measuring patient experience of services in New Zealand (Health Quality and Safety Commission, 2015c). The Commission has revamped an earlier existing hospital survey, and that survey is now run quarterly in all DHBs. The survey covers four key domains of patient experience: communication, partnership, co-ordination and physical and emotional needs, and can be used by individual DHBs to track change over time. Across New Zealand, the May 2015 survey shows very good performance on many indicators, and suggest a need to enhance experience particularly through enhanced communication about medication side effects (49% Yes); involvement of family or whānau in discussions about care (54% Yes); having enough information on how to manage a condition after discharge (60% Yes); being involved in discussions on care (67% Yes); a patient’s condition being explained in a way they could understand (71% Yes); being given conflicting information by staff members (73% No); having cultural support available (74% Yes) (where the percentage in the highest category [usually ‘Yes’] was lower than 75%). Relatively low response rates (24% in May 2015) remain an issue, however.

The Commission is following up this work with a new patient experience survey in primary health care. The survey will explore a patient’s experience of the whole health care system using primary care as a window. It focuses on the coordination and integration of care, rather than just the last visit to a GP’s surgery. The survey will be rolled out to interested

56

Primary Health Organisations from late 2015 on.

4.6.3 Commission Board and Staff Members’ Perspectives

At the time of the initial interviews, it was noted that New Zealand was very far behind in this area. It was noted that ‘we don’t have consumers on any of our Boards’, that very few Boards have a consumer advisory council, and that, apart from mental health services and cancer services, services do not actively engage with or employ consumers as part of a team to work with other consumers. It was noted that there was therefore a lot to be done in this area, and the Commission had worked slowly and steadily towards building that work up.

One key issue was finding the ‘right leadership and the right people to drive this in a way that…we wouldn’t get too much resistance’. It was noted that there was quite a lot of resistance in the sector to working with consumers, arising in part from views that it is the professionals who have the knowledge and expertise. Resistance was also seen to be related to concerns about raising the expectations of consumers, but some interviewees reflected that this is not the case and that consumers can help to identify new ways of doing things, and will often choose options that are not as expensive. Some of this resistance was also couched in terms of ‘we are all consumers of health services’, but being a consumer with a medical degree is not the experience of the majority. Commission staff were in agreement that consumer engagement and involvement was pivotal to improving quality in health care; it was suggested that these ‘voices need to be heard’.

A number of staff also stressed the priority area of consumer engagement and the need for the Commission to engage with the consumer perspective. It was acknowledged that consumer involvement was threatening for some in the sector, but that ultimately real improvements to quality and safety would be enabled by greater consumer involvement.

Commission staff stressed how important it was to have consumers involved at the DHB (and ultimately wider) level. The view was that engagement had been successful at the senior management level and amongst nurses and other health professionals ‘on the floor’ who were making a lot of effort to engage with patients around quality and safety issues. However, it was also the view that more work needed to be done with middle managers who are not rewarded for this engagement, nor is it a key performance indicator for them. As one observed:

‘….we think there are issues around middle management, a lot of the leaders at the top are great and the nurses and the people on the ground floor are fantastic. But the middle managers are the ones that you know, they’re not being rewarded for quality improvement, patient safety and consumer engagement type functions or deliverables, so you know, they’ve got enough to deliver’.

It was suggested that linking consumer engagement into accountability documents would be a way to facilitate greater engagement from middle managers. It was also noted that working toward improved patient experience, making services patient-centred,

57

and working toward developing greater health literacy were all issues that the Commission engaged in. Ultimately, they were moving toward a continuous improvement model rather than the current hierarchical model and the former was seen to have the potential to bring about significant savings for the health sector.

4.6.4 Sector Leaders’ Perspectives

A number of sector leaders also raised the importance, in their view, of including a consumer focus and developing stronger consumer involvement. As one observed:

‘I know we’ve got some patient experience thing… but what sort of oversight of ensuring that we are developing consumer involvement in planning services because that is where a lot of the value will come from.’

Others noted the importance of the patient experience survey as a way of engaging with consumers, and focusing attention on to issues that matter for patients.

4.6.5 Case Study Participants’ Perspectives

Interviewees in the case studies expressed concern that a greater level of consumer engagement and empowerment was necessary:

‘I think, some more community based programmes. I mean I think what they do well…I know that [what] hits the mark is the consumer engagement stuff and the emphasis on literacy. So I find it interesting, myself, that they focus a lot on literacy and getting the lingo right…about consumer engagement because I actually think that’s all really, really important: no point in designing a system if its…you know it’s not what the consumer actually wants.’

A number of interviewees suggested, however, that the Commission was currently under-resourced to expand their work to support greater consumer engagement. A number also suggested that having a consumer on the Commission board would be an important signal to the sector, in line with the approach being taken with Partners in Care, and it would provide support for further consumer initiatives.

4.6.6 Health Professionals’ Perceptions of Quality Survey

Overall, it was difficult to assess if the work of the Commission was leading to enhanced consumer empowerment and engagement, which is being assessed through an evaluation of the Partners in Care Plan. Of interest, however, is the finding in The Health Professionals’ Perceptions of Quality Survey that 77% responded that ‘Health professionals in this DHB involve patients and their families in efforts to improve care’; 15% were neutral; while on 8% disagreed or disagreed strongly. How this occurs is not clear from our research, and will be worth following up.

58

5.0 Have the Desired Changes in Safety Practice and Reductions in Harm and Cost Occurred?

5.1 IntroductionIn this section, we explore whether the desired changes in safety practice and reductions of harm and cost have occurred. To reach conclusions on these issues, we first report on findings from stakeholder interviews (Section 5.2); and then provide the results of the analysis of the QSMs to assess changes in practice, harm, and cost (Section 5.3).

5.2. Stakeholder Perspectives In our qualitative interviews, in terms of outcomes, it was noted by some participants that success would ‘look like’ manifest change in attitudes and behaviours towards health quality and safety that were sustainable as a result of cultural change in the health and disability sector. Others simply noted that success would be measured by a reduction in harm. It was also suggested that there was a difference in measuring inputs and outputs as a metric of success eg, uptake and use of the CLAB tool kit versus a measurable reduction in actual CLAB rates.

In all case studies, it appears that the work of the Commission is seen to be contributing to improved health quality and safety outcomes. The various programmes and activities according to participants have sharpened focus, led to concerted efforts to engage with areas of concern and it was felt that ultimately these gains will be evident in reduced rates – of medication error, surgical site infection, falls, CLAB rates, and there would likely be health quality and safety benefits resulting from the Commission’s work.

We found widespread satisfaction about the success with respect to CLAB rates. This success is attributed to the efforts of the Commission to ‘shine a light’ on this quality and safety issue and provide resources to address CLAB rates. Those we interviewed put this success down to the evidence being convincing (research, clinical and patient experience) and the CLAB intervention being a bundle that was simple, clear and easy to use. Reduction of rates became evident reasonably quickly and thus there was an effective response to using this intervention and for continuing to apply the bundle and maintain CLAB fee days. In addition, all DHB staff involved proudly reported the lowered CLAB rates – with notices marking the number of days they had been CLAB free.

5.3 Analyses of Quality and Safety Markers5.3.1 Overview of Trends in the Quality and Safety Markers

The main aim of the QSMs is to measure the success of the Campaign and the programmes that form part of the Campaign, and to determine whether the intended changes in practice and improvements in outcome have occurred. The QSMs are sets of related indicators concentrating on the four areas of harm/programmes covered by the Campaign: Falls; Healthcare associated infections: Central line associated bacteraemia (CLAB), Hand hygiene, Surgical site infection; Perioperative harm; and Medication safety. Given the centrality of the Campaign and these four programmes to the work of the Commission, and that the measures have been developed specifically to assess trends in quality and safety over time,

59

analyses of the QSMs also provide an assessment of the overall impact of the Commission’s work.

The QSMs are a mix of process and outcome measures. The process measures provide information about whether the desired changes in practice have occurred while the outcome measures assess whether harms and cost have been avoided. The process markers chosen are those that measure care processes and should be undertaken nearly all the time, so achievement thresholds have been set at 90 percent in most cases.

Baseline data were available for most of the markers from early 2012 although there is some variability with this. The comparisons in Table 2 relate to an assessment of data from baseline 2012 versus the final quarterly period where data were available for this evaluation (usually June 2015); follow-up data were available for 8 process and 13 outcome markers related to the key areas of the Commission’s work.

Process markers have tracked changes in relation to falls (2), hand hygiene (1), perioperative harm (1), central line-associated bacteraemia (CLABs) (1) and surgical site infections (3). All of the process markers report improvement, with the largest gain related to the increase in the proportion of operations where the correct antibiotic was used for antibiotic prophylaxis (72.7% increase). Longitudinal data related to the outcome markers were available for the key areas: falls (3), hand hygiene (1), perioperative harm (6) and CLABs (3). All of the CLAB outcome markers consistently indicate improvement with decreasing CLABs and CLAB-related costs. Likewise, the markers related to the number of falls resulting in fractured neck of femur, the associated additional bed days and costs associated with these fractured neck of femur events also suggest that improvements have occurred. The findings related to change in the number of DVT/PE events, their associated bed stay days and costs also suggest that progress has occurred. By contrast, the rate of staphylococcus aureus bacteraemia appears to be generally stable. Sepsis events appear to have increased and the additional bed days and costs related to these events have risen.

Medication reconciliation was introduced as a new marker related to medication safety in 2014. It is a structural marker and relates to the national rollout of electronic medication reconciliation. By the end of the 2014, it was observed that four DHBs had implemented electronic reconciliation, while a further five were planning to do so in 2015. Among the four DHBs where electronic medicines reconciliation has been initiated between 50-97% of relevant wards use this service. Early in 2015, two DHBs reported data for the first time in relation to three new medication process markers. Two of the markers relate to the provision of electronic medication reconciliation for people aged 65 years and over (55 years and over for Maori and Pacific people) within a defined time period (24 or 72 hours) while the third specifies the proportion of admissions where the service was provided as part of the discharge summary. Results related to these new process markers suggest that electronic medication reconciliation occurs for between 49-58% of eligible patients within 72 hours and the information is included in between 55-66% patients with their discharge summary.

From Table 2, there is evidence of strong and sustained improvements in most areas addressed by the Campaign over the period 2012-2015. The process marker results are

60

generally more pronounced than those observed with the outcome data and the results from both types of markers are usually consistent. Process markers would be expected to change before outcome markers and it is perhaps not surprising that some outcome markers have not significantly altered over the relatively short period of assessment.

Table 2: Summary of changes in HQSC markers between 2012 and 2015

Baseline Number of DHBs at goal level

Update to June 2015

Relative change Number DHBs at goal level

Improvement*

Falls

Falls resulting in fractured neck of femur

95 81 14.7% Decrease I

Additional bed days related to events

3,385 2,574 24% Decrease I

Estimated cost $2.6 million $2.2 million 15.4% Decrease I

Percentage of older patients assessed for fall risk

76 5 93 22.4% Increase 14 I

Percentage of older people assessed at risk of falling who receive a care plan

80 6 93 12.5% Increase 12 I

Hand hygiene

Percentage of compliant moments 63% 1 80% 27% Increase 13 I

Staph. aureus bacteraemia rate per 1000 bed days

0.13 0.13Sept 2014

0% S

Perioperative harm

Percentage of operations where all 3 parts of the surgical safety checklist used

71% 6 97% 36.6% Increase 17 I

Deep vein thrombosis (DVT)/pulmonary embolism events

baseline 81 cases Decrease I

Bed days DVT/PE baseline 2,793 Decrease I

Cost DVT/PE baseline $2.2 million Decrease I

Sepsis events baseline 225 Increase D

Bed days sepsis baseline 2,311 Increase D

Estimated cost baseline $1.8 million Increase D

61

Central line associated bacteraemia (CLAB)

Percentage of ICU central line insertions compliant with bundle

82% 8 96%Dec 2014

17.19% Increase 15 I

Number of avoided CLABs 76 88Dec 2014

15.8% Increase I

CLABs per 1000 bed days 3.3 0.9Dec 2014

72.7% Decrease I

Cumulative CLABs avoided and costs saved

260 >$5.2millionDec 2014

I

Cost avoided per year $1,527,000 $1,757,000Dec 2014

15.1% Increase I

Surgical site infection

Timely antibiotics 85%Sept 2013

1 96%March 2015

12.9% Increase 4 I

>2gm cefazolin 55%Sept 2013

3 95%March 2015

72.7% Increase 16 I

Skin prep 87%Sept 2013

8 99%March 2015

13.8% Increase 16 I

*Key: I = improvement, S = Stable, D = deterioration

5.3.2 Commentary on Specific Programmes5.3.2.1 Falls

The data reported in Table 2 suggests that the number of falls resulting in a hip fracture has decreased by 15% from baseline (down from 95 to 81) although from the run chart below there is some variability in the rate of hip fractures per 100,000 admissions evident between quarters associated with the relatively small number of cases. This instability related to small numbers ensures that making valid conclusions from comparisons between periods can be difficult and evaluations have sometimes relied instead on the results from process markers (Healey et al., 2014, Weinberg et al., 2011). International evidence suggests that the implementation of a falls programme may result in some reductions in outcome measures over a relatively brief period of assessment (2 years) (DiBardino et al., 2012). A difficulty with the interpretation of the number of falls as a marker is that it does not present the number of events in relation to a denominator. Potential denominators are the number of admissions over the time period or the number of inpatient bed days. The run chart presented in Figure 4 illustrates the quarterly number of falls per 100,000. Evaluating the number of falls resulting in hip fracture as a rate (that is per 100,000 admissions per quarter) rather than as a count illustrates some recent improvement for the period between December 2014 to June 2015 with the median number of monthly falls decreasing from eight to five (Figure 4). A run of six months suggests that that this is a significant reduction in the monthly rate of falls. It will be important to observe the results over the next few

62

months to see if this improvement is sustained.

Figure 4: In-hospital falls with hip fracture per 100,000 admissions June 2010-June 2015

The denominator for this marker could be further refined to specify the number of admissions (or bed days) for elderly people who represent those people most at risk of falling. The HQSC reports that admissions and bed stay numbers for the elderly have increased over the evaluation period and it is possible that this rise could be impacting on the number of hip fracture falls. The presentation of the number of inpatient falls associated with hip fracture on the website as both an absolute number as well as a rate for all admissions and those most at risk may assist with the interpretation of this information.

Consistent with the recent improvement associated with the outcome measure, the data from both of the two process markers suggest strong improvement over the evaluation period. Furthermore for the results related to the process markers the improvements have been sustained over a longer period of the duration of the evaluation. Both process markers have maintained an increase of over 10% improvement from baseline at the start of the evaluation period. It is likely that for the risk assessment marker results have now approached a ceiling where further improvement will be difficult. Consistent evidence exists for the benefits from risk assessment and multifactorial care planning (Bazian Ltd, 2005) so it is reasonable to expect that further outcome benefits could still be achieved over the longer term.

5.3.2.2 Hand Hygiene By June 2015, 80% of observed moments were associated with appropriate hand hygiene an

63

increase of 27% from baseline. Improvement was initially relatively slow which is consistent with the experience observed in other developed countries (Reichardt et al., 2013; Larson, 2013). An assessment of outcome, rates of S aureus bacteraemia per 1,000 bed days, illustrates that there has not been any significant improvement over the evaluation period with the results exhibiting considerable variability (Figure 5).

Figure 5: S aureus bacteraemia per 1,000 bed days by month Jan 2012 – June 2015

Difficulties exist with both increasing staff compliance and performing reliable observational evaluations. Despite the challenges consensus exists about the benefits of a hand hygiene programme (Pittet et al., 2009). Qualitative results illustrate some of the challenges but confirm support for the programme at the three DHB case study sites (see Full Evaluation Report for clinician experience).

5.3.2.3 Perioperative Harm The results from the evaluation indicate that the Surgical Checklist has been successfully implemented in New Zealand hospitals. Where previously just pockets of activity existed (Lee et al., 2012), now widespread adherence to the checklist has been achieved although isolated areas of resistance may remain (Vogts et al., 2011). The marker requiring DHBs to report compliance with all three parts of the surgical checklist has now been retired. A new marker aimed at measuring levels of teams work and communication is planned for implementation during 2015-16. In time, it is possible that the checklist may lead to a more substantial reduction in the rate of post-operative thromboembolic events (Truran et al., 2011) which declined only slightly over the evaluation period. However, it should be noted that bed days associated with the events (and their associated costs) decreased more dramatically.

Post-operative sepsis increased over the evaluation period. This finding is not consistent with those obtained for most of the other markers that generally report improvements over

64

the evaluation period. The underlying reason(s) for the rise in sepsis related events is not clear. The reduction in the number of bed stays suggests that the severity of the septic events may be reducing and it is possible that the complexity of cases have increased thus more cases were at risk of infection over the evaluation period. Some concerns have existed about the accuracy of the indicator when used without the present on admission flag (Cevasco et al., 2011; Cima et al., 2011). The adoption of this flag will likely improve the predictive value of the indicator and may clarify the impact of The Campaign on this outcome. Other possibilities include issues related to minor changes in coding that have occurred over the evaluation period, and increased attention to these events by clinical staff and coders along with improvements in diagnostic techniques that may have ensured that these events are more likely to be identified and/or reported. In any event, a number of interventions exist to potentially further improve performance among New Zealand hospitals in relation to this indicator (Hussey et al., 2013) and it is also notable that post-operative sepsis rates in this country have been among the highest in the OECD (OECD, 2013).

5.3.2.5 CLAB Some of the results associated with the work of the HQSC have been substantial such as those associated with the introduction of the Target CLAB Zero collaborative. Related to this collaborative over 260 CLAB infections have been avoided in two years, at a cost saving of over $5.2 million. The continued success of the CLAB collaborative is remarkable and the results are among the best in the world (Rosenthal et al., 2010). As this level of performance has now been observed for two years, the HQSC has decided that CLAB will no longer be included in the quality and safety marker set.

The qualitative results also highlighted CLAB results and how this success had raised the profile of quality and safety more generally. Further, these results were a source of considerable pride in the case study sites and clearly encouraged those involved and had a more general impact as an illustration of how practice can change and contribute to better outcomes.

5.3.2.6 Surgical Site Infection For primary surgical procedures, an antibiotic should be administered in the hour before the first incision (Najjar and Smink, 2015). The March 2015 data suggests that improvements with this practice have occurred as all 20 DHBs have now achieved 90 percent or higher compliance with this marker. Furthermore, the number of operations where the timing of the antibiotic process was recorded has increased to over 97 percent suggesting that the data may be more accurate and DHBs may be more engaged with the work. A second process marker relates to the prophylactic provision of cefazolin for any hip or knee replacement operations to reduce wound infection (Najjar and Smink, 2015). The use of cefazolin has increased; in the March 2015 quarter, 16 DHBs reached the threshold level (95%) compared with only three during the baseline quarter. Similarly the proportion of operations with skin preparation intended to reduce wound contamination involving either chlorhexidine or povidone iodine in alcohol (Mauger et al., 2014) has increased, with sixteen

65

DHBs achieving the threshold (100%).

It appears that widespread practice change has been achieved across most DHBs for these markers and further improvements relate to increasing the results at one or two outlier DHBs and ensuring that progress is sustained. After some improvement in the early half of 2014 results for an outcome marker related to surgical site infections per 100 hip and knee operations illustrate some volatility in the later part of 2014 and an initial period of improvement in 2015 (Figure 6). Some of the variability likely relates to the effect of small numbers and it may be both timely and useful to extend the marker to consider surgical site infections related to other surgical procedures aside from hip and knee replacement.

Figure 6: Surgical site infections per 100 hip and knee operations March 2013-March 2015

5.3.4 Sustaining Change

Lead DHBs in quality and safety initiatives might be expected to advance Campaign elements from the beginning of the Campaign and potentially with the most vigour. Analysing their experiences also provides information over longer periods of time, and might then be used to comment on the sustainability of change. The analyses show a mix of results. In some cases there is evidence of lead DHBs improving performance by a larger amount and over a longer period of time. In particular, the three Falls Collaborative DHBs in relation to the percentage of older patients assessed for fall risk and two out of three Falls Collaborative DHBs in relation to the percentage of older people assessed at risk of falling who receive a care plan; one of the lead DHBs involved in the perioperative harm surgical checklist; and one of the lead DHBs involved with the surgical site infection programme with respect to the administration of cefazolin as prophylaxis for hip and knee replacements. At times, a lack of greater improvement amongst lead DHBs was related to an initial good performance at baseline.

5.3.5 Limitations of These Analyses

First, in most cases no control group exists to enable the direct comparison of the effects of

66

an element of the Campaign at an intervention DHB in comparison with a control DHB. It is therefore difficult to exclude the influence of other changes over time or the potential for a Hawthorne effect to explain any results that have been obtained.

Second, some before and after assessments have been presented without consideration for the influence of random variation. In some instances events are relatively uncommon and the effects of random variation may be substantial. Results presented as statistical process control charts are preferred where control limits may help describe random variation.

Third, for several markers, results have been approaching 100% such that improvements may have reached a ceiling and further gains are unlikely. Assessments of relative change since baseline are less useful when this has occurred.

Fourth, in many areas (eg falls prevention) the Campaign has included a number of interventions introduced at different times. Evaluating the relative impact of the Campaign in relation to a complex introduction of a range of interventions is difficult. It is likely that the effects of one intervention may blend in to those of another and these cumulative effects may vary at each DHB.

Fifth, data are provided in relation to the results for the QSMs. In the absence of data related to other key indicators it is not possible to ascertain whether improvements in those areas that have been evaluated have occurred in conjunction with improvements in other areas of practice or if there may have been any deterioration in others areas that have not been monitored over the period.

Sixth, the baseline used for the evaluation is the start of one year (2012) and it is not clear whether that year is an appropriate and reliable baseline for all of the comparisons.

Seventh, the evaluation has been conducted over a relatively short time period (2 years) and a longer time frame is needed to explore changes in quality and safety, in particular to assess if improvements are sustained or deteriorations are reversed.

Finally, a number of assumptions are included in many of the markers. The validity and reliability of these assumptions have not been explored and sensitivity analyses have not been conducted. In some instances the markers are based on administrative data that are subject to the limitations of the medical records and coding systems applied to their data.

67

6.0 Has the approach to change the Commission has adopted resulted in sustainable improvement?

In general, it is rather early in the Commission’s life to be certain about whether or not its approach will result in sustainable improvement. This evaluation has taken place within five years of the Commission’s establishment and reaching a conclusion on sustainability after such a short period of time is difficult.

Nevertheless, there are many positive signs. In the qualitative work, we found strong support for the Commission: its independence, its role, its work, and its staff, and no-one questioned the need for a strong emphasis on health quality and safety in New Zealand. The Commission has, within a relatively short period of time, made its presence felt with national and local DHB and health quality and safety leaders; has gained planning and practical, on-the-ground support in its work with DHBs; and is reasonably well recognised with clinical staff around New Zealand.

Many of those we interviewed stressed the importance of having a sustained a focus on health quality and safety if improvements were to be achieved. A number, however, identified the difficulties involved.

Some reflected that key to sustainability would be the leadership of Boards, CEOs, and senior leaders within health organisations, known to be pivotal as a driver of quality and safety (Dorgan et al, 2010; Jha and Epstein, 2010; Goodall, 2011; Gauld and Horsburgh, 2012); greater engagement with health quality and safety issues at all levels, including middle managers; an emphasis on ensuring that health provider workplace cultures support a focus on health quality and safety; and ensuring that key leaders and staff gained and sustained skills in quality improvement. It was noted that this needed an on-going commitment to training, in part resulting from staff turnover.

Although some suggested that a key to sustained improvement would be leadership, supportive cultures, and high levels of engagement amongst sector staff, some also noted the importance of programmes and projects to focus attention in key topic areas, and as a way of emphasising the importance of quality and safety in health care in New Zealand.

Some felt that key to sustaining change would be the use of methodologies proven to be successful in bringing about change, with a collaborative approach, for example, being seen to be key to this.

This evaluation clearly also demonstrated that context can make a difference to how a national health quality and safety commission can achieve positive change; it is clear that resourcing is also a major issue – including the ability to allocate staff to health quality and safety roles, and to release staff to work on health quality and safety issues. This all suggests, however, that it will be important for continued strong signals to be sent to all those in the health sector about the importance of health quality and safety, and for adequate resourcing across the sector. This may be particularly important as the Commission expands its work more widely across the sector.

68

In our quantitative work, the national Health Professionals’ Perceptions of Quality Survey provides some information that is relevant to the issue of sustainability. In terms of a number of safety culture features, there were positive responses, particularly relating to the following (ie where more than 70% of respondents agreed or strongly agreed with key statements):

Health professionals [in this DHB] involve patients and their families in efforts to improve patient care (77% agreed or strongly agreed)

In this clinical area, it is easy to speak up if I perceive a problem with patient care (71%)

People and processes are in place to identify, analyse and act upon all adverse events to prevent future occurrences (71%)

I take an active interest in what happens in this organisation (78%) This organisation has zero tolerance for patient harm anywhere in the organisation

(74%) This organisation informs patients and their families when adverse events occur

(71%) This organisation uses evidence based guidelines in providing patient care (77%).

The areas that may be of concern (ie, where fewer than 50% of respondents agreed or strongly agreed with key statements) relate to deficiencies in relation to:

The DHB’s structure and work processes impede co-ordination across departments and work groups (48% agreed or strongly agreed)

Patient care is well co-ordinated across different parts of the DHB (40%) Patient care is well co-ordinated between the DHB and outside providers (37%) There is little co-ordination of quality improvement efforts across departments and

work groups (33%) This organisation inspires me to do the best they can in my job every day (42%), and I would recommend this organisation as a great place to work (46%).

The areas of most concern (ie where more than 30% of respondents disagreed or strongly disagreed with key statements) relate to deficiencies in relation to:

The DHB provides sufficient resources and support for improving patient care (43% disagreed or strongly disagreed)

I would feel completely comfortable having a family member treated here without my being present to monitor their care (34%)

Patient care today is of the quality I personally think it should be (41%).

In planning future work, the Commission may wish to pay particular attention to the latter two groups of quality and safety culture features.

69

In terms of the analyses of the QSMs, we found evidence of strong and sustained improvements in most areas addressed by the ‘Open for better care’ Campaign, with more pronounced results in the process markers, which would be expected to change before outcome markers do. An analysis of whether lead DHBs, which have had the longest time periods over which to implement change, have made greater gains than other DHBs found a mix of results; however, for most of the QSMs, lead organisations exhibited larger improvements compared with the combined results of all DHBs. This suggests that the longer the timeframe over which DHBs focus on an issue, the greater gains are likely to be made.

70

7.0 Conclusions and Recommendations

7.1 IntroductionThis section summarises the key findings from the evaluations, draws overall conclusions, and makes a series of recommendations about the Commission’s role and work. Section 7.2 focuses on the Role of the Commission; Section 7.3 on the Work of the Commission; Section 7.4 on the ‘Open for better care’ Campaign; Section 7.5 on Capacity and Capability; and Section 7.6 on Consumer Engagement and Empowerment.

7.2. The Role of the CommissionOverall, the role of the Commission appears to be very well understood as a leadership and facilitation role, providing strategic direction, advice, and resource. The Commission’s work is seen as highly important and the independence of the Commission is extremely well supported, enabling a focus on improvement instead of compliance. Moreover, many in the sector have been very impressed with the staff of the Commission and have appreciated their work in the sector.

There are some views that the Commission could take a stronger hand where there is poor compliance and that the importance of health quality and safety warrants a stronger approach at times. It appears, however, that the supportive approach to quality improvement taken by the Commission is very much appreciated across the sector and that this is enabling a positive response from the sector towards improving health quality and safety. If the Commission begins to be seen to be associated with a greater emphasis on compliance, it may lose influence.

Recommendation 1: It is recommended that the Commission’s status as an independent agency be maintained over the longer-term to enable it to build upon the foundations and place within the health system it has established and it is recommended that the Commission continue with its approach of supporting quality and safety improvement and avoid being seen to be focusing on compliance.

In terms of the Commission’s independence, some felt that more could be made of this, and in particular that the Commission should make more of its advocacy role, including influencing the Ministry of Health.

Recommendation 2: It is recommended that the Commission raise its visibility as a commentator on health quality and safety issues; raise its visibility in mainstream media; and adopt more of an advocacy role in the mainstream media such as providing commentary on adverse events and current issues in health care quality and safety.

Many interviewees stressed the importance of leadership and culture if New Zealand is truly going to improve quality and safety, although there was also recognition of the importance of programmes and projects in influencing quality and safety.

Recommendation 3: It is recommended that the Commission continue to focus on improving leadership capability at all levels in the health and disability sector, working

71

alongside other national agencies.

7.3 The Work of the CommissionIn terms of the work of the Commission, it has been noted that the Commission initially worked primarily with DHBs to improve quality and safety in DHB hospitals, and many of those we interviewed – including Board members and staff at the Commission itself – felt it important to also work with health providers in the wider health and disability sector. In more recent times, the Commission has expanded its focus to include the aged residential care, primary health care, and disability support services sub-sectors.

Recommendation 4: It is recommended that the Commission ensure a sector-wide approach is taken in its work.

Many interviewees also commented on the importance of consumer engagement. Many stressed the importance of working with consumers, having a patient-centred approach, and understanding patient experience. The move into engaging with consumers and moving toward a patient-centred approach is seen as positive and it was felt that this would strengthen ongoing implementation and ultimately facilitate sustained change in the sector.

Recommendation 5: It is recommended that the Commission continue to strengthen its focus on consumer engagement.

Strategic planning was stressed by some Commission staff as a very important activity and while programmes and projects provide a focus, it was felt these should not be conducted at the expense of strategic planning. In addition, some staff considered that the role of a Board was to engage in strategic planning rather than operational issues; this was particularly an issue in the early days of the Commission. As a maturing organisation, staff thought strategic planning was indeed becoming increasingly important.

Recommendation 6: It is recommended that the Commission Board continue to increasingly be focused on strategic development and less so on operational matters.

At the DHB level, it was noted that the Commission might bring a greater coherence to its work over time. Some interviewees also suggested that there should be a more focused, evidence-based approach taken to determining the Commission’s work programme. Such comments were made in light of the Commission picking up work from earlier health quality and safety initiatives, although it is not clear exactly which work might be seen to be less important, and there was support for the projects prioritised by the ‘Open for better care’ Campaign.

Recommendation 7: As more analysis is undertaken into health quality and safety, and more experience gained about what works to improve health quality and safety in New Zealand, it is recommended the Commission should revisit how it prioritises its work.

Recommendation 8: It is recommended that the Commission demonstrate a more

72

coherent approach in its work prioritisation – for example, having a clear framework within which its work is clearly identified and linked to key outputs and outcomes.

This prioritisation is particularly important because many of those we interviewed were concerned about the resource implications of expansions in the Commission’s work beyond DHBs and through greater consumer engagement. Concerns have been raised across the interviewees that the Commission does not have the resources to do more, and concerns were also raised by staff that with current resourcing constraints the Commission has had to stop programmes of work before they commence anything new.

Recommendation 9: It is recommended that the Commission ensure its resources are prioritised and are not stretched even more thinly as a result of taking a wider focus across the health and disability sector. This suggests the need to set a clearer small number of priorities and work more closely with the sector in determining these future priorities.

With respect to the early work of the Commission, participants in the DHBs reflected that the planning phases for the various programmes and particularly the timing of when and for how long the programmes would run for were initially problematic. However, following feedback, the Commission addressed timing and has now produced a calendar for the year so that the DHBs can plan in advance.

Recommendation 10: It is recommended that the Commission give further consideration to how long particular programmes or projects run for to ensure those implementing them have time to embed new practice.

The three DHB case studies allowed consideration of the implementation of the Commission’s projects in three different contexts. It was clear that different contexts do impact on individual organisations’ ability to participate in key national initiatives; however, it is not always obvious whether challenging contexts will support the implementation of new initiatives or make implementation more difficult.

Recommendation 11: It is recommended that the Commission continue to pay careful consideration to local contexts and work closely with sector organisations to adapt initiatives to local contexts.

Interviewees also noted the importance of resources (including staff capacity) at the DHB level.

Recommendation 12: It is recommended that the Commission pay attention to the implications of its work for health care providers and balance resource allocation at both Commission and provider levels. Resourcing may be a particular issue in smaller DHBs and in other small health providers and quality improvement capacity issues could focus more on those DHBs and other health providers with greatest need.

There was some variance in the visibility of the work of the Commission. The greatest variance within DHBs was between managerial and lead clinical staff, and front- line

73

workers, the latter who we have found to be largely unaware of the Commission itself or its function or role. This may reflect the fact that many front-line staff say that they ‘focus on their work’ not on ‘what others are doing’; thus, it may well be that quality and safety are being embedded in their daily work – and that no association is being made with any particular organisation or campaign. This simply means that front-line staff consider that they do “not have time to notice” nor do they think that noticing, a campaign, for example, is important; what they do is considered to be important. The engagement of senior staff and lead clinical staff is arguably a demonstrable success for the Commission and a number of participants in all three case study sites thought that true engagement and buy-in from senior management was crucial. A number of participants also suggested that it is important to engage middle management. It was also suggested that buy-in from front- line staff was more likely to happen if a clinical lead signalled some “practical” change that needed to occur for improved health quality and safety. It is worth noting, however, that we found good levels of awareness of the Commission in the Health Professionals’ Perceptions of Quality Survey.

Recommendation 13: It is recommended that further attention be paid to the leadership role of DHB hospital middle managers, gaining increased buy-in from front-line staff, and emphasising practical change.

7.4 The ‘Open for better care’ CampaignThe ‘Open for better care’ Campaign was not always as well-known with front-line staff, although specific programmes and activities were. The Health Professionals’ Perceptions of Quality Survey, however, showed a reasonable level of awareness of the Campaign (albeit there was lower awareness of the Campaign than of the Commission itself).

Amongst those we interviewed who were aware of the Campaign, there were mixed views about it, with some not appreciating the ‘public relations’ elements and concerned about the cost of the Campaign. There was support for the projects focused on during the Campaign. It was also considered that the Campaign had raised the profile of health quality and safety concerns in DHBs. Trends in the QSMs – albeit over a relatively short time period – suggest improving quality and safety in DHBs in some key topic areas supported by the Campaign. This all suggests that the Campaign and associated projects and activities have support and, alongside the Commission’s other work, are making a difference.

Recommendation 14: It is recommended that the Campaign and associated projects and activities should continue, and continue to be assessed both qualitatively and quantitatively for their impacts, particularly to assess whether or not change is sustained over a longer period of time than has been possible in this evaluation.

Recommendation 15: Key issues for the Commission to consider in relation to the continuation of the Campaign include: keeping the costs of the Campaign reasonable; paying attention to the resource implications of the Campaign and associated projects within DHBs and providing additional resources to DHBs to implement key programmes and activities; consulting more with DHBs and ensuring that the local context is well understood where new programmes and activities are being introduced; ensuring that

74

requirements do not become too directive, leading the Commission to being seen as yet a further compliance agency; engaging the public (consumers) more to take on a role in relation to health quality and safety; and focusing on practical interventions, particularly for front line staff.

7.5 Capacity and CapabilityAlmost all of those we interviewed pointed to significant gaps in capacity and capability in relation to quality improvement. Participants in the IAD Programme viewed it very positively and enthusiastically, while recognising the workload that it entailed. The same themes recurred throughout the interviews with the participants in the IAD programme, which were all highly supportive of the Programme. In addition to being useful in terms of personal development, the majority of participants also saw potential benefits from the Programme for their own DHB and for the wider health sector. There were, however, some concerns over the cost of the Programme, how to ensure the benefits from expenditure on capability building influence as many people as possible, and how to ensure capacity is available for those trained in quality improvement to use what they learn once back in their employed roles.

Recommendation 16: It is recommended that the Commission recognise the key gaps in capacity and capability around quality improvement in New Zealand and continue to support training programmes. The Commission should pay attention to obtaining good value-for-money from such programmes, and ensure that training is available across the sector. The Commission could consider, for example, ways of strengthening capability and capacity in quality improvement methodologies within the sector, through a ‘train the trainers’ approach.

Recommendation 17: It is recommended that future IAD programmes are further evaluated sometime after the completion of the programmes to assess their longer term value, in terms of the influence of the project participants worked on; later involvement in quality improvement activities and the resources made available within DHBs to undertake such activities; further skills development; and the facilitating and inhibiting factors in DHBs that may influence these activities.

7.6 Consumer Engagement and EmpowermentThere is strong support throughout the sector for strengthening consumer engagement and empowerment and recognition that work with consumers would support the work that the Commission is doing with health care providers. The Commission’s work in this area, however, was perhaps less visible in many of our interviews and it will be important to carefully evaluate the Partners in Care plan over the next year. The Health Professionals’ Perceptions of Quality Survey showed good levels of engagement with consumers on quality improvement, with 77% agreeing or strongly agreeing that ‘Health professionals [in this DHB] involve patients and their families in efforts to improve patient care’. It will be worth following up on how this occurs in practice.

It has also been suggested that greater engagement in the Commission’s work with middle managers would also support greater engagement in DHBs with consumers.

75

Concerns were, however, raised throughout the evaluations that the Commission would be stretched in terms of financial resources to do more on consumer engagement and empowerment.

76

Recommendation 18: It is recommended that the Commission recognise the potential for engagement and empowerment of consumers to be valuable in generating a positive health quality and safety culture, that it ensure sufficient resources are in place to support such work, and that the ‘Partners in Care’ programme be evaluated to assess its successes, strengths and weaknesses.

Recommendation 19: It is recommended that further consideration be given to the role of middle managers in strengthening engagement with consumers.

7.7 Sustaining ImprovementAs we have noted above, it is rather early in the Commission’s life to draw conclusions over whether the approach it is taking will lead to sustainable improvements in quality and safety in the next five-to-ten years. Nevertheless, there are positive signs, particularly in the support for the Commission’s role, work and staff. Important to sustaining change would be continuing to highlight health quality and safety issues; an emphasis on leadership as well as on programmes and projects; and attention to resourcing.

The results from the Health Professionals’ Perceptions of Quality Survey may be helpful in identifying areas where deficiencies in culture were identified, especially in relation to co-ordination of care within DHBs and between DHBs and outside providers, in terms of quality improvement efforts themselves, and in terms of improving the overall inspiration staff working in DHBs draw from their roles.

Recommendation 20: It is recommended that the Commission consider the findings from the Health Professionals’ Perceptions of Quality Survey and identify priority areas for further work in relation to co-ordination of care, co-ordination of quality improvement efforts, resourcing, and the overall inspiration staff draw from their roles.

Our analyses of the QSMs showed evidence of strong and sustained improvements in most areas addressed by the ‘Open for better care’ Campaign, with more pronounced results in the process markers, which would be expected to change before outcome markers do; the analyses also suggested that the longer the timeframe over which DHBs focus on an issue, the greater gains are likely to be made.

Recommendation 21: It is recommended that the Commission continue to monitor the QSMs to ensure that improvements are sustained over time.

77

8.0 References Bach, T. (2012) The involvement of agencies in policy formulation: Explaining variation in policy autonomy of federal agencies in Germany. Policy and Society 31(3):211-22.

Bazian Ltd (2005). Fall prevention programmes in older people. Evidence-Based Healthcare and Public Health, 9, 343-348.

Benn, J., Burnett, S., Parand, A., Pinto, A., Iskander, S., & Vincent, C. (2009). Studying large-scale programmes to improve patient safety in whole care systems: Challenges for research. Social Science & Medicine, 69(12), 1767-1776. doi: http://dx.doi.org/10.1016/j.socscimed.2009.09.051

Benning, A., Dixon-Woods, M., Nwulu, U., Ghaleb, M., Dawson, J., Barber, N., Lilford, R. (2011). Multiple component patient safety intervention in English hospitals: controlled evaluation of second phase. British Medical Journal, 342, d199. doi:http://dx.doi.org/10.1136/bmj.d199

Bisognano, M., & C. Kenney. (2012) Pursuing the Triple Aim: Seven Innovators Show the Way to Better Care, Better Health and Lower Costs. San Francisco: Jossey-Bass.

Canterbury District Health Board. (2014). Canterbury District Health Board Annual Report 2013/14. http://www.cdhb.health.nz/About-CDHB/corporate-publications/Documents/CDHB%20Annual%20Accounts%20to%2030%20June%202014.pdf

Canterbury District Health Board. (2015.) Our Region. Webpage. Accessed 16 December 2015. http://www.cdhb.health.nz/About-CDHB/Pages/Our-Region.aspx

Cevasco, M., Borzecki, A. M., Chen, Q., Zrelak, P. A., Shin, M., Romano, P. S., Itani, K. M. and Rosen, A. K. (2011). Positive predictive value of the AHRQ Patient Safety Indicator "Postoperative Sepsis": implications for practice and policy. Journal of the American College of Surgeons, 212, 954-61.

Christensen, T., Laegreid, P. (Eds.) (2006) Autonomy and Regulation: Coping with Agencies in the Modern State. Cheltenham: Edward Elgar.

Cima, R. R., Lackore, K. A., Nehring, S. A., Cassivi, S. D., Donohue, J. H., Deschamps, C., Vansuch, M. and Naessens, J. M. (2011). How best to measure surgical quality? Comparison of the Agency for Healthcare Research and Quality Patient Safety Indicators (AHRQ-PSI) and the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) postoperative adverse events at a single institution. Surgery, 150, 943-9.

Cresswell, J. W. (2009). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Thousand Oaks, Sage Publications.

78

DiBardino, D., Cohen, E. R. and Didwania, A. (2012). Meta-analysis: multidisciplinary fall prevention strategies in the acute care inpatient population. Journal of Hospital Medicine (Online), 7, 497-503.

Dorgan, S., Layton, D., Bloom, M., Homkes, R., Sadun, F., and van Reenan, J. (2010). Management in Healthcare: Why Good Practice Really Matters. London, McKinsey and Company/London School of Economics.

Elgie, R. (2006) Why do governments delegate authority to quasi-autonomous agencie? The case of independent administrative authorities in France. Governance 19(2):207-27.

Gauld, R., and Horsburgh, S. (2012). Clinical Governance Assessment Project: Final Report on a National Health Professional Survey and Site Visits to 19 New Zealand DHBs. Dunedin: Centre for Health Systems. http://www.otago.ac.nz/healthsystems/otago065240.pdf

Gauld, R. (2014). Ahead of its time? Reflecting on New Zealand’s Pharmac following its

20th anniversary. Pharmocoeconomics 32:937-42.

Goodall, A. H. (2011). "Physician-leaders and hospital performance: is there an association?" Social Science & Medicine 73(4): 535-539.

Gollop, R., Whitby, E., Buchanan, D., & Ketley, D. (2004). Influencing sceptical staff to become supporters of service improvement: a qualitative study of doctors’ and managers’ views. Quality and Safety in Health Care, 13(2), 108–114.

Gormley, A., Snary, E., Rocks S., and Pollard, S.J.T. (2013) Impact assessment of an independent agency for animal health in England. Human and Ecological Risk Assessment 19(4):1038-48.

Hardy, L.N., Gauld, R., and Holmes, J. (2015). How public hospitals respond to and use a national serious and sentinel events report: A qualitative study in New Zealand. Health Services Management Research, Online First, doi:10.1177/0951484815601875

Healey, F., Lowe, D., Darowski, A., Windsor, J., Treml, J., Byrne, L., Husk, J. and Phipps, J. (2014). Falls prevention in hospitals and mental health units: an extended evaluation of the FallSafe quality improvement project. Age and Ageing, 43, 484-91.

Health Quality and Safety Commission (2013). National Patient Safety Campaign Charter. Wellington: Health Quality and Safety Commission. Accessed 14 September 2015. http://www.open.hqsc.govt.nz/assets/Open-for-better-care/PR-files--images/OPEN-Charter-Sept-2013.pdf

Health Quality and Safety Commission (2014a). Annual Report. For the Period 1 July 2014 to 30 June 2014. Wellington: Health Quality and Safety Commission.

79

Health Quality and Safety Commission (2014b). Our Vision and Values. Health Quality and Safety Commission Website. Accessed 14 September 2015. http://www.hqsc.govt.nz/about-the-commission/our-vision-and-values/

Health Quality and Safety Commission (2014c). Briefing to the Incoming Ministry of Health. Wellington: Health Quality and Safety Commission. http://www.hqsc.govt.nz/assets/General-PR-files-images/Briefing-to-incoming-minister-Oct-2014.pdf

Health Quality and Safety Commission (2015a). Health Quality and Safety Commission Webpage. Mortality Review Committees. Accessed 18 December 2015. http://www.hqsc.govt.nz/our-programmes/mrc/

Health Quality and Safety Commission (2015b). Health Quality and Safety Commission Health Literacy Webpage. Accessed 18 December 2015.http://www.hqsc.govt.nz/our-programmes/health-quality-evaluation/projects/health-quality-and-safety-indicators/patient-experience/

Health Quality and Safety Commission (2015c). Health Quality and Safety Commission Patient Experience Webpage. Accessed 18 December 2015. http://www.hqsc.govt.nz/our-programmes/health-quality-evaluation/projects/health-quality-and-safety-indicators/patient-experience/

Hussey, P. S., Burns, R. M., Weinick, R. M., Mayer, L., Cerese, J. and Farley, D. O. (2013). Using a hospital quality improvement toolkit to improve performance on the AHRQ quality indicators. Joint Commission Journal on Quality & Patient Safety, 39, 177-84.

Hutt Valley District Health Board. (2014). Annual Report 2014. http://www.huttvalleydhb.org.nz/content/7465155e-9df2-4640-83ac-f6ab34c04fa0.html

Hutt Valley District Health Board. (2015a). Annual Plan Overview 2015/16. Lower Hutt: Hutt Valley District Health Board. http://www.huttvalleydhb.org.nz/content/a5ccdbf4-4835-4f8b-ad74-7328718d9b31.cmr

Hutt Valley District Health Board. (2015b). What We Do. Web page. Accessed 16 December 2015.http://www.huttvalleydhb.org.nz/content/f04f083a-8099-43e5-ada8-12430298e992.html

Institute for Healthcare Improvement. (2009). The Triple Aim. Optimizing health, care and cost. Healthcare Executive, 24, 64-6.

James, B.C., & Savitz, L.A. (2011) How Intermountain trimmed health care costs through robust quality improvement efforts. Health Affairs 30(6):1185-91.

Jha, A. K. and A. M. Epstein (2010). "Hospital governance and the quality of care." Health

80

Affairs 29(1): 182-187.

Khemani, S. (2007) Does delegation of fiscal policy to an independent agency make a difference? Evidence from intergovernmental transfers in India. Journal of Development Economics 82(2):464-84.

Koop, C. (2011) Explaining the accountability of independent agencies: the importance of political salience. Journal of Public Policy 31(2):209-34. Washington, DC: National Academies Press.

Larson, E. (2013). Monitoring hand hygiene: meaningless, harmful, or helpful? American Journal of Infection Control, 41, S42-5.

Lee, A., Raniga, S. and Hooper, G. (2012). The Time Out Procedure: have we changed our practice? The New Zealand Medical Journal, 125.

Mauger, B., Marbella, A., Pines, E., Chopra, R., Black, E. R. and Aronson, N. (2014). Implementing quality improvement strategies to reduce healthcare-associated infections: A systematic review. American Journal of Infection Control, 42, S274-83.Ministerial Review Group, (2009). Meeting the Challenge: Enhancing Sustainability and the Patient and Consumer Experience within the Current Legislative Framework for Health and Disability Services in New Zealand. Report of the Ministerial Review Group. ‘The Horn Report’. http://www.beehive.govt.nz/sites/all/filesMRG%20Report%20Meeting%20the%20Challenge.pdf

Ministry of Health (2015). My DHB. Website. Accessed 16 December 2015. http://www.health.govt.nz/new-zealand-health-system/my-dhb

Najjar, P. A. and Smink, D. S. (2015). Prophylactic antibiotics and prevention of surgical site infections. Surgical Clinics of North America, 95, 269-83.

Nunes, R., Rego, G., Bandao, C. (2009) Healthcare regulation as a tool for public accountability. Medicine, Health Care and Philosophy. 12:257-64.

Organisation for Economic Co-operation and Development (2013). Health at a Glance 2013: OECD Indicators, Paris, OECD Publishing.

Ovretveit, J., & Gustafson, D. (2002). Evaluation of quality improvement programmes. Quality and Safety in Health Care, 11(3), 270–275.

Palmer, G. (1987) Unbridled Power: An interpretation of New Zealand’s Constitution and

Government, Second Ed. Auckland: Oxford University Press.

Pawson, R. and Tilley N. (1997). Realistic Evaluation. London, Sage Publications.

81

Pittet, D., Allegranzi, B. and Boyce, J. (2009). World Health Organization World Alliance for Patient Safety First Global Patient Safety Challenge Core Group of Experts: The World Health Organization Guidelines on Hand Hygiene in Health Care and their consensus recommendations. Infect Control Hosp Epidemiol, 30, 611-622.

Reichardt, C., Koniger, D., Bunte-Schonberger, K., Van Der Linden, P., Monch, N., Schwab, F., Behnke, M. & Gastmeier, P. (2013). Three years of national hand hygiene campaign in Germany: what are the key conclusions for clinical practice? Journal of Hospital Infection, 83 Suppl 1, S11-6.

Rosenthal, V., Moreno, C. A., Mehta, A., Leblebicioglu, H., Medeiros, E. A., Raka, L., Hu, B., Jamulitrat, S., Khader, I. A., Novales, M. G. M., Hussain, Y., Hidalgo, R. F., Kanj, S. S., Villacres, A., Navoa-Ng, J. A., Mapp, T., Garcell, H. G., Barkat, A., Suljagic, V., Mitrev, Z., Duenas, L., Cuellar, L. & Hamdi, A. (2010). Central line associated bloodstream infection rates in 166 intensive care units of 54 cities in 23 developing countries: Findings of the International Nosocomial Infection Control Consortium (INICC). International Journal of Infectious Diseases, 14, e255-e256.

Schouten, L. M. T., Hulscher, M. E. J. L., Everdingen, J. J. E. V., Huijsman, R., & Grol, R. P. T. M. (2008). Evidence for the impact of quality improvement collaboratives: systematic review. BMJ, 336(7659), 1491–1494.

State Services Commission. (undated). Crown Entities Guidance. Webpage. Accessed 17 December 2015. http://www.ssc.govt.nz/crown-entities

Truran, P., Critchley, R. J. & Gilliam, A. (2011). Does using the WHO surgical checklist improve compliance to venous thromboembolism prophylaxis guidelines? Surgeon Journal of the Royal Colleges of Surgeons of Edinburgh & Ireland, 9, 309-11.

Vaughn, T., Koepke, M., Kroch, E., Lehrman, W., Sinha, S., & Levey, S. (2006). Engagement of leadership in quality improvement initiatives: Executive Quality Improvement Survey results. Journal of Patient Safety, 2(1), 2–9.

Vogts, N., Hannam, J., Merry, A. & Mitchell, S. (2011). Compliance and quality in administration of a Surgical Safety Checklist in a tertiary New Zealand hospital. The New Zealand Medical Journal, 124.

Wairarapa District Health Board. (2014). Annual Report 2014. http://www.huttvalleydhb.org.nz/content/7465155e-9df2-4640-83ac-f6ab34c04fa0.html

Wairarapa District Health Board. (2015a). Annual Plan Overview 2015/16. http://www.huttvalleydhb.org.nz/content/a5ccdbf4-4835-4f8b-ad74-7328718d9b31.cmr

82

Wairarapa District Health Board (2015b). What We Do. Web page. Accessed 16 December 2015.http://www.huttvalleydhb.org.nz/content/f04f083a-8099-43e5-ada8-12430298e992.html

Waitemata District Health Board. (2014). Waitemata District Health Board Annual Report 2013/14. http://www.waitematadhb.govt.nz/Portals/0/Documents/AboutUs/Reporting/AnnualReports/Annual_Report_2013_2014.pdf

Waitemata District Health Board. (2015). About Waitemata DHB. Website. Accessed 16 December 2015. http://www.waitematadhb.govt.nz/About-Us/About-DHBs

Walley, P., Rayment, J., & Cooke, M. (2006). Clinical systems improvement in NHS hospital trusts and their PCTs: A snapshot of current practice. Institute for Innovation and Improvement and The University of Warwick.

Weinberg, J., Proske, D., Szerszen, A., Lefkovic, K., Cline, C., El-Saregh, S., Jarrett, M. & Weiserbs, K. F. (2011). An inpatient fall prevention initiative in a tertiary care hospital. Joint Commission Journal on Quality & Patient Safety, 37, 317-25.

83

Appendix 1: Case StudiesHutt Valley District Health Board/Wairarapa District Health BoardHutt Valley DHB provides services for 138,370 people in Lower Hutt City (98,200) and Upper Hutt City (40,180). It has a diverse community; seventeen percent of the population identify as Māori (24,060), and 7% identify as Pacific. These Māori and Pacific populations are younger and have higher fertility rates. The local population is projected to increase by 3% (3,740 people) over the next ten years, which is largely driven by an increase in middle-aged and older people. By 2033, it is anticipated that at least one in five people will be older than 65 years and the population over 85 will have doubled; at the same time, the number of children and young people will decline (Hutt Valley District Health Board 2015a). Hospital services are delivered through Hutt Hospital located in Lower Hutt, where our interviews took place. The DHB employs around 2,400 staff (Hutt Valley District Health Board 2015b). It received $362.7 million in funding in 2015/16, with 1,011 full time equivalent employees at September 2015 (Ministry of Health 2015).

Wairarapa DHB provides services for 41,100 people in three local authorities: South Wairarapa (9,520 people), Carterton (8,230 people), and Masterton (23,350 people) over a large geographic area of 5,936 square kilometres. The community is diverse: seventeen percent of our population identify as Māori, and 2% identify as Pacific. Māori and Pacific populations are younger and have higher fertility rates. The local population is projected to increase by 4% (1,780 people) over the next ten years, which is largely driven by an increase in middle- aged and older people. The DHB has one of the highest concentrations of people aged over 75 of any DHB (19% of the population). By 2033, it is anticipated that at least one in five people will be older than 65 years, the population over 85 will double, and at the same time the number of children and young people will decline. (Wairarapa District Health Board, 2015a). Wairarapa DHB is the second smallest DHB out of the 20 DHBs in New Zealand. Approximately half of the population live in the small urban cluster areas, compared with the rest of New Zealand where 83% of the population live in urban areas, and a significant percentage of the Wairarapa population live in sparsely populated rural areas. Masterton is the largest urban cluster and main hospital is based in Masterton, at which we conducted our face-to-face interviews. It received $127.6 million in funding in 2015/16, with 271 full time equivalent employees at September 2015 (Ministry of Health 2015).

At the time of our evaluations, these two DHBs had undergone a merge of some functions that had previously been organised separately firstly for two DHBs (Hutt and Wairarapa) and then for three DHBs (adding in Capital and Coast DHB covering Wellington, Porirua, and the Kapiti Coast). Hutt DHB was at the time of our interviews overseen by a governance board which comprised seven members elected by the community, four members appointed by the Minister of Health (including the Chair), and a Crown Monitor. The Board Chair was shared with neighbouring Capital and Coast DHB, and the Community and Public Health Advisory Committee and Disability Services Advisory Committee shared members from each of the three DHBs. In December 2012, a single Chief Executive was appointed to manage Wairarapa and Hutt Valley DHBs, with a single Executive Leadership Team largely in place in April 2013.

84

At the same time, the Planning and Funding arms of the three DHBs amalgamated to form what is referred to as the 3DHB Service Integration and Development Unit (SIDU). This merge was endorsed by all three DHB Boards who also at the time pledged to adopt a ‘whole of system approach’ – working toward operating as one organisation-one team over multiple sites (Hutt Valley District Health Board Annual Report 2014). Thus, the planning and funding for the three DHBs was conducted by this combined Unit, where they plan, contract, monitor and evaluate health and disability services run by the three DHBs and their contractors (Hutt DHB Annual Report 2014). At the time of our interviews, there was a single general manager for quality and risk across the two DHBs, splitting their time physically across the two sites. An ‘accelerate and collaborate’ quality improvement programme developed in Canterbury was also being developed for the Hutt and Wairarapa DHBs at the time of our interviews.

Our evaluation research occurred during this significant change and this context clearly shaped some of the responses given in the face-to-face interviews conducted with clinical staff in both the Hutt and Wairarapa sites. Since the interviews were conducted at the Wairarapa/Hutt DHBs, each DHB once again has its own Chief Executive, but they are continuing to work together closely, with Capital and Coast DHB.

Canterbury District Health Board

Canterbury DHB is the second largest DHB in the country by both geographical area and population size - serving 529,905 people (Ministry of Health 2015) (12% of the New Zealand population) and covering 26,881 square kilometres and six Territorial Local Authorities. Its Māori (7.2%) and Pacific (2.0%) populations are younger and have higher fertility rates, and its Asian population (6.8%) that is the fastest growing demographic group. An ageing population is a key issue for the DHB: it has the largest total population aged over 75 and will do so for at least the next 15 years. By 2026 one in every five people in Canterbury will be over 65, and the number of people aged over 85 will have doubled. Hospital services are delivered through five main hospitals: Christchurch, Christchurch Women’s, Princess Margaret, Hillmorton, and Burwood, as well as at a number of smaller sites in the wider Canterbury district (eg, Ashburton, Kaikoura). Canterbury DHB provides services on a regional basis to people in other South Island centres where specialist services are not available. It also delivers services to the West Coast DHB, and the CEO of Canterbury DHB is also the CEO of West Coast DHB (Canterbury District Health Board 2015). It received $1,276.5m in funding in 2015/16, with 4,206 full time equivalent employees at September 2015 (Ministry of Health 2015).

Canterbury DHB, unlike the other District Health Board case study sites, has had to respond to a range of serious challenges following major earthquakes in September 2010, February 2011, and June 2011. The earthquakes caused significant damage to many of the DHB’s buildings and assets. More than 200 buildings were damaged, over 12,000 rooms required some level of repair, and a number of temporary infrastructure facilities had to be installed to continue operations, including emergency boilers, water supplies and fire sprinklers (Canterbury District Health Board Annual Report 2014). In

85

addition, the DHB has had to work with a stressed community, drive the relocation of services, and respond to a loss of hospital beds and continue to deliver services to its population throughout this time.

The DHB received an insurance pay out of $320 million in October 2014 and the Government gave a commitment to $650M for Burwood and Christchurch Hospital developments. At Burwood, the purposed designed facilities will support new ways of working with the elderly who require hospital level rehabilitation and care. This facility will open in 2016 (Canterbury District Health Board Annual Report 2014). At Christchurch Hospital, a new Acute Services Building will be constructed and will be operational by 2018. A refurbishment of the Hillmorton site allows for staff to be moved there to provide oral health services and an adult mental health inpatient unit (Canterbury District Health Board Annual Report 2014). In addition, the DHB has faced additional costs related to outsourcing surgery, provision of after-hours care, aged residential care costs, additional community mental health services and other community based costs, all associated with the impact of the earthquakes on the DHB population. These additional costs have been covered by the Ministry of Health to cover the deficit the DHB would have incurred because of these costs (Canterbury District Health Board Annual Report 2014). Clearly managing these developments has involved engaging with quality, safety, and risk in very challenging circumstances.

These challenges were reflected in our interviews, where those interviewed noted that every service would have moved between 2013 and 2016, with half moving three or four times. The DHB has had to manage a significant increase in psychiatric emergency contacts and is working with many schools where there have been increases in suicidal behaviours and behavioural issues, and work with many more inpatients than there are beds.

Waitemata District Health BoardWaitemata District Health Board (DHB) serves the communities of Rodney, North Shore and Waitakere. It is the largest New Zealand DHB by population (582,765 people) (Ministry of Health 2015); 60% European New Zealand, 18% Asian, 10% Māori, and 10% Pacific peoples. Twenty percent of the population are under the age of 15 years, with 35% of these Māori, 13% of the population are over the age of 65 years, with 2% being over 85 years old (Waitemata District Health Board Annual Report 2014). The district is relatively affluent and this district has the highest life expectancy in New Zealand. Waitemata DHB operates North Shore Hospital in Takapuna and Waitakere Hospital in west Auckland. It received $1,335.9 million in funding in 2015/16, with 3,195 full time equivalent employees in September 2015 (Ministry of Health 2015).

86