Professor Jeffrey BraithwaiteDirector, Institute of Health Innovation

Director, Centre for Clinical Governance ResearchUniversity of New South Wales, Australia

EPSO Meeting

Copenhagen, 23 October 2008

Accreditation research

Research partnership

The ARC Linkage Team • Professor Jeffrey Braithwaite• Dr David Greenfield• Dr Marjorie Pawsey• Professor Johanna Westbrook• Professor Bill Runciman • Professor Sally Redman• Professor Robert Gibberd• Conjoint A/Professor Mary

Westbrook• Dr Justine Naylor• Ms Sally Nathan• Ms Maureen Robinson• Ms Judie Lancaster

• Mr Brian Johnston• Dr Desmond Yen• Ms Lena Low• Ms Heather McDonald• Ms Darlene Hennessey• Mrs Margaret Jackson• ACHS staff• ACHS surveyors• ACHS member organisations• Consumer Reference Group

Systematic review

4

Systematic review

5

Systematic reviewResearch category Summary of findings

Professions attitudes to accreditation Mixed

Accreditation promotes change Yes it does

Accreditation has an organisational impact Unclear impact

Accreditation has a financial impact Yes; but cost-benefits unknown

Quality measures eg Cis and accreditation Not clearly related

Program assessment: are accreditation programs credible and worthwhile?

Mixed results

Consumers’ views/patient satisfaction and accreditation

No clear relationships

Public disclosure of accreditation Results support doing this

Does accreditation promote professional development?

Generally yes

Surveyor issues Not much research

The research: four studies

A prospective, multi-method, multi-disciplinary, multi-level, collaborative project for researching health sector accreditation

The research: four studies

Research aim A: To explore the relationshipsbetween accreditation, clinical performance,organisational culture, and consumer participation

The research: four studies

Research aim B:

To examine the influence of surveyors on both the accreditation process and outcome and their own health organisations

The research: four studies

Objectives

Objectives

Aim B: Examine the influence of surveyors by assessing: 1) the reliability of the accreditation process; and 2) the organisational influence of accreditation surveyors.

Aim A: Examine the relationships between accreditation status and processes, and individual and organisational performances, notably clinical performance and organisational culture.

Study 3: Prospective study of intra- and inter-rater reliability of EQuIP surveyors and survey teams.

Study 4: Prospective and retrospective study of organisational influence of accreditation surveyors.

Study 1 Prospective study of the relationships between accreditation and clinical and organisational performances, and consumer participation.

Study 2 Prospective study of health services participating and not participating in accreditation.

1 2 3 4

5 6

Study 1

To determine whether there are relationships between EQuIP performance and

• organisational culture

• organisational climate

• consumer participation

• leadership

• clinical performance

Study 1

12

• This is a three year ARC grant which conducted four major studies of accreditation

• One key study, study 1, examined 19 randomly sampled health care organisations looking at accreditation performance, organisational climate, organisational culture, consumer involvement, leadership, and clinical indicator performance

Study 1

13

• We took six variables• And measured them in the 19

randomly sampled health care organisations, each of which had participated in accreditation in Australia through ACHS EQuIP

• Each of the variables was measured and the data interpreted by a research team blinded from the other research teams

Study 1

• Small: 7• Medium: 6• Large: 6

• Public: 13• Private: 6

• Metropolitan: 8• Regional: 3• Rural: 7• Remote: 1

• Each Australian state and a territory represented

14

Characteristics of the sampled organisations

Study 1

15

Data and procedures• Participant organisations were ranked

1 …19 on the basis of performance:• Accreditation: statistical ranking of

performance based on ACHS EQuIP surveyor reports

• Organisational culture: ~1,000 semi structured interviews with organisational members

Study 1

16

Data and procedures• Organisational climate:

ethnographic, non-participant observations and informal interviews, one week at each site

• Consumer participation: semi-structured interviews

• Leadership: semi-structured interviews

Study 1

17

Data and procedures• Clinical indicators: proportion of

clinical indicators for that organisation that were better than the national average

• In summary: independent measures of the six variables were taken, and on each variable the data were subject to a rank order correlation

Study 1

18

A note on rank order correlation• Correlation coefficients can range from

-1 to +1; they measure the degree of relationship between two variables

• A perfect linear relationship between two variables gives a correlation coefficient of 1

Study 1 Summary:

• Random sample n = 19 organisations

• ACHS: EQuIP outcome + survey research tool

• UNSW research team: cultural, climate, consumer participation, leadership assessment + clinical data analysis

• Research outcome sought: understanding organisational relationships

Study 1

ACulture Climate Cons-

umerLeader-ship

CI

A1.00 0.618*** 0.378 0.215 0.616*** 0.450

p (2-tailed)

0.005 0.110 0.377 0.005 0.080

n 19 19 19 19 19 16

Study 1: Outcomes

Braithwaite et al. Health service accreditation as a predictor of clinical and organisational performance: a blinded, random, stratified study. Submitted to International Journal for Quality in Health Care, 13/10/08.

Braithwaite, J., Greenfield, D., Johnston, B., Scrivens, E. and Shaw, C. (2008) Recent results analysed from large accreditation studies with new research results on accreditation, standards and surveying. Abstract in ISQua 2008. Twenty-fifth International Safety and Quality Conference: Healthcare quality and safety: meeting the next challenges, Copenhagen, Denmark: International Society for Quality in Health Care, October 19-22

Study 1: Outcomes

Nathan, S., McLure, K., Greenfield, D., Pawsey, M. and Braithwaite, J. (2008) “Taking our first steps together: the experiences of a research team and a consumer advisory panel in a national study of health service accreditation.” Involving People in Research Symposium, Perth: 5-6 March.

Braithwaite, et al. (2007) Accreditation: assessing the evidence. Transforming patient safety, accreditation and risk in health care in international context. Abstract in the ISQua 2007. Twenty-fourth International Safety and Quality Conference: Transforming health care in the electronic age, Boston, United States of America, International Society for Quality in Health Care, October 2.

Study 2

A comparison between health services participating and not participating in an accreditation program

Study 2

Details:

• Purposive sample N = 3 organisations

• UNSW research team: cultural assessment + clinical data

• Research outcome sought: understanding accreditation effects

Study 2 Measure\ Acc:

Non-AccZ H C Y F Q

Accreditation performance

17 - 7 - 13 -

Organisational culture

8 22 15 16 5 19

Organisational climate

17 13 3 12 6 10

Leadership 9 13 21 12 11 22

Consumer 10 15 1 16 18 12

Study 2: Outcomes

• Of the twelve indicators, in 9/12, or 75%, the health services participating in accreditation outranked those health services that did not participate in accreditation

Study 2

Other research findings:

One study failed to find any differences between accredited and non-accredited (rehabilitation) programs (Mazmanian et al. 1993)

Another study found improved outcomes when a (trauma) service accredited (Simons et al. 2002)

Study 2

Outcome:

Paper being drafted

Study 3

To assess the intra- and inter-rater reliability of ACHS-EQuIP surveyors and survey teams

Study 3 • Details

Five parts:Scenario exercise: intra-reliability Scenario exercise: inter-reliability (team)Focus groups: ACHS staff + surveyorsFocus group: member organisations, An examination of two teams in practice

• Research outcome sought: reliability of survey teams

Study 3: Outcomes • What promotes reliability of surveyors and

survey teams?– A defined accreditation program where members

participate in developing standards– Trained, experienced surveyors– Trained survey team leaders– Shared expectations of accreditation, standards,

surveying processes– A system of self-governance with checks and

balances along the way– = reliability

Study 3: Outcomes Greenfield, D., Pawsey, M., Naylor, J. and Braithwaite, J. (2008) Are healthcare accreditation surveys reliable?, International Journal of Health Care Quality Assurance, in press (Accepted 14/01/08).

Greenfield, D., Braithwaite, J. and Pawsey, M.P. (2008) Health care accreditation surveyor style typology, International Journal of Health Care Quality Assurance, 21 (5), 435-443.

Greenfield, D., Pawsey, M., Naylor, J. and Braithwaite, J. (2008) Improving the reliability of an accreditation program: using research to educate and to align practice. Poster in ISQua 2008. Twenty-fifth International Safety and Quality Conference: Healthcare quality and safety: meeting the next challenges, Copenhagen, Denmark: International Society for Quality in Health Care, October 19-22

Study 4

To examine the influence of ACHS-EQuIP surveyors on their own health services

Study 4 • Details:

Two PhD studies in progress:

Retrospective study of accreditation outcomes with surveyors

Prospective in-depth case studies of health services with surveyors

• Research outcome sought: understanding surveyor influence

Study 4: Outcomes • Details:

– Becoming a surveyor has benefits for both the surveyor and the surveyor’s host organisation

– Surveyors get exposed to other organisations’ processes, how they do things, what they do well and not so well

– This allows them to take back that learning to their home organisation

– They do this formally and informally

Study 4: Outcomes

PhD expected to be completed 2010 and 2011.

Lancaster, J., Braithwaite, J. and Greenfield, D. Benefits of participating in accreditation surveying. Submitted to International Journal of Health Care Quality Assurance, 9/9/08.

Other research outcomes

Selected papers:

Greenfield, D., Braithwaite, J., Pawsey, M.P., Johnston, B. and Robinson, M. (2009) Distributed leadership to mobilise capacity for accreditation research, Journal of Health Organisation and Management, in press (Accepted 20/6/08).

Greenfield, D. and Braithwaite, J. (2008) Health care accreditation research: a systematic review, International Journal of Quality in Health Care, 20: 172-183.

Other research outcomes

Selected papers:

Greenfield, D. and Braithwaite, J. (2007) Researching accreditation, E-Hospital, Journal of the Association of European Hospital Managers, 9 (5): 18-19.

Braithwaite, J., Westbrook, J.I., Pawsey, M., Greenfield, D., Naylor, J., Iedema, R.A., Runciman, B., Redman, S., Jorm, C., Robinson, M., Nathan, S. and Gibberd, R. (2006) A prospective, multi-method, multi-disciplinary, multi-level, collaborative, social-organisational design for researching health sector accreditation [LP0560737], BMC Health Services Research, 6, 113-123.

Other research outcomes

Selected presentations:

Braithwaite, J., Greenfield, D., Johnston, B., Scrivens, E. and Shaw, C. (2008) Recent results analysed from large accreditation studies with new research results on accreditation, standards and surveying. Abstract in ISQua 2008. Twenty-fifth International Safety and Quality Conference: Healthcare quality and safety: meeting the next challenges, Copenhagen, Denmark: International Society for Quality in Health Care, October 19-22.

Braithwaite, J., Greenfield, D. and Westbrook, M. (2008) Contrasting and converging perspectives on organisational culture and climate. In Conference Proceedings of Culture and Climate: Cracking the Code. The Sixth International Conference on Organisational Behaviour in Health Care [obhc2008], Sydney: Society for the Study of Organising in Health Care, March 26-28.

Other research outcomes

Selected presentations:

Greenfield, D., Travaglia, J., Pawsey, M., Lloyd, J. and Braithwaite, J. (2008) Who is accountable for quality and safety? Poster in Bold aims, bold outcomes: The Sixth Australasian Conference on Safety and Quality in Health Care, Christchurch, New Zealand: Australian Association for Quality in Health Care, September 1-3.

Greenfield, D., Braithwaite, J. and Pawsey, M. (2007) Mobilising academic, industry and government stakeholders in collaborative research partnerships for improved patient safety through accreditation research. Abstract in the Patient Safety Research Conference: Shaping the European Agenda, Porto, Portugal: European Commission Sixth Framework Programme for Research and the Portuguese Ministry of Health, September 24-26.

Summary and conclusion

– 12 papers published or in press– 7 papers in production– 10 presentations to international conferences – 4 further presentations planned– Australian Accreditation Research Network established

Research project, comprising four studies, successfully completed

International interest stimulated

Questions?