agreement among healthcare professionals in diagnosing case vignette-based surgical site infections...

Agreement among healthcare professionals in diagnosing case vignette-based

surgical site infections in 10 European countries.

G. Birgand 1, D. Lepelletier 2, G. Baron 3, S. Barrett 4, A-C Breier 5, C. Buke 6, L. Markovic-Denic 7, P. Gastmeier 5, J. Kluytmans 8, O. Lyytikainen 9,

E. Sheridan 10, E. Szilagyi 11, E. Tacconelli 12, N. Troillet 13, P. Ravaud 3, JC Lucet 1 and the VARSSI study group

1 Infection control unit, Bichat-Claude Bernard Hospital, Paris, France; 2 Hygiene Department, Nantes University Hospital, France; 3 Centre d’Épidémiologie Clinique, Hôpital Hôtel Dieu, Paris, France; 4 Medical microbiology and infection control, Southend

Hospital, UK; 5 Institute of Hygiene and Environmental Medicine, Charité - University Medicine Berlin, Germany; 6 Department of Infectious Diseases and Clinical Microbiology, Ege University Medical Faculty, Izmir, Turkey; 7 Institute of epidemiology, Faculty of

Medicine, University of Belgrade, Serbia; 8 Medical microbiology and infection control, Amphia medical center Breda, The Netherlands; 9 Department of Infectious Disease Surveillance and Control, National Institute for Health and Welfare (THL),

Helsinki, Finland; 10 Department of Healthcare-Associated Infection and Antimicrobial Resistance, HPA Centre for Infections, London, UK; 11 Department of Epidemiology, Budapest, Hungary; 12 Department of Infectious Diseases, Università Cattolica del

Sacro Cuore, Rome, Italy; 13 Department of infectious diseases, Central Institute, Valais Hospital, Sion, Switzerland.

Disclosure statement

• Financial support: – French Ministry of Health (national grant PREQHOS 0901)

• Conflict of interest:– Pfizer: Travel grant for the ICAAC 2011

G Birgand VARSSI study

ECCMID Berlin 2013

Background• Public reporting of HAI to improve public information and

performance: process indicators or infection rates

• However, SSI rates vary according to several factors – Co-morbidities, contamination class, conditions of the surgical

procedure – Robustness of SSI rates depends on definitions, training, and experience

• Recent studies:– Wilson AP, BMJ 2004: variable interpretations of definitions – French study: disagreement among healthcare professionals in SSI

diagnosis using case-vignettes (Lepelletier , Plos one 2012)– European network: large differences in SSI recognition across countries

(Wilson J, JHI 2007)


Objectives

• To assess agreement in SSI diagnosis among Infection Control Physicians (ICPs) and surgeons involved in SSI surveillance in 10 European countries

- Main objective = intra-country/intra-specialty variability- Secondary objective = intra-specialty/inter-country

variability- Criteria n°1: presence/absence of SSI- Criteria n°2: depth

• To assess the impact of SSI definition availability on diagnosis


ECCMID Berlin 2013

10 participating countriesG Birgand VARSSI study

MethodsElaboration of case vignettes

• 20 case vignettes built from patients hospitalised in 6 surgical units of 3 French university hospitals– Cardiac (n=5), Digestive (n=5), Orthopedic (n=4), ENT

(n=2), Gynecology (n=2) and Neurosurgery (n=2)– Wound discharge and/or infectious signs – Bedside evaluation with collection of biological,

bacteriological and radiological signs

• Case vignettes standardised by the same investigator– Same chronological information – Internet-secured relational database with personal login

and password

ECCMID Berlin 2013


20 vignettes

10 ICPs x 8 vignettes

10 Surgeons x 8 vignettes

20 x 4 = 80 readings per specialty

160 readings per country

X10 countries

1600 readings

1 2 3 4 5 6 7

No SSI SSI

SSI certainly absent

SSI certainly present

Existence of SSI

Intra-class correlation coefficient (ICC) with 95%CI

Methods


ECCMID Berlin 2013

1 2 3 4 5 6 7

1

2

3

No SSI SSI

SSI certainly absent

SSI certainly present

Superficial SSI

Deep or organ/space SSI

Don’t know

Existence of SSI

Intra-class correlation coefficient (ICC) with 95%CI

Depth of SSI

kappa coefficient (kC) with 95%CI

Methods


MethodsInterpretation of agreement

• Intra-correlation coefficient (presence/absence of SSI) – < 0.40 = poor agreement– 0.41-0.71 = good agreement– > 0.71 = excellent agreement

• Kappa coefficient (SSI depth) – < 0.40 = poor agreement– 0.41-0.60 = fair agreement– 0.61-0.80 = good agreement– > 0.81 = excellent agreement

Landis JR. Biometrics 1977

Nunnally JC. Psychometric theory 1994


ECCMID Berlin 2013

Number of participants

10 ICPs + 10 Surgeons



100 ICPs and 86 Surgeons


ResultsAssessment of agreement about SSI diagnosis


Intra-country correlation

N

SSI diagnosis scoring, 7-point Likert scaleIntra-correlation coefficient ICC

ICP

Turkey 80 + 80 0.44

Serbia 80 + 80 0.26

Hungary 80 + 80 0.28

Italy 80 + 72 0.31

Germany 80 + 80 0.65

Finland 80 + 80 0.30

Netherlands 80 + 32 0.45

Switzerland 80 + 80 0.40

France 80 + 72 0.44

UK 80 + 32 0.59

Inter-country 1488 0.41




N


ICP Surgeons

Turkey 80 + 80 0.44 0.23

Serbia 80 + 80 0.26 0.06

Hungary 80 + 80 0.28 0.24

Italy 80 + 72 0.31 0.08

Germany 80 + 80 0.65 0.46

Finland 80 + 80 0.30 0.21

Netherlands 80 + 32 0.45 0.00

Switzerland 80 + 80 0.40 0.08

France 80 + 72 0.44 0.30

UK 80 + 32 0.59 0.04

Inter-country 1488 0.41 0.24


N


ICP Surgeons All

Turkey 80 + 80 0.44 0.23 0.23

Serbia 80 + 80 0.26 0.06 0.20

Hungary 80 + 80 0.28 0.24 0.31

Italy 80 + 72 0.31 0.08 0.20

Germany 80 + 80 0.65 0.46 0.55

Finland 80 + 80 0.30 0.21 0.30

Netherlands 80 + 32 0.45 0.00 0.42

Switzerland 80 + 80 0.40 0.08 0.26

France 80 + 72 0.44 0.30 0.41

UK 80 + 32 0.59 0.04 0.04

Inter-country 1488 0.41 0.24 0.24



No impact of definition on the agreement between participants regarding the presence of SSI

ResultsAssessment of agreement about SSI depth


CountriesSSI depth, 4-point scale (Kappa coefficient)

ICP

Turkey 0.26

Serbia 0.05

Hungary 0.20

Italy 0.23

Germany 0.50

Finland 0.27

Netherlands 0.25

Switzerland 0.34

France 0.44

UK 0.38

Inter-country 0.28




ICP Surgeons

Turkey 0.26 0.25

Serbia 0.05 0.05

Hungary 0.20 0.16

Italy 0.23 0.09

Germany 0.50 0.31

Finland 0.27 0.14

Netherlands 0.25 0.17

Switzerland 0.34 0.14

France 0.44 0.17

UK 0.38 0.19

Inter-country 0.28 0.19


ICP Surgeons All

Turkey 0.26 0.25 0.27

Serbia 0.05 0.05 0.09

Hungary 0.20 0.16 0.19

Italy 0.23 0.09 0.31

Germany 0.50 0.31 0.35

Finland 0.27 0.14 0.25

Netherlands 0.25 0.17 0.25

Switzerland 0.34 0.14 0.24

France 0.44 0.17 0.27

UK 0.38 0.19 0.34

Inter-country 0.28 0.19 -



No impact of definition on the agreement between participants

regarding SSI depth

Discussion• Large disagreement among healthcare professionals in SSI

diagnosis:– varied both between and within specialties

need for a multidisciplinary approach

• One country (Germany) with the best agreement – Regular cross-hospital evaluation of diagnosis accuracy through

surveillance network

• Limitations:– SSI diagnosis made by individuals without teamwork– Scoring via an online database without contact with patients– Selection bias (recruitment by European leaders of participants

presumed to be expert on this topic)


Conclusion

• Original approach to assess the variability of SSI diagnosis– Large panel of ICPs and surgeons in 10 European countries

• SSI rates should be used with caution for benchmarking or public reporting

• SSI surveillance and feedback remain important tools for SSI prevention– Further studies are needed to improve agreement

regarding the SSI diagnosis


ECCMID Berlin 2013

Aknowledgment• The VARSSI study group:

– Turkish, Finnish, Hungarian, German, Italian, Serbian, Swiss, Duch, British and French ICPs and Surgeons participants

• Participating wards:– Pr Nataf, Cardiac surgery, Pr Marmuse, Digestive

surgery; Pr Massin, Orthopedic surgery, Bichat-Claude Bernard University Hospital

– Pr Despins, Cardiac Surgery, University Hospital, Nantes;


ECCMID Berlin 2013

Thank you for your attention

ResultsDistribution of scores

Countries NInfect Contr Phys Surgeons

Median (IQR) Median (IQR)

Turkey 80 + 80 6 (4 - 7) 7 (4 - 7)

Serbia 80 + 80 7 (3 - 7) 6 (3 - 7)

Hungary 80 + 80 6 (4 - 7) 7 (3 - 7)

Italy 80 + 72 6 (4 - 7) 6 (3 - 7)

Germany 80 + 80 7 (3.5 - 7) 6 (3 - 7)

Finland 80 + 80 6 (3 - 7) 6 (3 - 7)

Netherlands 80 + 32 6 (3 - 7) 6 (4 - 7)

Switzerland 80 + 80 7 (4.5 - 7) 7 (6 - 7)

France 80 + 72 6 (3 - 7) 5 (2 - 7)

UK 80 + 32 6 (4 - 7) 6 (4.5 - 7)

Total 1488 6 (4 - 7) 6 (3 - 7)


ResultsDistribution of scores

Countries NInfect Contr Phys Surgeons

Median (IQR) Min-Max Median (IQR) Min-Max

Turkey 80 + 80 6·0 (4·0-7·0) 1·0-7·0 7·0 (4·0-7·0) 1·0-7·0

Serbia 80 + 80 7·0 (3·0-7·0) 1·0-7·0 6·0 (3·0-7·0) 1·0-7·0

Hungary 80 + 80 6·0 (4·0-7·0) 1·0-7·0 7·0 (3·0-7·0) 1·0-7·0

Italy 80 + 72 6·0 (4·0-7·0) 1·0-7·0 6·0 (3·0-7·0) 1·0-7·0

Germany 80 + 80 7·0 (3·5-7·0) 1·0-7·0 6·0 (3·0-7·0) 1·0-7·0

Finland 80 + 80 6·0 (3·0-7·0) 1·0-7·0 6·0 (3·0-7·0) 1·0-7·0

Netherlands 80 + 32 6·0 (3·0-7·0) 1·0-7·0 6·0 (4·0-7·0) 2·0-7·0

Switzerland 80 + 80 7·0 (4·5-7·0) 1·0-7·0 7·0 (6·0-7·0) 1·0-7·0

France 80 + 72 6.0 (3.0-7.0) 1·0-7·0 5.0 (2.0-7.0) 1·0-7·0

UK 80 + 32 6·0 (4·0-7·0) 1·0-7·0 6·0 (4·5-7·0) 1·0-7·0

Total 1488 6·0 (4·0-7·0) 1·0-7·0 6·0 (3·0-7·0) 1·0-7·0



N


ICP Spontaneous

scoring

Surgeons Spontaneous

scoring

AllSpontaneous

scoring

Turkey 80 + 80 0·44 (0·22-0·68) 0·23 (0·03-0·50) 0·23 (0·03-0·50)

Serbia 80 + 80 0·26 (0·05-0·53) 0·06 (0·00-0·35) 0·20 (0·04-0·41)

Hungary 80 + 80 0·28 (0·07-0·54) 0·24 (0·04-0·51) 0·31 (0·16-0·54)

Italy 80 + 72 0·31 (0·10-0·57) 0·08 (0·00-0·36) 0·20 (0·07-0·42)

Germany 80 + 80 0·65 (0·45-0·82) 0·46 (0·23-0·69) 0·55 (0·37-0·74)

Finland 80 + 80 0·30 (0·09-0·57) 0·21 (0·01-0·48) 0·30 (0·15-0·52)

Netherlands 80 + 32 0·45 (0·23-0·69) 0·00 (0·00-0·35) 0·42 (0·23-0·65)

Switzerland 80 + 80 0·40 (0·18-0·65) 0·08 (0·00-0·34) 0·26 (0·12-0·48)

France 80 + 72 0·44 (0·22-0·68) 0·30 (0·07-0·57) 0·41 (0·24-0·63)

UK 80 + 32 0·59 (0·38-0·80) 0·04 (0·00-0·62) 0·04 (0·00-0·62)

Inter-country 1488 0·41 (0·28-0·61) 0·24 (0·14-0·42) 0·24 (0·14-0·42)



Countries

SSI depth, 4-point scale(Kappa coefficient)

Not scored ICP Not

scored Surgeons Not scored All

Turkey 19 0·26 (0·21-0·31) 17 0·25 (0·20-0·30) 36 0·27 (0·24-0·30)

Serbia 21 0·05 (0·00-0·10) 25 0·05 (0·00-0·10) 46 0·09 (0·07-0·11)

Hungary 18 0·20 (0·15-0·25) 22 0·16 (0·10-0·22) 40 0·19 (0·16-0·22)

Italy 21 0·23 (0·17-0·29) 23 0·09 (0·01-0·17) 44 0·31 (0·18-0·34)

Germany 20 0·50 (0·45-0·55) 21 0·31 (0·26-0·36) 41 0·35 (0·32-0·38)

Finland 24 0·27 (0·22-0·32) 23 0·14 (0·09-0·19) 47 0·25 (0·22-0·28)

Netherlands 23 0·25 (0·20-0·30) 7 0·17 (0·00-0·34) 30 0·25 (0·18-0·32)

Switzerland 11 0·34 (0·28-0·40) 18 0·14 (0·08-0·20) 29 0·24 (0·21-0·27)

France 21 0·44 (0·39-0·49) 27 0·17 (0·09-0·25) 48 0·27 (0·23-0·31)

UK 20 0·38 (0·32-0·44) 6 0·19 (-0·02-0·40) 26 0·34 (0·25-0·42)

Inter-country 198 0·28 (0·27-0·29) 189 0·19 (0·17-0·21) - -



agreement among healthcare professionals in diagnosing case vignette-based surgical site infections...

Documents

department of healthcare

ssi diagnosis

ssi surveillance

hygiene department

department of epidemiology

ssi recognition

medical microbiology

associated infection