What is the Best Way to Track Surgical Complications?

Jacques X. Zhang, B.Sc.Diana Song, MDJulie Bedford, RN, MSNDouglas J. Courtemanche, MD, MS, FRCSC Marija Bucevska, MDJugpal S. Arneja, MD, MBA, FAAP, FACS, FRCSC

Comparing ACS NSQIP versus Traditional M&M Rounds

2

Conflicts of Interests

• Dr. Courtemanche is a director and shareholder with Resilience Software, which made T-Res

• T-Res is used to collect data for M&M Rounds at UBC and the database provided some of the data for the research

What is M&M and NSQIP-P?

3

M&M rounds ACS NSQIP Pediatrics

What is it? Rounds to discuss medical complications and deaths

Discussions are protected under Section 51 of the Evidence Act.1

- Standardized- Tracks post-op complications- Subset of surgical patients - Allows risk-adjustedbenchmarking between hospitals.2-4

Background

1. bclaws.ca2. Khuri SF. 2005;138(5):837-43. 3. Khuri SF, Daley J, Henderson W, et al. 1998;228(October):491-507.4. Ingraham AM, Richards KE, Hall BL, Ko CY. Adv Surg. 2010;44(1):251-267. .

How does it work? Complications are entered

retrospectively into a database.

Some discussed at rounds.

Next Slide…

• NSQIP-P definitions are very strict

4

How NSQIP-P Works

Khuri SF. 2005;138(5):837-43. Khuri SF, Daley J, Henderson W, et al. 1998;228(October):491-507.Ingraham AM, Richards KE, Hall BL, Ko CY. Adv Surg. 2010;44(1):251-267. .

Background

5

M&M rounds ACS NSQIP Pediatrics1-3

Rounds are biweekly or quarterly as organized by the department

Risk-adjusted feedback is provided on a semi-annual basis in the form of a odds ratio for the hospital/department

Used for decades NSQIP P pilot in 2008, BCCH joined in 2011

Surgeon is reviewer Surgical Clinical Reviewer (NSQIP) is reviewer

Looks at all cases Looks at a subset of all cases through rigorous sampling

Cases reviewed quarterly (90 days) Tracks post-op occurrences up to a set 30 days

Complication reported only if surgeon aware Active tracking of patient 30 days post-op

All relevant complications tracked Only NSQIP complications are tracked

Recall and reporting bias Follows strict NSQIP guidelines and definitions

Not risk adjusted Risk-adjusted, allows benchmarking

Low inter-rater reliability High inter-rater reliability4

Low cost and labour High cost ($200,000/yr at BCCH) +labour (2 SCRs/ 2 surgeons champions)

Side-by-Side Comparison

1. Khuri SF. 2005;138(5):837-43. 2. Khuri SF, Daley J, Henderson W, et al. 1998;228(October):491-507.3. Ingraham AM, Richards KE, Hall BL, Ko CY. Adv Surg. 2010;44(1):251-267. .4. Shiloach M, Frencher SK, Steeger JE, et al. J Am Coll Surg. 2010;210(1):6-16

Background

Why this paper?

• Limited studies on:– Pediatric NSQIP in general – Pediatric M&M vs NSQIP– Plastic surgery specific data

• PURPOSE:– To determine the best way to track pediatric

plastic surgeries by comparing complications tracked by NSQIP-P vs traditional M&M rounds, in 2012-2013

6

Purpose

Methods

• For the first 2 full years (2012 and 2013) of NSQIP data: – Extract complications (numerators) for both M&M and

NSQIP– Extract total cases (denominators)

• NSQIP is a subset of M&M – Go through op logs and classify M&M data into “NSQIP

categories” eligible vs ineligible• Stratify data into major and minor complications

7

2012 2013 2012 + 20130

1

2

3

4

5

6

7

8

9

5.48

7.69

6.626.45.79

6.11

Raw Complication Occurrence Rate of NSQIP vs M&M

NSQIPM&M

Com

plic

ation

Rat

e (%

)NSQIP and M&M have similar

rates

8

… However, M&M data includes cases that NSQIP potentially excludes … Need to remove the M&M data that is NSQIP ineligible

NS - Not statistically significant (2-sample Z test)

Results

9

Results

NSQIP vs M&M: adjusted rates

2012 + 20130

1

2

3

4

5

6

7

8

6.62

6.115.71

Apples vs Apples: NSQIP vs M&M (NSQIP eligible)

NSQIPM&M rawM&M (NSQIP eligible)

Com

plic

ation

Rat

e (%

)

10

NS - Not statistically significant (2-sample Z test)

Results

11

~50% of all cases are not tracked by NSQIP!

Half of all cases are not tracked by NSQIP!

n= 648or (51.4% of all cases)

n=613or (48.6% or all cases)

Results

What is the concordance and discordance rate?

2012 + 2013 M&M+ M&M-NSQIP + 13 27 40NSQIP - 24 584 608subtotal 37 611 648

NSQIP ineligible 40 573 61377 1184 1261

12

Concordance rate for M&M Eligible = 13/37 = 35.1% Concordance rate for NSQIP = 13/40 = 32.5% Discordance rate for M&M = 64/77 = 83.1% (MM+, NSQIP-)Discordance rate for NSQIP = 27/40 = 67.5% (MM-, NSQIP+)

6

2

1

2

2

Types of Occurrence: Patients in both NSQIP and TRES M&M

DehiscenceHematomaDehiscence + Infec-tionBleedingComplex (multiple complications)

Results

What is the discordance rate between NSQIP and M&M?

NSQIP inelig

ible

Eligible but n

ot sampled

Sampled but not a

NSQIP complica

tion

Sampled but complica

tion > 30 days

M&M caught s

omething that N

SQIP miss

ed05

1015202530354045

40

615

3 0

2012-2013 MM+ NSQIP- n=64

Num

ber o

f Cas

es

13

Discordance rate for M&M = 64/77 = 83.1%

Results

What is the discordance rate between NSQIP and M&M?

Documented, not reported Unaware Not a plastics complication0

2

4

6

8

10

12

14

16

18

16

9

2

2012-2013 MM- NSQIP+ n=27

Num

ber o

f Cas

es

14

Discordance rate for NSQIP = 27/40 = 67.5%

Results

NSQIP and M&M track different complications

Infection*

Dehiscence

*

Pneumonia*

Urinary*

Transfusio

n*

Other Resp

iratory

Seroma/H

ematoma

Graft / Pro

sthesis

/ Flap Failu

re

Device Failu

re

Delayed Healing

Scars

Death / Morta

lity*

Other0

0.5

1

1.5

2

2.5

3

3.5

Types of Occurrences

NSQIPM&M

Com

plic

ation

Rat

e (%

)

15

* = NSQIP tracked complication

Results

What about the severity of the complication?

• Stratified into major vs. minor complication– Major complication that leads to:

• Death• Readmission• Re-operation

– Minor complication • Anything else

– Results: we find ~50/50 split in both systems• 85% “Major” for matched complications (MM+ NSQIP+)

16

Results

7 major complications (2012-2013) missed by M&M

Match to M&M Documented, not reported

Unaware Not a plastics complication

0

2

4

6

8

10

12

14

16

18

2

97

2

11

7

2

0

2012-2013 MM- NSQIP+ n=27

Major complicationMinor complication

Num

ber o

f Cas

es

17

Results

Best way to track pediatric plastics complications?

A combination of both M&M rounds and NSQIP…

• NSQIP and M&M have similar

occurrence rates but each has their benefits over the other. NSQIP provides the strict rate of morbidity whereas M&M provides the description.

18

Discussion

MOH

Hospital

Department

Division

Surgeons

NSQIP

M&M

Feedback and Purpose:

19

M&M rounds ACS NSQIP Pediatrics1-3

Rounds are biweekly or quarterly as organized by the department

Risk-adjusted feedback is provided on a semi-annual basis in the form of a odds ratio for the hospital/department

Used for decades NSQIP P pilot in 2008, BCCH joined in 2011

Surgeon is reviewer SCR (nurse) is reviewer

Looks at all cases Looks at a subset of all cases through rigorous sampling

Cases reviewed quarterly (90 days) Tracks post-op occurrences up to a set 30 days

Complication reported only if surgeon aware Active tracking of patient 30 days post-op

All relevant complications tracked Only NSQIP complications are tracked

Recall and reporting bias Follows strict NSQIP guidelines and definitions

Not risk adjusted Risk-adjusted, allows benchmarking

Low inter-rater reliability High inter-rater reliability4

Low cost and labour High cost ($200,000/yr at BCCH) +labour (2 SCRs/ 2 surgeons champions)

Side-by-Side Comparison

1. Khuri SF. 2005;138(5):837-43. 2. Khuri SF, Daley J, Henderson W, et al. 1998;228(October):491-507.3. Ingraham AM, Richards KE, Hall BL, Ko CY. Adv Surg. 2010;44(1):251-267. .4. Shiloach M, Frencher SK, Steeger JE, et al. J Am Coll Surg. 2010;210(1):6-16

Conclusions

20

Conclusion

1. NSQIP misses 50% of cases2. Not all complications are tracked3. NSQIP overall rate is accurate compared to

M&M4. NSQIP methods are strictly defined and

rigorous while M&M is subject to recall and reporting bias

We suggest expansion of NSQIP-P to include currently excluded cases and an extension of the NSQIP-P study interval.

21

Limitations

• Limited to the plastic surgery department at a single institution, and only for a 2-year study period

• Rates for the M&M complications might be artificially under-reported due to the errors in M&M data

• Low number of complications combined with a large variety of plastic procedures may increase the variability in rate

Acknowledgements

Dr. ArnejaDr. CourtemancheDr. Diana SongMarija BucevskaJulie BedfordThe plastics team

23

ADDITIONAL SLIDES

What is the Best Way to Track Plastic Surgery Outcomes: Comparing ACS NSQIP vs M&M rounds

24

QA timeline

1894

1907

1911

1935

1940s

1991

1994

1999

2001

2004

2008

2010

2011

Codman develops anesthesia record Codman helps create the ACS End-Result Concept into practice Anesthesia Mortality Committee

Becomes the Anesthesia Study Commission (Prelude to M&Ms) 2000: T-res

NSQIP Peds at BCCH NSQIP Peds to 40 hospitals

NSQIP Peds initiation (4 hospitals) NSQIP becomes open subscription program

VA, ACS Patient Safety in Surgery Study, 14 large non-VA academic hospitals

VA study PSI at 3 academic non-VA surgical dptNSQIP established in all 132 major VA surgical centers

NSQIP inception in 44 VA hospitals in the NVASRS

However, complication rates are still similar

2012 + 20130

1

2

3

4

5

6

7

8

6.62 6.53

NSQIP vs M&M (NSQIP ineligible)

NSQIP M&M (NSQIP ineligible)

Com

plic

ation

Rat

e (%

)

25

NS - Not statistically significant (2-sample Z test)

Results

NSQIP and M&M obtain similar severity of complications

NSQ

IP

M&

M ra

w

M&

M (N

SQIP

elig

ible

)

M&

M (N

SQIP

inel

igib

le)

2012+2013

01234567

3.31 3.33 2.93 3.26

3.31 2.78 2.783.26

Major vs Minor Complications

majorminor

Com

plic

ation

Rat

e (%

)

26

NS - Not statistically significant (2-sample Z test)

Results

Matched cases are mostly major complications

05

1015202530354045

2

20

313

1 011

20

3

2

2 0

2012-2013 MM+ NSQIP- n=64

Major complicationMinor complication

Num

ber o

f Cas

es

27

Results

28

NSQIP and M&M serve different purposes…

• NSQIP-P returns a hospital and department odds ratio (hospital wide view)

• Whereas M&M returns a divisional and surgeon level rate (patient level view)

MOH

Hospital

Department

Division

Surgeons

NSQIP

M&M

Discussion

Theoretically: Reality: Feedback and Purpose:

All cases All cases

M&MNSQIP

Conclusions

• Similar rates, however:1. NSQIP misses 50% of all cases.2. NSQIP morbidity rate is confirmed against M&M,

even for NSQIP ineligible cases, as well as severity of complications. Validation of the program.

3. Differences in definitions and subjectivity in M&M led to low concordance rate, with NSQIP being a more rigorous system.

4. We suggest expansion of ACS-NSQIP to include currently excluded cases and an extension of the ACS-NSQIP study interval.

29