two concerns for ivf laboratorians, david mcculloh phd
TRANSCRIPT
Two Numeric Concerns for IVF Laboratorians:
David H. McCulloh, Ph.D., H.C.L.D., C.C.
New York University Langone Medical Center
NYU Fertility Center
1) Staffing the lab, and 2) Statistical Significance versus Clinical Relevance
Disclosures:
Assistant Laboratory Director -
Andrology Laboratory Director -
Director of Clinical Science -
President & Principal Scientist -
Learning Objectives:
• The types of procedures that we perform in the ART laboratory are changing – The complexity of procedures impacts personnel
requirements
• There are two components to staffing the ART lab – Fixed component – Variable component
• Adjustments to the calculator for part-time programs • There are ways to estimate the clinical relevance of
statistically significant observations
Volume of Procedures each Year
0
500
1000
1500
2000
2500
2000 2005 2013 2014
Year
Tota
l Nu
mb
er
of
Pat
ien
t-P
roce
du
res
Distribution of Procedures
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2000 2005 2013 2014
Year
Frac
tio
n o
f P
atie
nt-
Pro
ced
ure
s (%
)
Thaws
Banking
IVF/Bx
IVF/ET
It isn’t the same anymore…
• NYUFC is getting busier
• NYUFC has changed the types of procedures that are performed
– More diverse
– More complex
• Now we treat individual oocytes not just patients
• Must track each oocyte/embryo for PGS/PGD
– More witnessing
Old Rule of Thumb:
The IVF lab needs one embryologist for every 100 IVFs
IVF Traditional
IVF Contemporary
IVF /PGS
Procedure Time
Witness Complexity Procedure
Time Witness Complexity
Procedure Time
Witness Complexity
Preparation All 30 0 6 60 0 12 80 0 16
Oocyte Retrieval 60 10 12 60 10 12 60 10 12
Sperm Preparation 60 10 6 60 10 6 60 10 6
Insemination/ICSI 40 20 12 40 20 12 40 20 12
Fertilization Check 40 10 6 40 10 6 40 20 6
Day 2 Check 20 0 2 20 0 2 20 0 2
Day 3 Check 20 0 2 20 0 2 20 0 2
Day 3 Transfer 40 10 16 40 10 16 0 0 0
Day 3 Cryo 40 10 8 0 0 0 0 0 0
Assisted Hatching 20 0 4 20 0 4 60 0 12
Extended Culture 0 0 0 40 10 4 40 10 4
Day5 Check 0 0 0 20 0 2 20 0 2
Day 5 Transfer 0 0 0 0 0 0 0 0 0
Day 5 Biopsy 0 0 0 0 0 0 80 40 32
Day 5 Cryo 0 0 0 40 20 8 80 40 16
Day 6 Check 0 0 0 20 0 2 20 0 2
Day 6 Biopsy 0 0 0 0 0 0 80 40 32
Day 6 Cryo 0 0 0 40 20 8 80 40 16
Document Management 30 0 0 30 0 0 30 0 0
Number of hours 6.67 1.17 1.23 9.17 1.83 1.60 13.50 3.83 2.87
Total Time (hrs) 9.07
12.60
20.20
Complexity has increased!
• More person-hours per case
– Witnessing
– Tracking individual oocytes/embryos
– Prolonged culture duration
– Multiple cryopreservation events per patient
– Biopsy
Table 3. Activities in the IVF laboratory including components and estimated complexity level
Activity Component 1 Component 2 Component 3 Component 4 Component 5 Component 6 Complexity
Case Set-up Day -1 Record Review Need Assessment
Dish Labeling Media Preparation
Dish Preparation
2
Oocyte Retrieval; Day 0 12 Oocytes
Lab Preparations
Follicular Fluid Search
Cumulus Dissection/Wash
Oocyte Culture Witnessing
2
Oocyte Cryo; 10 Oocytes
Record Review/Pt ID
Media/Dish Preparation
Cryo Container Preparation
Denuding/ Evaluating Eggs
Vitrification Witnessing 4
Oocyte Thaw; 10 Oocytes
Record Review/Pt ID
Media Preparation
Dish Preparation
Oocyte Warming
Oocyte Culture Witnessing 4
Surgical Sperm Retrieval
Lab Preparations
OR Procedures
Tissue/Sample Processing
Tissue/Sample Cryo
Witnessing
2
Sperm Preparation; Simple
Semen Analysis
Gradient Preparation
Sample Preparation
Analysis Witnessing 1
Sperm Preparation; Complex
Semen Analysis
Special Treatments
Sample Preparation
Analysis Witnessing 1
Insemination; Standard
Record Review/Pt ID
Oocyte Preparation
Insemination Drop Prep
Insemination Witnessing 2
ICSI; Simple; 12 Oocytes
Record Review/Pt ID
Oocyte Preparation
Dish Preparation
Microinjection Witnessing 4
ICSI; Complex; 12 Oocytes
Record Review/Pt ID
Oocyte Preparation
Dish Preparation
Sperm Search Microinjection Witnessing 4
PICSI Record Review/Pt ID
Oocyte Preparation
Dish Preparation
Sperm Search Microinjection Witnessing 4
Insemination; Split ICSI/Standard
Record Review/Pt ID
Oocyte Preparation
Insemination Dish Preparation
Microinjection Witnessing 4
Fertilization Check; Standard
Oocyte denuding
PN Assessment
Zygote Culture Witnessing
2
Fertilization Check; ICSI
Pronucleus Assessment
Zygote Culture
Witnessing
1
Staffing: Two components
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Number of IVFs per Year
Nu
mb
er
of
pe
rso
nn
el r
eq
uir
ed
Based on Procedure Performance
Fixed number to staff the lab
Greater of the two methods
ASRM Guidelines
Calculation of Staffing Requirements for an Embryology Laboratory
Minimum Personnel Required (MPR)
MPR(procedures) =
# IVF’s (without PGS/PGD) X 0.00672
+ # IVF’s (with PGS/PGD) X 0.01077
+ # FET’s X 0.00192
+ # Oocyte Freezes X 0.00299
+ # Oocyte Thaws X 0.00686
+ Days of Operation (Days of QC activity) X {(# Incubators QC’d X 0.0000347) + 0.000267}
MPR(2 per day) =
Days of Procedures X 0.008
MPR = Maximum(MPR(procedures), MPR(2 per day))
Load the Staffing Program, Dave….
But there is a problem….
• The Calculator really works only for full time labs
• We need a correction for part-time facilities
• Must divide the Total Person Years required
– By the staffing hours of operation per year
(hours of operation in one year)
= Results X (hours/person-year) / (hours of operation)
Clinical Relevance vs. Statistical Significance
• Problem
– Statistical significance is generally based on determinining if there is significance between mean values
• The best estimate of the mean value is the mean with a vaariability of +/- Standard Error of the Mean
• The best estimate of a single variate is the mean with a variability of +/- Standard Deviation
– Usually, a patient will provide an N of 1 to the outcome
List of Examples: Are these Clinically Relevant Effects?
• hCG on day 28 (14 days after retrieval) predicts outcome (beating fetal heart)
• Premature Births comparing Day 3 with Day 5 Transfer
• Incubator parameters (QC params) predict pregnancy loss (live birth vs. preg. loss)
An Analytical Trap
• Pregnancy Test (hCG) values predict outcome?
– Comparison of predictor values between two groups segregated by outcome
– Instead, Receiver Operator Characteristic curve should be used
[hCG] on Day 28 (619 delivered and 1299 did not deliver)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 10 100 1000
Cu
mu
lati
ve I
nci
de
nce
[hCG] on Day 28 (mIU/mL)
Delivered
Did Not Deliver
1347 1040
20
40
60
80
100
120
140
160
180
200
Yes No
hC
G (
mIU
/m
L)
on
da
y 2
8
Fetal Heart Beat Observed with Ultrasound?
hCG is Significantly Different for Patients with/without FHB (p ~0.00000046)
But…. Can we use hCG to predict whether an FHB will be seen?
But…. Can we use hCG to predict whether an FHB will be seen?
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Tru
e P
os
itiv
e (
Se
ns
itiv
ity)
False Positives (1 - Specificity)
Low Day 28 hCG Predicts NO Fetal Heartbeat (AUC = 0.673)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Tru
e P
osi
tiv
es
(Se
nsi
tiv
ity
)
False Positives (1 - Specificity)
0.5
0.7
0.8
0.9
0.95
Incidence of Loss after +β
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 10 100 1000
Inci
de
nce
of P
regn
ancy
Lo
ss
(Fra
ctio
n o
f P
osi
tive
hC
Gs)
[hCG] (mIU/mL on day 28)
Does [hCG] Predict Outcome?
• Despite the highly significant difference between [hCG]’s comparing
– Fetal Cardiac Activity seen
– Fetal Cardiac Activity NOT seen
• [hCG] only poorly predicts whether FCA will or will not be seen
Adverse Outcomes: Day 3 vs. Day 5
Ob. Gyn. (2012) 120(1): 69-75
Ob. Gyn. (2012) 120(1): 69-75
Kalra et al. Data
• Using a large data set from SART spanning 3 years (69,039 live births in 2004 - 2006)
• Concluded that culture to day 5 (relative to day 3) resulted in higher incidence of premature delivery
• Close examination of their data reveals that the mean delivery date was 2.8 days earlier for day 5 embryos (37.8 versus 38.2 weeks)
• Mean birth mass at delivery was 29.2 grams lower for day 5 than for day 3 (3219.7 g versus 3,248.9 g)
P > 0.75
Comparison of Gestational Age at Birth: Day 3 vs. Day 5 at NYUFC
Comparison of Birth Weight: Day 3 vs. Day 5 at NYUFC
P > 0.2
Using the data
• Is this significantly different? – 2.8 day / 0.14 (SE) yes, indeed 2.8 days is significant
• A 2.8 day difference in the means
• But is 2.8 days clinically relevant? – 2.8 days within a 70 day range (4 X S.D.)
• Is this significantly different? – 29.2 g / 2.8 g (SE) yes, indeed 29.2 g is significant
• A 29.2 g difference in the means
• But is 29.2 g clinically relevant? – 29.2 g within a 2448 g range (4 X S.D.)
Incubator QC Parameters • Analysis of quality control data
– Fluctuations – are there trends with QC values?
• Multiple regression
• Can we find laboratory parameters that are associated with outcomes?
– Clinical Pregnancy
– Live Birth
– Live Birth per Clinical Pregnancy
• The inverse of clinical pregnancy loss…
Table I: Values of Daily Laboratory Parameters1
1 Displaying the mean + standard deviations (n) for the parameters examined for association with outcomes (Clinical Pregnancy, Live Birth, or Live
Birth per Clinical Pregnancy). 2 the day of embryo culture where 0 is the day of oocyte retrieval. 3 age (years) of the female patient on the day that gondatotropin injections were initiated. 4 temperature (oC) 5 percentage of the incubator gaseous environment that comprised carbon dioxide (%) 6 relative humidity (%)
Culture
Day2
Female
Patient
Age3
Incubator
Temperature4
Incubator
CO25
Incubator
Humidity6
Room
Temperature4
Room
Humidity5
Inverted
Microscope
Stage
Temperature4
Dissecting
Microscope
Stage
Temperature4
FIV
Temperature4
Number of
Patients in
Incubator
0 35.9
+ 4.6
(1503)
37.07
+ 0.13
(1503)
5.39
+ 0.29
(1503)
88.2
+ 5.8
(1503)
21.7
+ 1.1
(1503)
36
+ 14
(1503)
36.80
+ 0.39
(1503)
36.75
+ 0.42
(1503)
37.00
+ 0.10
(1503)
2.16
+ 0.99
(1503)
1 35.9
+ 4.6
(1508)
37.08
+ 0.13
(1508)
5.39
+ 0.29
(1508)
88.1
+ 6.5
(1508)
21.6
+ 1.0
(1508)
36
+ 14
(1508)
36.79
+ 0.38
(1508)
36.76
+ 0 36
(1508)
37.00
+ 0.07
(1508)
2.17
+ 0.99
(1508)
2 35.9
+ 4.6
(1498)
37.07
+ 0.15
(1498)
5.39
+ 0.30
(1498)
88.1
+ 6.2
(1498)
21.6
+ 1.1
(1498)
36
+ 14
(1498)
36.80
+ 0.41
(1498)
36.75
+ 0.37
(1498)
37.00
+ 0.07
(1498)
2.20
+ 1.01
(1498)
3 35.8
+ 4.6
(1513)
37.07
+ 0.14
(1513)
5.39
+ 0.29
(1513)
88.1
+ 6.0
(1513)
21.6
+ 1.0
(1513)
36
+ 14
(1513)
36.80
+ 0.41
(1513)
36.72
+ 0.38
(1513)
37.00
+ 0.07
(1513)
2.19
+ 1.01
(1513)
4 36.7
+ 3.9
(343)
37.07
+ 0.12
(343)
5.42
+ 0.29
(343)
87.7
+ 5.1
(343)
21.6
+ 1.0
(343)
35
+ 14
(343)
36.86
+ 0.42
(343)
36.75
+ 0.40
(343)
37.00
+ 0.02
(343)
2.15
+ 0.98
(343)
5 36.6
+ 4.0
(345)
37.07
+ 0.20
(345)
5.41
+ 0.29
(345)
87.7
+ 5.8
(345)
21.6
+ 1.0
(345)
35
+ 14
(345)
36.83
+ 0.37
(345)
36.7
+ 0.40
(345)
37.00
+ 0.02
(345)
2.11
+ 1.04
(345)
Table IV: Multiple Logisitic Regression Coefficients – Live Birth per Clinical Pregnancy1
Cultur
e
Day2
Femal
e
Patient
Age3
Incubator
Temperatur
e4
Incubato
r CO25
Incubato
r
Humidity6
Room
Temperatur
e4
Room
Humidity6
Inverted
Microscope
Stage
Temperatur
e4
Dissecting
Microscope
Stage
Temperatur
e4
FIV
Temperatur
e4
Number
of
Patients
in
Incubato
r
Offset
0 -0.140 0.0281 3.607
1 -0.131 -0.739 0.0362 6.550
2 -0.135 -0.929 10.886
3 -0.128 -0.466 0.0303 5.454
4 -0.164 -1.345 56.688
5 -0.175 -0.778 36.091
1 Displaying only those coefficients for parameters found to be associated with the outcome (Live Birth per Clinical Pregnancy) when the Akaike
Information Criterion was applied following maximum likelihood fitting. Blank cells were retained to indicate parameters that were examined but
found not to be associated with Live Births per Clinical Pregnancy. 2 the day of embryo culture where 0 is the day of oocyte retrieval. 3 age (years-1) of the female patient on the day that gondatotropin injections were initiated. 4 temperature (oC)-1 5 percentage of the incubator gaseous environment that comprised carbon dioxide (%)-1 6 relative humidity (%)-1
Lab Parameters: Multiple Regression
• Several Parameters Examined Simultaneously – Some params only significant when adjustments made
• Sensitivity of embryos may vary with stage – More sensitive prior to compaction?
• Surprising sensitivity of clinical pregnancy loss – increased losses with – Lower Humidity - Days 0 – 3
– Higher CO2 - Days 1,2,3
– This sensitivity is important to consider!
• Must be cautious of multicollinearity
Regression Equation for Day 3:
• Ln(ORlivebirth/clinical pregnancy) =
-0.128 X Female Patient Age (years)
-0.466 X Incubator [CO2] (%)
+0.0303 X Incubator Humidity (%)
+ 5.454
(from multiple logisitic regression; N = 1498)
Relative effects on 73.7% LB/CP • Age: 35.8 + 4.6 years
+ 2 std dev (26.6 – 45 years)
Change of live birth rate -43.7% (90.0 – 46.2%)
• CO2: 5.39 + 0.29 %
+ 2 std dev (4.81 – 5.97% CO2)
Change of live birth rate -10.4% (78.6 – 68.1%)
• Humidity: 88.1 + 6.0 %
+ 2 std dev (76.1 – 100.1% humidity)
Change of live birth rate +14.1% (66.1 – 80.1%)
Relative Contributions: Age, Humidity
7577
7981
8385
8789
9193
9597
99
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
Patient Age (years)
Humidity (%)
Live
bir
th /
clin
ical
pre
gnan
cy
Clinical Relevance vs. Statistical Significance
• Approaches to the solution
– Using the Standard Deviation as a measure of variability
– Comparisons of the change in outcome compared to the Standard Deviation are more realistic estimates of the effect – than –
– The change in outcome compared to the Standard Error of the Mean
Others have thought about this
• Effect Size: (M1 – M2)/Std Dev.
< 0.2 - small effect
0.2 – 0.7 - moderate effect
> 0.7 - large effect
Kazis et al., (1989); Cohen (1977)
Outcome Independent variable
difference Std. Dev. Effect Size
Fetal Cardiac Activity
hCG on day 28
50 mIU/ml 92 mIU/ml 0.54
Gest. Age Xfer Day (3/5) 2.8 days 17.5 days 0.16
Birth Weight Xfer Day (3/5) 29.2 grams 612 grams 0.048
Preg Losses Incubator param
10.4% (CO2) 14.1% (humid) (+/- 2 Std Dev)
~45% 0.23 (CO2)
0.31 (humid.)
Table of Effect Sizes
Using the data
• Is this significantly different? – 2.8 day / 0.14 (SE) yes, indeed 2.8 days is significant
• A 2.8 day difference in the means
• But is 2.8 days clinically relevant? – 2.8 days / 17.5 days (SD) is a very small effect (0.16)
• 70 day range (4 X S.D.)
• Is this significantly different? – 29.2 g / 2.8 g (SE) yes, indeed 29.2 g is significant
• A 29.2 g difference in the means
• But is 29.2 g clinically relevant? – 29.2 g / 612 g (SD) is a very small effect (0.04)
• 2448 g range (4 X S.D.)
Statistics, hmmmph! • Definition of Statistics: The science of producing
unreliable facts from reliable figures. Evan Esar
• There are lies, damned lies and statistics. Mark Twain
• Statistics are no substitute for judgment. Henry Clay
• Facts are stubborn, but statistics are more pliable. Mark Twain
• Statistics: the mathematical theory of ignorance. Morris Kline
• Like dreams, statistics are a form of wish fulfillment. Jean Baudrillard
• If your experiment needs statistics, you ought to have done a better experiment. Ernest Rutherford
Thank you!