Analyzing the Power and Error of Listeria monocytogenes Growth

Challenge StudiesMark Powell

U.S. Department of Agriculture

Washington, DC

IAFP 2009, Grapevine, TX, 12-15 July

Introduction

• For ready-to-eat (RTE) foods that do not support growth of L. monocytogenes, food safety criteria limit of 100 colony forming units (cfu)/g.– EC Regulation 2073/2005– FDA (2008) draft compliance policy guide– Codex (2009) microbiological criteria

• For RTE foods that do support growth of L. monocytogenes, “zero tolerance” (i.e., not detected in a regulatory sample).

• Design and interpretation of challenge studies to determine whether RTE are unable to support growth of L. monocytogenes.

Introduction

• Type I (F+) error (α): probability that H0 is rejected when true.

• Type II (F-) error (β): probability that H0 is not rejected when Ha is true.

• Power = (1-β). • By convention, α ≤ 0.05 and (1-β) ≥ 0.8

Fixed Exceedance Values

• To distinguish real growth from measurement uncertainty, L. monocytogenes challenge study protocols apply a fixed exceedance value: difference (δ) < M.

• EU/CRL (2008): difference between the initial and final sample median concentrations < 0.5 log10 cfu/g for all batches tested (Mm = 0.5 log).

• CCFH (2009): ≤ (on average) 0.5 log10 cfu/g increase for at least the expected shelf life (Mxbar = 0.5 log).

• FDA (2008): < 1 log10 increase during replicate trials (assume Mxbar = 1 log).

Fixed Exceedance Values

• M ~ ISO “expanded uncertainty” (U)

• x ± U = x ± 2σx

– where σx = std. error of meas. uncertainty

• 2 (k factor) ≈ z(1-0.05/2) = 1.96

– α = 0.05; 2-sided interval

• If σx = 0.25 log, → M = 0.5 log

• If σx = 0.50 log, → M = 1.0 log

Variance of a difference

• If two means are independent:

– where:

– Assuming equal σx and n:

222

00 txtxtxtx

nxx /22

2n if ONLY 2 2

0 x

xtxtx n

Quantitative Measurement Uncertainty

• σx ≠ 0.25 or 0.5 or any other fixed value.• EC, FDA, and CCFH reference ISO

Method 11290-2 for enumerating L. monocytogenes in RTE foods.

• Scotter et al (2001): std dev reproducibility (sR) = 0.17 - 0.45 log cfu/g in food samples.

• sR: an intra-laboratory measure of quantitative measurement uncertainty.

Challenge Study Designs Differ

• Number of sampling times• Number of batches• Experiment-wise α depends on:

– Number of comparisons– Whether multiple comparisons are

independent or dependent.• Independent: (μfinal – μinitial) X multiple

batches• Dependent: μ(t) – μ(t0) within a batch

Challenge Study Designs Differ

• EU/CRL (2008): k = 2 sampling times (initial and final), b ≥ 3 batches, sample size (n) = 3 samples per sampling time.– c ≥ 3 multiple, independent pair-wise comparisons.– std dev w/in batch < 0.3 log at t0.

• FDA cites Scott et al. (2005): k = 5-7 sampling times, sample size (n) = 2-3 samples per sampling time.– c = k-1 dependent pair-wise comparisons per trial

(μ(t) – μ(t0)).– No minimum number of batches.

Type I error for fixed exceedance value (Mxbar)

• For a single comparison test of H0: δ ≤ 0:

• For multiple independent comparisons:

• For multiple dependent comparisons, Monte Carlo simulation, with α = proportion (F+)

• Based on Scotter et al (2001), consider σx from 0.15 log cfu/g to 0.50 log cfu/g

nMMtxtxp x

xx

2

0

2,0|1

c 11

Type I error for difference in means fixed exceedance value (Mxbar) = 0.5 log cfu/g

** p<0.01

std. dev. (log

cfu/g)

sample size (n) = 2 sample size (n) = 3

independent comparisons (c) independent comparisons (c)

1 2 3 4 5 6 1 2 3 4 5 6

p(type I error) ≤ α

0.15 ** ** ** ** ** ** ** ** ** ** ** **

0.20 0.01 0.01 0.02 0.02 0.03 0.04 ** ** ** ** 0.01 0.01

0.25 0.02 0.04 0.07 0.09 0.11 0.13 0.01 0.01 0.02 0.03 0.04 0.04

0.30 0.05 0.09 0.14 0.18 0.22 0.25 0.02 0.04 0.06 0.08 0.10 0.12

0.35 0.08 0.15 0.21 0.27 0.33 0.38 0.04 0.08 0.12 0.15 0.19 0.22

0.40 0.11 0.20 0.28 0.36 0.43 0.49 0.06 0.12 0.18 0.23 0.28 0.32

0.45 0.13 0.25 0.35 0.44 0.51 0.58 0.09 0.17 0.24 0.30 0.36 0.42

0.50 0.16 0.29 0.40 0.50 0.58 0.65 0.11 0.21 0.30 0.37 0.44 0.50

Type I error for difference in means fixed exceedance value (Mxbar) = 0.5 log cfu/g

** p<0.01

std. dev. (log

cfu/g)

sample size (n) = 2 sample size (n) = 3

dependent comparisons (c) dependent comparisons (c)

1 2 3 4 5 6 1 2 3 4 5 6

p(type I error) ≤ α

0.15 ** ** ** ** ** ** ** ** ** ** ** **

0.20 0.01 0.01 0.02 0.02 0.03 0.03 ** ** ** ** 0.01 0.01

0.25 0.02 0.04 0.06 0.07 0.08 0.10 0.01 0.01 0.02 0.02 0.03 0.03

0.30 0.05 0.09 0.11 0.14 0.16 0.18 0.02 0.04 0.05 0.06 0.08 0.09

0.35 0.08 0.13 0.17 0.21 0.23 0.26 0.04 0.07 0.09 0.12 0.14 0.15

0.40 0.11 0.18 0.23 0.27 0.31 0.33 0.06 0.11 0.14 0.17 0.20 0.22

0.45 0.13 0.22 0.28 0.33 0.36 0.39 0.09 0.14 0.19 0.23 0.26 0.29

0.50 0.16 0.25 0.32 0.37 0.41 0.45 0.11 0.18 0.24 0.28 0.32 0.34

Power of F-test for One-Way ANOVA

• SAS© PROC Power

– where:– Fω = non-central F dist– Fcrit = critical value of the F dist with k-1 and k(n-1) df

– ω (non-centrality parameter) = – H0: μi = μ for all i– Ha: μmax – μmin = δ– Power depends on δ and growth pattern under Ha

,,|111 21 dfdfFF crit

2

1

2 /.

k

iin

Pattern that maximizes power for δ = 1 log

0

0.5

1

1.5

2

2.5

1 2 3 4 5 6

time

log1

0 cf

u/g

Pattern that minimizes power for δ = 1 log

0

0.5

1

1.5

2

2.5

1 2 3 4 5 6

time

log1

0 cf

u/g

Power curves for one-way ANOVA F-test (α = 0.05, δ = 1 log cfu/g)

with sample size n = 2 and sampling times k = 2-7

0.0

0.10.2

0.3

0.4

0.50.6

0.7

0.80.9

1.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6

Std Dev (log10 cfu/g)

Po

we

r

n=2 k=7

n=2 k=6

n=2 k=5

n=2 k=4

n=2 k=3

n=2 k=2

max

min

Power curves for one-way ANOVA F-test (α = 0.05, δ = 1 log cfu/g)

with sample size n = 3 and sampling times k = 2-7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6

Std Dev (log10 cfu/g)

Po

we

r

n=3 k=7

n=3 k=6

n=3 k=5

n=3 k=4

n=3 k=3

n=3 k=2

max

min

Conclusions

• Applying any fixed acceptance criteria exceedance value (e.g., less than a 0.5 log or 1 log increase) to distinguish real growth from quantitative measurement uncertainty over different experimental designs and/or measurement uncertainty values implies highly inconsistent type I error probabilities.

Conclusions

• None of the L. monocytogenes growth challenge study designs currently being considered are likely to provide an F-test with α = 0.05 and power ≥ 0.8 to detect a 1 log increase in mean concentration over the entire range of measurement uncertainty values for enumeration of L. monocytogenes reported in food samples in a validation study of ISO Method 11290-2.

Conclusions

• Satisfying these conventional experimental design criteria would require a larger sample size, lower measurement uncertainty, or both.

Thank you