validation of a rapid microbial method - pda midwest of a rapid microbial method approach to...
TRANSCRIPT
Validation of a Rapid Microbial MethodApproach to Validation
Case Studies: Equivalence Verification of an Alternate Assay
for Microbial Limits Screening and Sterility Testing of
Pharmaceutical Products
Erin Patton, MS
Senior Product Specialist
Charles River Labs, Microbial Solutions
Overview
Two-tier approach to validating a rapid microbial method
• Provides a pathway to validation that streamlines the validation and
regulatory process
• Consistent with industry guidance
• Has been effectively utilized to obtain regulatory approvals both in
the US and EU
Two Case Studies that utilize approach
• Study I – Comparison of 24-hour Rapid Alternative Bioburden assay
and Compendial Pour Plate Method for Micro Limits Testing
• Study II – Comparison of 5-day Rapid Alternative Sterility assay and
Compendial Sterility Test using Membrane Filtration
Regulatory Overview
Regulators do not “approve” RMM technologies
• User responsible for seeking approval for use with their
products and/or processes
A growing number of companies have received approval for the use
of RMMs for finished product testing using various submission
strategies
Guidance for the Validation of Rapid Microbial Methods
• PDA TR33
• USP<1223>
• EP 5.1.6
Establish a multifunctional team:
Microbiologist, Technician, QA/Regulatory Specialist, Statistician
Validation Approach / Qualitative MethodPDA TR33 Guidance
PDA TR33Validation
Criteria
Qualitative Method
User or Supplier
DemonstratedRecommended Approach
Equivalence / Comparative Testing
Yes User Equivalency Protocol
Limit of Detection Yes UserGenerated from
Equivalency Protocol
Method Suitability Yes UserTypical
Sample Effects
Specificity Yes UserTypical
Spiking studies
Ruggedness Yes Supplier Drug Master File
Robustness Yes Supplier Drug Master File
Two Tier Approach to Validation
Tier One: Equivalency and Limit of Detection Performed on one product
Steps
• Time to Detect Study (for sterility only)
• Equivalency Study (Equivalency, Limit of Detection)
Tier Two: Method Suitability and Specificity Performed on all other products or product groups
Steps
• Sample Effects (Method Suitability)
• Spiking Study (Specificity)
Use of vendor data to support Robustness and Ruggedness claim
including DMF reference
Approach to Equivalence VerificationStudy Objectives
The primary goal is to demonstrate the equivalence of two
microbiological test methods for the qualitative screening of a
pharmaceutical product for microbial contamination.
The ability of the alternate method to detect contamination in a
product matrix is compared to that of the compendial method in a
series of side-by-side assays conducted on identical samples.
Equivalence is tested by measuring the relative rates of agreement
and disagreement between the two test methods.
Approach to Equivalence VerificationStudy Design
Demonstrate the comparability of the two methods in a series of side by
side assays, on multiple lots of product, with different inoculum levels.
Relative rates of agreement and disagreement between the methods are
used to assess the accuracy, precision, specificity of the alternate method
using a receiver operating characteristics table (ROC).
The hypothesis that the two methods are equivalent will be tested using an
acceptance criteria of 70% for specificity, accuracy and precision. The
hypothesis of superiority will be tested using McNemar’s test.
Superiority will be defined as a statistically significant difference in the limits
of detection between the methods using McNemar’s test.
The microorganisms chosen for comparison were those recommended by
USP/EP/JP Pharmacopoeias
Fifteen + pairs of assessments are conducted for each organism. • Five tests at each concentration of inoculum
• Three inoculum levels
• Six plus different organisms
McNemar’s test is used to test the superiority of the new method,
with respect to the ability to detect a contaminated sample
When demonstrating equivalence of two methods, USP <1223>
requires an examination of Accuracy, Precision, and Specificity • High values (>70%) are expected if the methods are equivalent.
• If the new method is superior, the values for these parameters
may be low
Approach to Equivalence VerificationStudy Design
Approach to Equivalence VerificationStudy Design Alternate Method Compendial Test
10
Inoculum Load(cfu/test)
A. brasiliensis
C. albicans
P. aeruginosa
E. coliS. aureus
B. subtilis
C. sporogenes
P. acnes
A. brasiliensis
C. albicans
P. aeruginosa
E. coliS. aureus
B. subtilis
C. sporogenes
P. acnes
A. brasiliensis
C. albicans
P. aeruginosa
E. coliS. aureus
B. subtilis
C. sporogenes
P. acnes
T1 T2 T3 T4 T5 T1 T2 T3 T4 T5
1
0.1
STA
TIS
TIC
AL
AN
ALY
SIS
*Appropriate organism panel TBD
The values in the table represent the number of paired samples having
positive/negative results for the alternate test and the compendial test
+
+-
-
Compendial
Alt
ern
ate
A+ +
D- -
B+ -
C- +
ROC Table Agreement Positive
Agreement Negative
Disagreement + / -
Disagreement - / +
A
D
B
C
Approach to Equivalence VerificationStatistical Analysis; Receiver Operator Characteristics Table
Accuracy
Precision
Specificity
Sensitivity
+
+-
-
Compendial
Alte
rna
te 19
19
8
8
0.70
0.70
0.70
0.70
Equivalent
+
+-
-
Compendial
Alte
rna
te 27
0
27
0
0.00
0.50
0.50
1.00
Superior
+
+-
-
Compendial
Alte
rna
te 18
0
18
18
0.00
0.50
0.33
0.50
Inferior
The ROC table tests for equivalence between an alternate method and a “gold standard.”
It does not take into account the possibility that the alternate method may be superior to
the gold standard.
McNemar’s test is used to test the hypothesis that the alternate method is superior to
the compendial method in its ability to detect a contaminated inoculated sample.
Approach to Equivalence VerificationInterpretation of Hypothetical Results
Taking into account that the referenced method is the compendial one, the accuracy,
precision and specificity of the alternative method are expressed in terms of relative
rates of false positive and false negative results.
Accuracy, calculated as (A+D) / (A + B + C + D), gives an indication on the
equivalency/closeness of both methods.
Precision, calculated as A / (A + B), gives a degree of agreement between both
methods on the positive results obtained with repeated tests.
Specificity, calculated as D / (D+B), gives information on the degree of
interference due to external events during the analysis.
Sensitivity, calculated as A / (A+C), gives a degree of agreement between the
methods taking into account the number of discordant negative results with the
alternate method.
Approach to Equivalence VerificationStatistical Analysis; Receiver Operator Characteristics Table
Approach to Equivalence Verification
If the rate of positives are not found to be significantly different using
McNemar’s test, the hypothesis of equivalence is tested using 70% as the
acceptance criteria for accuracy, precision, specificity and sensitivity.
This specification is based on expected results for each concentration,
taking into account the probability of growth of the microorganisms at
different concentrations of inoculum.
This specification also takes into account the variability linked to very low
inoculation levels.
Analysis
LOGISTIC REGRESSION
LOD 1 cfu
Approach to Equivalence VerificationStatistical Analysis; Limit of Detection
Case Studies
Results of two studies designed to demonstrate equivalence of Amplified
ATP Bioluminescence assay to compendial methods for detecting microbial
contamination in pharmaceutical products
Study I – Micro Limits Testing of non-sterile, non-filterable betamethasone
suspension
• Direct inoculation protocol
• Commissioned by Celsis and performed by an independent lab using a
validated method and product provided by a commercial client.
Study II – Sterility Testing of sterile, filterable Saline solution
• Membrane filtration protocol
• Commissioned by a pharmaceutical client and performed by an
independent lab using a validated method and product provided by the
client.
Case Study One: Micro Limits Testing
Product – Betamethasone Suspension
(Betamethasone acetate/Betamethasone sodium phosphate 4mg/3mg per mL)
• Non preserved, non-filterable solution
Organisms and Culture Conditions
• USP compendial plus E. coli organisms were obtained as lyophilized Bioballs™
• Bioballs™ were dissolved in saline and serially diluted to the final concentration.
Compendial Method parameters
• Microbiological quality was evaluated through plate count method
» 1 ml of product
» SDA plates incubated at 20-25˚C for 5 days
» TSA plates incubated for 3 days at 30-35˚C.
Study Overview / Product – Betamethasone Suspension
Alternate Method parameters
• Microbial quality evaluated using AMPiScreen™ enhanced Bioluminescence assay
» 1 mL of Product into 25 mL of TSB.
» Incubation for 24 hours at 30-35˚C with shaking.
» After incubation, samples were agitated on a linear shaker for 30 min. with
glass beads
» Two 50 µL aliquots of the sample were transferred into duplicate
cuvettes and placed into the Advance Luminometer.
» Bioluminescence was measured and results were recorded as relative light
units (RLUs)
• Samples with mean RLU values 3X the baseline control were scored as positive
Case Study One: Micro Limits TestingStudy Overview
AMPiScreen™(Broth)
Plate Count(Agar)
A. brasiliensis
C. albicans
P. aeruginosa
E. coli
S. aureus
B. subtilis
A. brasiliensis
C. albicans
P. aeruginosa
E. coli
S. aureus
B. subtilis
A. brasiliensis
C. albicans
P. aeruginosa
E. coli
S. aureus
B. subtilis
+
+-
-
Plate Count
AM
PiS
cre
en 42
28
12
8
10
1
0.1
Inoculum Load
(cfu/test)5
9 13 7 5 12
1 2 1
9 4 7 6 6
8 15 10 10 7
1 1 1
1 1
7 6 11 3 5
1 1
1 1
1
1 1 2 1
6 7 7 3
1 2 1 1
4 10 10 11 5
9.2
6.4
10
6.4
5.8
8
Mean
Case Study One: Micro Limits TestingResults
+
+-
-
Plate Count
AM
PiS
cre
en 42
28
12
8
ALL CONCENTRATIONS
McNemar’s Test for SuperiorityProb(+| ATP+) = (42+ 12) / 90 = 60.0%
Prob(+| PC) = (42+ 8) / 90= 55.6%
Statistic (S) = 1.565
P-value (one sided) = .10502
The McNemar’s test concluded that the AmpiScreen
method was not significantly better at detecting a contaminated
sample than the plate count method.
Receiver Operator CharacteristicsAccuracy (42+28) / (42+12+8+28) * 100 = 77.8%
Precision (42/ (42+ 12)) * 100 = 77.8%
Specificity (28 / (28+ 12)) * 100 = 70.0%
Sensitivity (42/ (42+ 8)) * 100 = 84.0%
Each of the values computed for Accuracy, Precision, Specificity,
and Sensitivity satisfied the 70% acceptance criteria. As such,
the AMPiScreen™ method is equivalent to the compendial plate
count method for detecting a contaminated sample.
Case Study One: Micro Limits TestingAnalysis / Superiority or Equivalency
Case Study One: Micro Limits TestingAnalysis / Probability of Detection
Organism Concentration (cfu/mL of sample)
Probability of Detection / Inoculum 10 1 0.1 All
AMPiScreen (ATP+) 0.93 0.67 0.20 0.60
Plate Count (PC) 1.00 0.57 0.10 0.56
Difference -0.07 0.10 0.10 0.04
Case Study One: Micro Limits TestingStatistical Analysis; Limit of Detection
Logistic Regression Curve for Probability of Detection for AMPiScreen Method
Log10 value of the LOD was –1.85, or 0.014 CFU/mL.
Computed LOD 1 CFU
The probability of detecting contamination at 1 cfu/mL was identical.
Logistic Regression Curve for Probability of Detection for Plate Count Method
Log10 value of the LOD was –1.1, or 0.079 CFU/mL.
Computed LOD 1 CFU
Case Study Two: Sterility Testing
Product – Sterile Saline
• Filterable solution
Organisms and Culture Conditions
• USP compendial plus P. acnes organisms were obtained as lyophilized Bioballs™
from bioMerieux where available
• Bioballs™ were dissolved in saline and serially diluted to the final concentration.
• Compendial Method parameters
Microbiological quality was evaluated through membrane filtration method
250 mL of product was filtered and either TSB or FTM broth added to incubation vessels
TSB broth samples were incubated at 20-25˚C for 14 days
FTM broth samples were incubated at 30-35˚C for 14 days
Study Overview / Product – Sterile Saline
Alternate Method parameters
• Microbial quality evaluated using AMPiScreen™ enhanced Bioluminescence assay
250 mL of product was filtered and either TSB or FTM broth added to incubation vessels
TSB broth samples were incubated at 20-25˚C for 5 days
FTM broth samples were incubated at 30-35˚C for 5 days
After incubation, samples were vortexed for 30 seconds
Two 50 µL aliquots of the sample were transferred into duplicate
cuvettes and placed into the Advance Luminometer.
• Samples with mean RLU values 3X the baseline control were scored as positive
Case Study Two: Sterility TestingStudy Overview / Product – Sterile Saline
Case Study Two: Sterility TestingResults AMPiScreen Sterility Test
10
Inoculum Load(cfu/test)
A. brasiliensis
C. albicans
P. aeruginosa
S. aureus
B. subtilis
C. sporogenes
P. acnes
A. brasiliensis
C. albicans
P. aeruginosa
S. aureus
B. subtilis
C. sporogenes
P. acnes
A. brasiliensis
C. albicans
P. aeruginosa
S. aureus
B. subtilis
C. sporogenes
P. acnes
T1 T2 T3 T4 T5 T1 T2 T3 T4 T5
1
0.1
STA
TIS
TIC
AL
AN
ALY
SIS
+
+-
-
Compendial
AM
PiS
cre
en 42
42
11
10
ALL CONCENTRATIONS
McNemar’s Test for SuperiorityProb(+| ATP+) = (42+ 11) / 105 = 50.5%
Prob(+| PC) = (42+10) / 105 = 49.5%
x2 Statistic (S) =0.476
The McNemar’s test concluded the the Celsis AMPiScreen
method was not significantly better at detecting a contaminated
sample than the plate count method.
Receiver Operator CharacteristicsAccuracy (42+42) / (42+11+10+42) * 100 = 80%
Precision 42 / (42+11) * 100 = 79%
Specificity (42 / (11+42) * 100 = 79%
Sensitivity (42 / (42+ 10) * 100 = 81%
Each of the values computed for Accuracy, Precision, Specificity,
and Sensitivity satisfied the 70% acceptance criteria. As such,
the Celsis AMPiScreen™ method is equivalent to the
compendial plate count method for detecting a contaminated
sample.
Case Study Two: Sterility TestingAnalysis / Superiority or Equivalency
Case Study Two: Sterility TestingStatistical Analysis; Limit of Detection
Logistic Regression Curve for Probability of Detection for AMPiScreen Method
Log10 value of the LOD was –1.00, or 0.1 CFU/mL.
Computed LOD 1 CFU
The probability of detecting contamination at 1 cfu/mL was identical.
Logistic Regression Curve for Probability of Detection for Sterility Test Method
Log10 value of the LOD was –0.65, or 0.224 CFU/mL.
Computed LOD 1 CFU
Conclusions
Experimental Design
• Adaptive design that places both methods on equal footing
» Reveals subtle differences with respect for organisms and sensitivity
Equivalency
• Results demonstrate the systems are equivalent for both micro
limits testing and sterility testing
Validity of Approach
• Has been used to successfully obtain regulatory approval
Approach to Demonstrating Equivalency to Compendial Methods
