is it method validation, verification or or just semantics?€¦ · how to meet iso 17025...
TRANSCRIPT
Is it Method Validation,
Verification or or Just
Semantics?
Michael BrodskyBrodsky Consultants
Objectives
Distinguish method validation from verification
Consider the requirements for method validation and in-house verification from an ISO Standard 17025 perspective
Review AOACI OMA and RI SLV and Multi-Lab Collaborative Study validation protocols as applicable to “in-house” validation
Outline the requirements for a verification protocol that defines a method’s fitness-for purpose
What is Test Method
Validation?
Validation is the establishment of one or more performance characteristics for a test method
By single laboratory validation (SLV) By multi-laboratory collaborative study Always by comparative analysis to
a “Reference” Method
Verification
Demonstration of Analytical competency.
Performance characteristics of the method, when performed, as prescribed, by laboratory analysts, conforms to the performance characteristics of the method as established during the validation study.
Ensures that the method is fit for its intended purpose
Includes determination of uncertainty of measurement (quantitative methods).
Performance Characteristics of
Microbiological Methods
Relative Accuracy (%Recovery)
Precision (Repeatability and Reproducibility)
Specificity (Selectivity for target analyte)
Sensitivity (Distinguishing target from non target)
Inclusivity (Range of target analytes detected by method)
Exclusivity (Range of non-target analytes excluded)
False Positive and False Negative Rates
Limits of Detection (LOD)
Limit of Quantitation (LOQ)
Scope of application
Categories of Microbiological Test Methods
Performance Characteristics Included in a Validation Study
Performance Characteristic Identification Quantitative Qualitative
(P/A)
Relative Accuracy No Yes No
Matrix Effects (Scope) No Yes Yes
Precision No Yes No
Sensitivity Yes Yes Yes
Specificity Yes Yes Yes
Inclusivity Yes Yes Yes
Exclusivity Yes Yes Yes
False Positive Rate No Yes Yes
False Negative Rate No Yes Yes
LOD No Yes Yes
LOQ No Yes No
Ruggedness Yes Yes Yes
Linearity/Range No Yes No
Categories of Microbiological Test Methods
Performance Characteristics Included in the Verification of a Validated Method
Performance
Characteristic
Identification Quantitative Qualitative
(P/A)
Verification
(Where
Applicable)
Relative Accuracy No Yes No Yes
Matrix Effects (Scope) No Yes Yes No
Precision No Yes No Yes
Sensitivity Yes Yes Yes Yes
Specificity Yes Yes Yes Yes
Inclusivity Yes Yes Yes No
Exclusivity Yes Yes Yes No
False Positive Rate No Yes Yes No
False Negative Rate No Yes Yes No
LOD No Yes Yes No
LOQ No Yes No No
Ruggedness Yes Yes Yes No
Linearity/Range No Yes No No
Methods requiring validation are:
Modified official methods
In-house developed methods
Previously validated methods extended to a component, analyte or matrix not previously tested or included in the validation study
17025 Requirements
Food Categories and Types
Food Categories and Types(Based on Physiochemical and Innate
Microbial Characteristics)
Food Categories/Matrices1. Raw milk and dairy products
2. Heat processed milk and dairy products
3. Ready-to-eat, ready-to-reheat meat products
4. Raw poultry and ready-to-cook poultry products
5. Ready-to-eat, ready-to-reheat meat poultry products
6. Eggs and derivatives
7. Raw and ready-to-cook fish and seafoods (unprocessed)
8. Ready-to-eat, ready-to-reheat fishery products
9. Fresh produce and fruit
10. Processed fruits and vegetables
11. Infant formula and infant cereals
12. Dried cereals, fruits, nuts, seeds and vegetables
13. Chocolate, bakery products and confectionary
14. Multi-component foods or meal components
15. Pet food and animal feed
16. Environmental samples (food or feed production)
Food Categories and Types
9. Fresh produce and fruit
Types:
i. Cut ready-to-eat fruit
ii. Cut ready-to-eat vegetables
iii. Produce grown in or in contact with the ground
iv. Sprouts
v. Raw fruit/vegetable juices (unpasteurized)
vi. Leafy greens
vii. Vegetables and fruits (unprocessed) not described above
Food Categories and Types
15. Pet food and animal feed
Types:
i. Animal origin ingredients
ii. Plant origin ingredients
iii. Other ingredients
iv. Dry food (aw ≤ 0.7)
v. Wet food (aw > 0.7)
vi. Canned
vii. Animal feeds (bovine, ovine, pig)
viii. Animal feeds (poultry)
ix. Animal feeds (fish)
AOACI Pre-Collaborative/PTM Study
Design (SLV)
For Each Food Category to be claimed:
≥2 food types/category
If claiming if applicable to a broad range of foods:
Test ~20 samples from 9 different Food categories
AOACI Pre-Collaborative/PTM Study
Design (SLV) (Quantitative Method)*
20 analyses for each food type:
5 portions of each matrix @
Low, medium and high levels of contamination and uncontaminated
Comparison of recovery to reference method using ANOVA and comparison of means
*Use as guide for “in-house’ validation
AOACI Pre-Collaborative/PTM Study
Design (SLV) (Qualitative Method)*
30 analyses for each food type:
5 replicate test portions per level the high inoculation level,
20 for the fractional positive level
5 for the uncontaminated level
Comparison to reference method using Chi-
square analysis &/or Probability of Detection (POD models)
*Use as guide for “in-house’ validation
AOACI Collaborative Study –
Quantitative 10 - 12 laboratories
Minimum of 8 labs with acceptable data
Test samples*
1 Matrix
Low, medium and high contamination +uncontaminated
Five samples tested at each level
Both naturally and artificially contaminated food
Compare against a reference method
Repeatability, Reproducibility and differences between means
*Use as guide for “in-house’ validation
AOACI Collaborative Study
(Qualitative Method)
12 - 15 laboratories
Minimum of 10 labs with acceptable data
Test samples*
1 matrix
12 test portions per high analyte level
12 test portions per fractional Positive samples
12 uncontaminated test portions
*Use as an alternate guide for “in-house’ validation
Comparison to reference method using Chi-square analysis &/or Probability of Detection (POD models)
"I think you should be more explicit
here in step two."
17025 Requirements
Official reference methods already published and intended for a specific matrix will be incorporated into the current method document format.
Do not need to be fully validated.
The ability to perform the analysis must be verified using spiked samples, proficiency samples or CRMs
Verification of Quantitative
Microbiological Methods
• Precision (Repeatability and Reproducibility)
• Uncertainty of Measurement• Fitness-for-Purpose
Validation Data from
Collaboratively Studied Methods
Provides reference values for RSDR and RSDr
e.g. Pour Plate counting (SMEDP)
e.g. RSDr ≤ 7.7% (0.077) (within analysts)
e.g. RSDR ≤ 18.2% (0.182) (between analysts)
Validation Data from
Collaboratively Studied Methods
Can also be used to estimate uncertainty of measurement
e.g. Pour Plate counting (SMEDP)
e.g. RSDr ≤ 7.7% (0.077) (within analysts)
e.g. RSDR ≤ 18.2% (0.182) (between analysts)
Calculation of Combined
Uncertainty
Calculate the combined uncertainty (Uc) using standard propagation of error rules (the square root of the sums of squares of SDs known as the “root sum of squares” - RSS).
Validation Data from
Collaboratively Studied Methods
Calculation of Combined Uncertainty (Uc):
Root Sum of Squares: √(RSDr)2 + (RSDR)
2
For Pour Plate (HPC) Sum of Squares: (0.077)2 + (0.182)2 =0.0371Uc = √(0.0371) = 0.193=19.3%
Expanded uncertainty (Ue): (Use coverage factor k=2* for 95% confidence) = 2 x 19.3% = 38.6%
*≥30 Observations
Verification of Qualitative
Microbiological Methods
• Specificity• Sensitivity• Fitness-for-Purpose
How Many Samples Are Needed
for Verification?
?
How Many Samples Are Needed
for Verification?
No fixed number
But, a minimum of 15 positive samples run in duplicate (30 observations) per matrix is not unreasonable
10 is the minimum requirement
ISO: Microbiology of food and animal feed –Method Verification– Part 4: Protocol for the verification of reference and alternative methods implemented in a single laboratory
HC/CFIA: Part 5: Guidelines to Verify Standard Food Microbiological Methods for Implementation in Routine Testing
Who’s Doing What
Minimum of 10 artificially inoculated samples
Use CRM, if available
For Qualitative methods inoculate with 1-5 CFU per test portion.
For Quantitative methods, the levels of contamination shall cover the range of the method
Analyse samples on 10 different occasions or days
be performed by at least 2 technicians working independently and with separate samples and reagents.
ISO Protocol for the verification of reference and
alternative methods implemented in a single
laboratory (under review )
For Quantitative methods, ISO proposes to accept:
participation in interlaboratory comparisons such as proficiency testing and plotting z-scores to show any trends;
use of microbiological RM or CRM;
recovery experiments with spiked samples using a non-selective method.
How to Interpret Verification?
For Qualitative Methods, Analyze 3-5 samples spiked at 3-5 times the reported
or determined LOD (1-3 CFU/analytical unit)
For Quantitative Methods using selective medium, measure repeatability by
analyzing 10 or more replicate samples of a representative food matrix naturally or artificially contaminated
Part 5: Guidelines to Verify Standard Food
Microbiological Methods for Implementation in
Routine Testing by Health Canada (April 2015)
For Qualitative methods, ISO and HC agree: 100% (comparative) sensitivity All samples must be correctly identified
For Quantitative methods, for acceptable precision as suggested by HC, the repeatability (r) must be less than half of the reproducibility (R) data. 2r < R. If R value if published.
If there is no published performance characteristic data, ???
Calculate Ue
How to Interpret Verification?
17025 Requirements
Fitness-for-Purpose
The data obtained from the method approval process must show it is fit for the intended use and relevant to customers’ needs.
Does the method have performance characteristics that meets the expectation of the laboratory and the needs of the client?
References AOAC International Methods Committee Guidelines For
Validation Of Qualitative And Quantitative Food Microbiological Official Methods Of Analysis, Appendix J.
The Fitness for Purpose of Analytical Methods, A Laboratory Guide to Method Validation and Related Topics, EURACHEM Working Group
How to Meet ISO 17025 Requirements for Method
Verification, 2015, AOAC Technical Division for Laboratory
Management (TDLM), the Analytical Laboratory
Accreditation Criteria Committee (ALACC)
ISO/FDIS 16140-2 Microbiology of the food chain —
Method validation — Part 2: Protocol for the validation of
alternative (proprietary) methods against a reference method
(ISO/TC 34/SC 9) 2014-09-05 (Draft)
