Validation of Molecular Methods in a Public Health Laboratory

Denise M. Toney, PhDCommonwealth of Virginia

Division of Consolidated Laboratory ServicesJune 5th, 2007 - APHL Annual Meeting

Introduction• “The key desirable attribute of a laboratory test is its ability to produce

accurate and precise results consistently over extended periods of time with an appropriately rapid turnaround time so that the test results are of clinical utility.”

• Laboratories must have a:– detailed method for evaluating the accuracy and precision of new tests

before these tests can be offered or replace older tests (verification)– method for demonstrating that the laboratory can obtain performance

specifications comparable to those established and reported by the manufacturer (verification)

– process for continued review of established tests to assure consistency of results and personnel performance of testing and results (validation)

Reference: Cumitech 31 (1997)

CLIA Clinical Laboratory Improvement Act

• CLIA applies to all facilities that perform:“examination of materials derived from the human body for the purpose of providing information for the diagnosis, prevention or treatment of any disease or impairment of, or the assessment of the health of, human beings”

• CLIA is applicable to all clinical testing

• Accreditation or licensure is carried out by a number of public and private agencies

– College of American Pathologists (CAP)– Joint Commission on the Accreditation of Health Care Organizations

(JCAHO)

Verification• Definition: The documentation of either commercial or home brew test

accuracy

• CLIA regulations (42 CFR Section 493.1253) indicate that for laboratory developed assays (ie. Home-brew assays) the following performance characteristics be determined:

• Analytical Sensitivity • Analytical Specificity (including effect of interfering substances)• Precision• Accuracy

– Positive predictive value– Negative predictive value

• Measuring range of test results• Reference intervals (normal values)

Public Health Approaches to Verification of Molecular Assays

Sensitivity (Limit of detection):Analytical: Document the lowest concentration or quantity of target that

can be consistently detected• Dilutions of bacteria, viruses, or nucleic acid• Control materials with know copy numbers (IS481)

Diagnostic: Compares whether the proportion of diseased subjects have a positive test result

Specificity (Cross reactivity or false positives):Analytical: Test for the ability to ONLY detect the target of interest

• Competitor organisms• Challenge inoculums• Effect of interfering substances

Diagnostic: Compares whether the proportion of healthy subjects have a negative test result

Other Analytical ConsiderationsMinimize false positives

After 30 cycles, DNA is amplifiedover a billion fold

• Assess target homology with like organisms (Gen Bank)• Monitor for sources of contamination

• Crossover contamination – Positive to negative – Amplified nucleic acids

• Amplicon contamination of reagents, laboratory surfaces, ventilation ducts, and skin, hair, clothing

• Incorporate contamination controls• Facility• Equipment• Procedural• Personnel

Other Analytical ConsiderationsMinimizing False Negatives

• Monitor for changes in target sequence that would reduce primer and probe recognition

• Purification and concentration of nucleic acid to remove inhibitors and increase template concentration

• Sample dilution to detect inhibitors

• Detection: Use of proper controls• Target specific positive controls (extraction and PCR setup)• Internal controls (ie. human β actin gene)

– Control for adequate specimen collection, proper extraction, PCR reaction setup

• Inhibition controls (ie. 16S or RNP)

Verification to Validation• Validation is the documentation that a previously verified test can

repeatedly give the expected results• Quality control and quality assurance• Personnel competency assessment• Internal and external proficiency testing• Correlation with clinical findings

• Validation studies must be performed using the testing protocol in its final version including all reagent specifications

• Validation studies should be documented and approved – SME– QA Officer – Laboratory Director

Designing a Validation Study• Establish number and type of specimens to be tested (agent dependent)

– Stock isolates/cultures, clinical samples, nucleic acid– Minimum of 20 positive clinical specimens and 50 negative

(Novel assay: 50/100)• Establish acceptance criteria

– CLIA does not specify– Rule of thumb: 95% + agreement

• depends on the analyte and type of test (screening vs confirmatory)• Calculate accuracy (sensitivity/specificity), precision, reportable range

• How to resolve discrepancies– Sequence or characterize products (ie. PCR assays)– Compare to clinical presentation – Refer to a reference laboratory

• Verification/Validation Proposal – reviewed and approved

Validation Study

Proposal---

Final Validation

Report

Quality Assurance Program Supports ValidationVirginia’s Molecular Testing Program

• Pre and Post Analytical– Employee training– Corrective Actions– PM (daily, weekly, monthly, yearly)– SOPs– Internal and External PT– Peer review of all data on worksheets or

computerized

• Analytical– Assay verification and validation– Testing worksheets to document reagents, lot

numbers, and equipment – Controls (Pos/Neg/Internal/External) – Data Analysis and Interpretation

New Employee Orientation Training Form and Molecular Laboratory Access

• Documents training elements

• Permanent employee record

• Focuses and documents safety and technical training overview

• Covers the basics - all new employees regardless of education and competency receive core training and information

Analytical TrainingPersonnel

• New Employee and Molecular laboratory orientation

• Safety review• Instructional training

– Review of core literature; theory– Review of SOP – Partner with competent scientist to

observe procedure – New employee observed while testing

(time ranges with expertise of individual)• Independent testing (time ranges with

expertise of individual)• Internal competency assessment• Blinded proficiency testing

Employee Competency Assessments

Annual Monitoring

• Annual review of procedures and SOPs• Annual PT Testing/Competency Assessments

– In house developed and/or external• Blinded QC testing (random samples selected and retested)• Written documentation of errors and corrective actions (internal

and external) – Re-training or direct observation– Improvements in method to minimize chance of repeat error– Continued errors could result in suspension or termination

Corrective Actions

• It’s NOT about assigning blame• It’s about documenting what has

occurred; how it was investigated; and what measures have been put in place to “hopefully” prevent it from occurring again

• Mistakes affect everyone• Errors can be an indication of a

larger problem– Submitter problem– Reagent issues– Personnel issues– Equipment problems– Workflow problems

Procedural Documentation

Detailed worksheets– Check lists– Pre-calculations– Lot # and kit # tracking– All worksheet results

reviewed upon completion – Documentation of all

procedural changes or issues

– Results entered into computer system and reviewed by second scientist

Contamination ControlProcedural

• Amplicon containment– Chemical and enzymatic methods

for inactivating PCR amplicons– Closed-system PCR reactions

liberation of amplicons through broken reaction vessels can still lead to contamination events

• Reagents– Standardize all reagents– Aliquot all reagents– Primer/Probe

documentation/Verification

Contamination ControlEquipment

Dedicated devices– PCR workstations– Robotic extraction

equipment– Pipettes (calibrated)– Centrifuges– Aerosol barrier tips– Gloves (change frequently

& between samples)– Closed PCR systems– Disposable lab coats– Pens

Contamination controls can reduce but not eliminate contamination

Contamination ControlFacility

Extraction(BSCs )Clean Room

(reagents &PCR setup)

HVAC

HVAC +

Post Amplification

AnteroomSTART

HVAC -

HVAC ++

Physical separation of pre and postamplification workspaces

Pre amplification• Ante-room • Extraction room• Automation• Clean room (reagents &

setup)Post amplification

• Negative pressure • Single passage air vented to

the outsideUnidirectional workflowRestricted accessBio-seal roomsRemote access to real time PCRcyclers

Difficulties with Verification and Validation of Molecular Assays

• Lack of appropriate or available samples for verification (ie. Pertussis, NV, A/H5, Cyclospora)

• Often measure a different target from traditional gold standard methods (phenotypic versus genotypic)

• More sensitive than gold standard methods • Issues of false positivity/contamination versus increased sensitivity

• Absence of external proficiency testing programs • Planning for all sample types or conditions that may be received in

emergency or outbreak situations (IMPOSSIBLE – next slide)• Assay turnover or continual assay improvements/modifications

• As soon as an assay is validated; a new version or change is released• CLIA regulations state WHAT must be done, but don’t give any specifics

on HOW it must be done or how to handle validation limitations

Issues Preventing Labs from Conducting Appropriate Assay Verifications and Validations

• COST!!• Lack of personnel to conduct studies or

to oversee validation process• Lack of Time• Too many assays to implement and

validate and many have unique considerations

• Notion that standard methods from federal agencies have already been verified and validated or do not require additional evaluation

• Too much to monitor• Difficult to know where to begin• Lack of clear public health guidance

Public Health ExceptionsIts not all diagnostic….

• Emergency and outbreak response testing is a priority to public health; “official” verification and validation is not always possible

• LRN assays• Challenging or new matrices• New pathogens (WNV, SARS, Influenza A/H5)

• Public health surveillance testing is NOT diagnostic (Salmonellaserotyping)

• Genotyping assays are NOT diagnostic (PFGE, spoligotyping, MLVA)• Disclaimers

“This test was developed and its performance characteristics determined by (laboratory name). It has not been cleared or approved by the U.S. Food and Drug Administration”

Considerations for Assays of Public Health Significance

• If rapid deployment of assays are necessary for the benefit of the public’s health and/or reference or control materials are not available, public health labs:

– May receive temporary exemption from CLIA requirements for establishing performance characteristics

• Split samples to be tested in reference lab• Confirm positive results using alternate assay, if possible

– Should review and reference validation data from assay developer (ie. CDC)

– Should use competent and trained personnel to perform testing– Must perform testing exactly as written in method

• Examples: Influenza A/H5, Mumps outbreak, Pertussis outbreak, Agents of terrorism, SARS, WNV

Summary• Verification and Validation of methods is extremely important in public health

molecular biology laboratories• Major goal is to reduce false positives and negatives– contamination control

• Contamination control of Facility, Equipment, Personnel, Procedures with abundant controls

• Eliminate or overcome inhibitors • Assess reagent and specimen/sample quality

• Continual assessment of assay and personnel performance• External and internal audits• Participation in training, continuing education and proficiency programs

• Verification and validation studies and QA needs to be balanced with laboratory resources; all cost money and time

• Start simple but start somewhere• Don’t reinvent the wheel• Use your resources and colleagues

References

• August MJ, Hindler JA, Huber TW, Sewell DL. Cumitech 3A. Quality Control and Quality Assurance Practices in Clinical Microbiology. American Society for Microbiology (1990)

• Elder BL, Hanson SA, Kellogg JA, Marsik FJ, Zabransky RJ. Cumitech 31. Verification and validation of Procedures in the Clinical Microbiology Laboratory (1997)

• Molecular Diagnostic Methods for Infectious Diseases; Approved Guideline – Second Edition. CLSI. MM3-A2, Vol 26, No.8.

• Proficiency Testing (External Quality Assessment) for Molecular Methods; Approved Guideline – CLSI. MM14-A, Vol 25, No. 24.

• Centers for Disease Control and Prevention, MMWR Weekly Report, Regulations for Implementing the Clinical Laboratory Improvement Amendments of 1988: A summary, February 28, 1992 Vol 41 (RR-2)

• FDA’s CLIA homepage: www.fda.gov/cdrh/clia/index.html• CLIA regulations: www.phppo.cdc.gov/clia• NCCLS: www.nccls.org• CAP molecular checklist: www.cap.org/html/checklist_html/cklst_mol.html

Denise M. Toney, Ph.D.Commonwealth of VirginiaDivision of Consolidated Laboratory Services600 North 5th StreetRichmond, Virginia 23219804-648-4480 ext. 282denise.toney@dgs.virginia.gov

Questions??