1

HIV Drug Resistance Training

Module 7:

HIV Genotyping Assay Validation

2

Topics

Getting Ready to Validate Validation Concepts Validation Procedures

3

Objectives

Describe principles behind validation procedures.

Identify the procedures needed to gain confidence that the results are accurate and reproducible.

Given specific laboratory, customize procedures.

4

getting ready to validate

Why is it important to validate assays?What should you do to prepare for validation?When do you need to validate?

5

Validation of Genotyping Assay Performance

Helps ensure quality results Easier to compare results from labs using

different methods To establish performance characteristics

used to compare assays Required prior to testing any samples for

WHO Validation is assay-specific

– For instance, plasma vs. DBS sensitivity could be different, and extractions steps are not shared, so these methods should be validated separately.

6

Minimum Requirements

Minimal requirements for an assay validation to establish/confirm analytical performance characteristics:– Sensitivity– Specificity– Accuracy– Precision– Reproducibility

7

Preparation: Procedures

Standard Operating Procedures (SOPs) – For what?– How?

Quality control (QC) SOPs– For what?

Personnel?

8

Preparation: Validation Protocol

Written validation protocol finalized before results are analyzed– No changes to protocol allowed once validation

experiments started Determine acceptance criteria for each

section in advance Validation performed under the same

conditions (facility, equipment, reagents, personnel) that will be used for real samples

9

When to Repeat Validation Procedures

If there is failure to pass any criteria:– Document corrective action and re-validate– Do NOT simply repeat experiments with same

assay If procedure changes:

– Minor changes equivalency testing– Major changes re-validation for in-house

assays:• Primer modifications and replacement• Change RT and/or PCR enzymes

– See module 11, QA/QC

10

Discussion

Why is it important to validate assays? What should you do to prepare for

validation? When do you need to validate?

11

validation concepts

What criteria are we measuring against?

12

Components of Genotyping Assay Validation

Accuracy Precision/Reproducibility Sensitivity for RT-PCR amplification Sensitivity for detection of minority variants Specificity

13

Accuracy Definition

Are “known” mutations detected?

14

Accurate Inaccurate

Accuracy: How close do we get to the expected (target) value or result.

15

Accuracy Discussion

Best tested using: – clonal virus or lab constructs (e.g. site-directed

mutants)– very well-characterized, non-clonal sample (e.g.

artificial 50/50 mixtures of clones) Test for:

– “Resistance-associated” mutations– Likely to be frequently encountered

Pro: Usually the easy to measure Con: Difficult to define criteria ahead of

time for very new assays

16

Precision Definition

Ability to generate the same result on multiple aliquots of the same sample within a test run (intra-assay variability) – Fewer samples (e.g. 3 to 5), more replicates

(e.g. 5 to 10)

17

Precise(although inaccurate!)

Imprecise(although sometimes accurate)

Precision: How close are all the results to each other?

18

Accuracy and Precision

The degree of fluctuation in the measurements is indicative of the “precision” of the assay.

The closeness of measurements to the true value is indicative of the “accuracy” of the assay.

Quality Control is used to monitor both the precision and the accuracy of the assay in order to provide reliable results.

19

Precise: NoAccurate: No

Accuracy and Precision

Precise: YesAccurate: No

Precise: YesAccurate: Yes

20

Reproducibility

Ability to generate the same result on multiple aliquots of the same sample in different test runs (inter-assay variability)– More samples (e.g. up to 40), fewer replicates (2

or 3)– Over time (e.g. 2 weeks)– Between operators– Among lots of critical reagents

21

Reproducibility: When conditions change, how does that affect the

results?

22

Reproducibility and Precision Criteria

Based on nucleotide sequence identity in:– Pairwise comparisons or– Comparison to a consensus sequence

Mismatches may be considered a difference (if “compatible” e.g. A and R, or R and D, but not A and Y, or M and K)– Acceptance criteria may depend on complexity

of samples tested Based on amino acid sequence identity

– “resistance-associated” mutations can be analyzed separately

23

Critical Reagent Lot-to-lot Variability is a Component of Assay Reproducibility

Reagent vendors test products (e.g. PCR enzymes) using procedures different from HIVDR genotyping

Cannot assume that performance is the same, between lots

Incorporate at least 2 lots of critical reagents in validation experiments

Perform lot-release testing as part of routine QC (see module 11)

24

Sensitivity: Amplification

Definition: What is the minimum viral load required to generate an accurate result?

Considerations:– Importance placed on multiple subtypes varies

with intended application– Overlaps with linearity if samples also

genotyped and sequences compared – May or may not involve probing multiple variant

detection (mixtures) at low VL

25

Sensitivity: Detection of Minority Variants

In samples with mixtures What % of the pool must minor variant be

in order to be reliably detected?– Can separate out post-amplification steps (e.g.

by mixing plasmids and sequencing vs. mixing viruses and amplifying then sequencing)

– In reality, affected by viral load, though this is not often highlighted

26

Dependence of Sensitivity of Detection of Minor Variants and Input Viral Load

Assume that 200 µl plasma used for RNA extraction, 25% used for RT-PCR; RT successful for 20% of RNA molecules; minority variant present at 10% of total

Viral load (copies/ml)

RNA copies in RT rxn

Amplifiable genomes

Copy no. (minor variant)

100,000 5,000 1000 100

10,000 500 100 10

1000 50 10 1

27

Discussion

What criteria are we measuring against?

28

validation procedures

How can we set up validation procedures for our lab?

29

Validation Criteria (Example)

Accuracy– 100% of known mutations must be detected

Precision/Reproducibility– ≥ 90% of pairwise comparisons must be at least

98% identical, mixtures counted as difference Sensitivity for amplification

– ≥ 95% of samples with viral loads between 500 and 1000 copies/ml must be amplified and successfully genotyped (n ≥ 10)

Sensitivity for detection of minority variants– defined as the lowest % at which a mutation is

detected in > 50% of replicate tests

30

Specimen Selection

Specimens used in the validation experiments should be chosen to test the specific aspect of assay performance (accuracy, reproducibility, etc.)

Especially for precision and reproducibility, chose specimens that are similar to those that will be tested routinely (i.e. specimen type, genetic subtype, viral load range)

Specimens should be well-characterized in pre-validation experiments before starting the actual validation

31

Validation Procedure Customization

Details of the variables to be considered are likely to be different in each lab– Physical set-up– Personnel (number of qualified analysts)– Degree of automation

Validation protocols can be reviewed in consultation with other accredited labs or with WHO staff

32

Discussion

How can we set up validation procedures for our lab?

33

Reflection

What do we need to do to make sure we are correctly validating our assays?

34

Summary

Getting Ready to Validate Validation Concepts Validation Procedures