this project has been funded in whole or in part with federal funds from the division of aids...
TRANSCRIPT
![Page 1: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/1.jpg)
This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases, National Institutes of Health, Department of Health and Human Services, under contract No. HHSN272201200009C, entitled NIAID HIV and Other Infectious Diseases Clinical Research Support Services (CRSS).
Verification of Performance Specifications
An Advanced View of Method Validation
Version 5.0, August 2012
![Page 2: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/2.jpg)
2
Identify test classifications Define what each validation experiment details for
testing methods Discuss what is recommended to perform each of the
validation experiments for testing methods Recognize how to evaluate data obtained from each of
the validation experiments
Objectives
![Page 3: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/3.jpg)
3
A rapid Human Immunodeficiency Virus (HIV) test would likely be classified as a:
A. High complexity, modified assayB. Moderate complexity, unmodified assayC. Food and Drug Administration (FDA)-approved,
modified assayD. Waived, FDA-approved, unmodified assay
Pre-Assessment Question #1
![Page 4: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/4.jpg)
4
The precision of a test method gives information related to the method’s:
A. Systematic errorB. Comparison of results to a reference methodC. ReproducibilityD. Likelihood of being affected by hemolysis, lipemia and
icterus E. Both A and B
Pre-Assessment Question #2
![Page 5: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/5.jpg)
5
When transferring reference intervals of 20 specimens used, what is the minimum number that must fall within manufacturer’s reference intervals?
A. 20B. 18 C. 16D. 15
Pre-Assessment Question #3
![Page 6: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/6.jpg)
6
Which linear regression equation component gives information regarding constant bias?
A. yB. xC. m (slope)D. b (intercept)
Pre-Assessment Question #4
![Page 7: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/7.jpg)
7
Selecting a Method
Evaluate diagnostic tests Characteristics of testing
methods References: Technical literature
and manufacturer’s information Select method of analysis Validate method performance Implement method Perform tests with appropriate
Quality Control (QC) and External Quality Assurance (EQA)
![Page 8: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/8.jpg)
Method Validation
Why must we validate?
When should we validate?
What should we validate?
8
What is method validation?
![Page 9: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/9.jpg)
9
Why is validation important? Division of Acquired Immunodeficiency Syndrome
(DAIDS) requirement How important is it that the results produced by the
testing method are reliable? Shouldn’t the laboratory know the level of performance
of an adopted test method?
Method Validation (cont’d)
![Page 10: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/10.jpg)
Tests to Validate
Waived
Non-waived
• Unmodified FDA-approved
• Modified and/or Non-FDA-approved
10
![Page 11: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/11.jpg)
11
Vendor Publications http://www.fda.gov/MedicalDevices/
ProductsandMedicalProcedures/InVitroDiagnostics/LabTest/ucm126079.htm
FDA Approval Resources
![Page 12: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/12.jpg)
12
What would you consider to be the complexity, per Clinical Laboratory Improvement Amendments (CLIA), of the glucose assay in the workbook?
A. WaivedB. ModerateC. High
Skill Check
![Page 13: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/13.jpg)
13
What would you consider to be the complexity of a rapid urine pregnancy assay?
A. WaivedB. ModerateC. High
Skill Check
![Page 14: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/14.jpg)
14
What would you consider to be the complexity of performing a manual white cell differential using a stained whole blood smear?
A. WaivedB. ModerateC. High
Skill Check
![Page 15: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/15.jpg)
15
Before you begin: Be sure you are familiar with the test method before
starting Know what to expect from the method (package insert,
discussions with technical assistance, and field service representatives)
Do not include results outside of stated reportable ranges
Predict your findings; establish limits/evaluation criteria
Method Validation
![Page 16: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/16.jpg)
16
Terms for Discussion
Central Tendency
Dispersion
![Page 17: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/17.jpg)
17
Terms for Discussion (cont’d)V
alue
s
Run
![Page 18: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/18.jpg)
18
Some error is expected Examples
Error must be managed Understanding Defining specifications of allowable error Measurement
Error in Test Methods
![Page 19: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/19.jpg)
19
Total Error of Testing System
Total Allowable Error• CLIA Guidelines per analyte• Other Guidelines
Systematic Error
Random Error Total Error
![Page 20: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/20.jpg)
20
Error Assessment
In either direction,
unpredictable
In one direction, cause results to be high or low
Combined effect
Systematic Error
(SE)
Random
Error
(RE)
Total Error
(TE)
![Page 21: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/21.jpg)
21
Low End Performance Standards Recommendations derived from upper portion of
reportable range are more difficult to achieve at lower concentrations
Maximum Total Error Allowed Considered to be 30% by David Rhoads, except for
amplification methods
Total Error Considerations
![Page 22: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/22.jpg)
22
Systematic Error Slope/Proportional error Intercept/Constant error Bias
Random Error Mean Standard deviation (SD) Coefficient of variation (CV)
Systematic and Random Errors
![Page 23: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/23.jpg)
23
Tools for Use
Spreadsheets
with
calculationsValidation
Software (Westgard, Analyze-
It, EP Evaluator)
Statistical
calculators,
graph paper
Data-Crunching
Tools
![Page 24: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/24.jpg)
24
One quantitative test taken through the validation process
One qualitative method taken through the validation process
How We Will Work Through This Module
![Page 25: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/25.jpg)
Reportable Range
Precision
Accuracy
Reference Intervals
Sensitivity
Specificity
Elements of Validation
25
![Page 26: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/26.jpg)
Repeat testing over short and long term (one day and 20 days, respectively)
20 samples of same material (typically two levels; e.g., Glucose at 50 and 300 mg/dL)
Standard solutions Control materials Pools (short term only)
Precision Definition: Reproducibility Gives information related to random error
Introduction
What is needed
How we perform
the testing
26
![Page 27: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/27.jpg)
27
Precision: How We Evaluate the Data
Mean Standard deviation (SD) Coefficient of Variation (CV)
Short term: 0.25 of allowable total error Long term: 0.33 of allowable total error
Calculate the following:
What amount of random error is allowable, based on CLIA criteria?
![Page 28: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/28.jpg)
28
Link for: Clinical Laboratory Improvement Amendments (CLIA) College of American Pathologists (CAP) Royal College of Pathologists of Australasia (RCPA) Others
http://www.dgrhoads.com/db2004/ae2004.php
Allowable Total Error Database
![Page 29: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/29.jpg)
29
Precision: Levey-Jennings (LJ) ChartsV
alue
s
Run
![Page 30: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/30.jpg)
30
Precision: How We Evaluate the Data
Mean SD CV: More commonly used, allows for
easier comparison
How do we compare to manufacturer’s data?
![Page 31: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/31.jpg)
31
Precision Example
Mean of Level 1 Glucose
CLIA Total Allowable Error
Total Allowable Error Level 1 Glucose
Random error allowed:
90 mg/dL
6 mg/dL or ± 10%
0.1 x 90 = 9 mg/dL
0.25 x total allowable
0.25 x 9 mg/dL
2.25 mg/dL
0.33 x total allowable
0.33 x 9 mg/dL
2.97 mg/dL
Long-term precision
Short-term precision
![Page 32: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/32.jpg)
32
Work with Levey-Jennings graph and data Work with mean and standard deviation to calculate a
coefficient of variation, as well as a mean and a coefficient of variation to calculate a standard deviation
Determine if precision data is acceptable
Activity
![Page 33: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/33.jpg)
Accuracy Definition: How close to the true value Comparison of methods Gives information related to systematic error Potential conflicts on interpretation of results
(reference values)
Introduction
40 different specimens Cover reportable range of method Quality versus quantity
What is needed
Duplicate measurements of each specimen on each method
Minimum of five days, prefer over 20 (since replicate testing is same)
How we perform the
testing
33
![Page 34: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/34.jpg)
Accuracy: How We Evaluate the Data
Graph the Data:
Test method on Y-axis
Reference (comparative) method on X-axis
Shows analytical range of data, linearity of response over range and relationship between methods
Real time Difference plot
Comparison plot Calculate
y = mx + b
b represents constant error
m represents proportional error
34
![Page 35: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/35.jpg)
35
Visual Inspection for AccuracyTe
st M
etho
d
Reference Method
Intercept
(x1, y1)
(x2, y2)Slope = (y2- y1) / (x2- x1)
![Page 36: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/36.jpg)
36
Slope: Usually not significantly different from 1 Intercept: Not significantly different from 0 Significant difference with Medical Decision Points
Accuracy: How We Evaluate the Data
![Page 37: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/37.jpg)
37
Slope Measure of proportional bias
m = (y1-y2)/(x1-x2) or “rise/run” Slope greater than 1 means the Y (Test) values are
generally higher than the X (Comparative) values Slope of 1.11 means the Y (Test) values are on
average 11% higher than the X (Comparative) values
Calculate Appropriate Statistics
![Page 38: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/38.jpg)
38
Intercept of the Line Measure of constant bias between two methods
Y (Test) value at the point where the line crosses the Y axis
If Y intercept is 12, then all Y (Test) values are at least 12 units higher than the X (Comparative) values
Calculate Appropriate Statistics (cont'd)
![Page 39: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/39.jpg)
39
Accuracy
What type of bias do you see?
![Page 40: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/40.jpg)
Accuracy (cont’d)
Constant Bias Proportional Bias
40
![Page 41: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/41.jpg)
41
Can a linear regression formula offer predictive value in relation to method comparisons?
A. YesB. No
Skill Check
![Page 42: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/42.jpg)
42
Create graph based on sample set Determine slope from best-fit line Determine Y-intercept from best-fit line Explain the relationship between comparative and test
results
Activity
![Page 43: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/43.jpg)
CLSI recommends four measurements of each specimen; three are sufficient
Series of samples of known concentrations (e.g., standard solutions, EQA linearity sets)
Series of known dilutions of highly elevated specimen or spiked specimens; EQA specimens
At least four levels (five preferred)
Reportable Range / Linearity Definition: Lowest and highest test results that
are reliable Especially important with two point calibrations Analytical Measurement Range (AMR) and
derived Clinical Reportable Range (CRR)
Introduction
What is needed
How we perform the
testing
43
![Page 44: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/44.jpg)
44
Reportable Range:How We Evaluate the Data
Measured values on Y-axis versus Known or assigned values on X-axis
Plot mean values of:
Compare with expected values (typically provided by manufacturer)
Visually inspect, draw best-fit line, estimate reportable range
![Page 45: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/45.jpg)
45
Reportable Range Activity
AssignedValue
Experimental Results
Average Rep #1 Rep #2 Rep #3 Rep #4
10.0 ____ 11.0 10.0 11.0 10.0
100.0 ____ 99.0 103.0 103.0 101.0
300.0 ____ 303.0 305.0 304.0 306.0
500.0 ____ 505.0 506.0 505.0 506.0
800.0 ____ 740.0 741.0 744.0 742.0
![Page 46: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/46.jpg)
46
Reportable Range Activity (cont'd)
AssignedValue
Experimental Results
Average Rep #1 Rep #2 Rep #3 Rep #4
10.0 10.5 11.0 10.0 11.0 10.0
100.0 101.5 99.0 103.0 103.0 101.0
300.0 304.5 303.0 305.0 304.0 306.0
500.0 505.5 505.0 506.0 505.0 506.0
800.0 741.8 740.0 741.0 744.0 742.0
![Page 47: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/47.jpg)
Reportable Range Activity (cont'd)
Linearity Scatter Plot
0.0
100.0
200.0
300.0
400.0
500.0
600.0
700.0
800.0
0 100 200 300 400 500 600 700 800 900
Assigned Concentrations (units)
Reco
vere
d V
alu
es (
Mean
s)
47
![Page 48: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/48.jpg)
AMR vs. CRR
Analytical Measurement Range (AMR)
Linearity
Clinically Reportable Range (CRR)
Allows for dilution or other preparatory steps beyond routine
48
![Page 49: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/49.jpg)
49
If you do not have enough specimen to perform a dilution, upon which reportable range component must you rely?
A. AMRB. CRRC. Neither A or BD. Both A and B
Skill Check
![Page 50: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/50.jpg)
50
Utilizing the marketing materials from the two chemistry linearity kits in your handouts:
1. Determine which kit would be more appropriate for use with the chemistry assay you chose earlier
2. Explain your reasoning
Linearity Materials
![Page 51: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/51.jpg)
51
Given your choice of linearity kits, you perform your AMR experiments by performing four replicates of each level of known concentration solution. The data you obtain is displayed on the next slide.
1. Review data; record any initial observations2. Graph data on supplied graph paper3. Determine your assay’s AMR
Graph Activity
![Page 52: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/52.jpg)
Level Rep 1 Rep 2 Rep 3 Rep 4
1 24 23 25 24
2 196 197 171 194
3 359 360 358 361
4 530 532 529 535
5 700 695 702 709
Linearity Experiment Results
52
![Page 53: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/53.jpg)
Activity
Using an Excel spreadsheet, create a graph and calculate
linear regression statistics from the data provided
53
![Page 54: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/54.jpg)
54
Rep 1 Rep 2 Rep 3 Rep 4Lab's
AverageKnownConc
24 23 25 24 24 25
196 197 171 194 195.7 200
359 360 358 361 360 375
530 532 529 535 532 550
700 695 702 709 702 725
![Page 55: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/55.jpg)
55
0
100
200
300
400
500
600
700
800
0 200 400 600 800
Re
cove
red
Known Concentration
AMR Verfication
![Page 56: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/56.jpg)
56
Your medical director, in consultation with clinicians, determines that for proper study participant care the Clinically Reportable Range (CRR) for glucose is15 – 1400 mg/dL
Given your linearity experiment results and the package insert, devise a dilution protocol to be contained within our Glucose SOP
Dilution Protocols
![Page 57: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/57.jpg)
57
Given your AMR, CRR, and dilution protocol, how would you handle the following analyzer results?
1. 12 mg/dL2. 800 mg/dL3. 1600 mg/dL
Reportable Results
![Page 58: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/58.jpg)
58
Reference Intervals
Definition: Normal range in healthy population Used for diagnosis/clinical interpretation of
results
Introduction
Pre-defined “normal” criteria for screening purposes
Transferring: 20 “normal” individuals’ specimens Establishing: 120 “normal” individuals’ specimens
What is needed
Perform testing on all samples Document results
How we perform the
testing
![Page 59: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/59.jpg)
Transferring Establishing
18 of 20 must fall within manufacturer’s ranges
Calculate mean and SD of data for each group
Reference Intervals = mean ± 2 SD (if Gaussian Distribution only, otherwise, additional calculations recommended)
59
Reference Intervals:How We Evaluate the Data
![Page 60: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/60.jpg)
60
Activity
Determine if assay is eligible for transference of reference intervals
Review a sample set of data to determine if transference may be performed; if not, determine next step(s)
![Page 61: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/61.jpg)
Sensitivity
Definition: Lowest reliable value; lower limit of detection, especially of interest in drug testing and tumor markers
Different terminologies used by different manufacturers
Introduction
Blank solutions Spiked samples
What is needed
20 replicate measurements over short or long term, depending on focus
How we perform the
testing
61
![Page 62: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/62.jpg)
Sensitivity: How We Evaluate the Data
Lower Limit of Detection (LLD):
Mean of the blank sample, plus two or three SD of blank
sample
Biological Limit of Detection:
LLD plus two or three times SD of
spiked sample with concentration of detection limit
Functional Sensitivity:
Mean concentration for spiked sample whose CV = 20%; lowest limit where quantitative data is
reliable
Three methods used:
62
![Page 63: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/63.jpg)
63
Activity
Using the manufacturer’s package inserts, find the
related information for sensitivity. How was it
calculated?
![Page 64: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/64.jpg)
Specificity Definition: Determination of how well a method
measures the analyte of interest accompanied by potential interfering materials
Introduction
Standard solutions, participant specimens or pools
Interferer solutions (standard solutions, if possible; otherwise, pools or specimens) added at high concentrations
What is needed
Duplicate measurementsHow we
perform the testing
64
![Page 65: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/65.jpg)
Specificity: How We Evaluate the Data
Tabulate results for pairs of samples (dilution and interferent)
Calculate means for each (dilution and interferent)
Calculate the differences Calculate the average interference
of all specimens tested at a given concentration of interference
65
![Page 66: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/66.jpg)
66
Compare diagnosis Assume comparative (reference) method is accurate Determine the following:
True Positives, True negatives False Positives, False negatives
Calculate sensitivity and specificity and compare to manufacturer
Qualitative Assays
![Page 67: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/67.jpg)
67
Negative and Positive Quality Controls Use QC materials recommended by manufacturer for
verification purposes Determine validity of other results, e.g., method
comparisons Evaluate failed runs if they occur during verification
process
Qualitative Assays: Control of Validation
![Page 68: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/68.jpg)
68
How is it performed? Runs of specimens with analyte concentrations near
the cutoff point Three specimens, one at cutoff, one just below cutoff,
and one just above cutoff (± 20% recommended) Replicate measurements of each of three specimens
(20 each, minimum) How is it evaluated?
Determine percentage of positives and negatives for each specimen
Evaluate cutoff, as well as other two specimens
Qualitative Methods: Precision
![Page 69: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/69.jpg)
69
How is it performed? Specimens typical of population (to be tested in future
use of method) 50 positive specimens and 50 negative specimens
recommended; minimum 20 each Performed over 10 to 20 days
How is it evaluated? Discrepant results near cutoff? Most often sensitivity and specificity used to describe
performance
Accuracy/Method Comparisons
![Page 70: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/70.jpg)
70
Qualitative Methods
Comparative or ReferenceMethod Result
Positive Negative
Test Method Result
Positive True Positive False Positive Positive Predictive Value
Negative False Negative True NegativeNegative
PredictiveValue
Sensitivity Specificity
False Positive Rate - False Positives divided by total number of Negatives
False Negative Rate - False Negatives divided by total number of Positives
True vs. False
![Page 71: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/71.jpg)
71
Qualitative Methods (cont'd)
Comparative or ReferenceMethod Result
Positive Negative
Test Method Result
Positive True Positive False Positive Positive Predictive Value
Negative False Negative True NegativeNegative
PredictiveValue
Sensitivity Specificity
Sensitivity = 100 x True Positives divided by (True Positives + False Negatives)
Specificity = 100 x True Negatives divided by (True Negatives + False Positives)
![Page 72: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/72.jpg)
72
Qualitative Methods (cont'd)
Comparative or ReferenceMethod Result
Positive Negative
Test Method Result
Positive True Positive False Positive Positive Predictive Value
Negative False Negative True NegativeNegative
PredictiveValue
Sensitivity Specificity
Predictive Values - Operation of a test on a mixed population of Positive and Negatives
A property of the test and the population; and affected by prevalence of Positives
Positive Predictive Value = True Positives divided by (True Positives + False
Positives) Negative Predictive Value = True Negatives divided by
(True Negatives + False Negatives)
![Page 73: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/73.jpg)
73
High Diagnostic Value 100% Sensitivity 100% Specificity
What happens if True Positive rate is equal to the False Positive rate?
Evaluation Criteria
![Page 74: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/74.jpg)
74
Activity
Estimate sensitivity and specificity of a qualitative method given a data set.
![Page 75: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/75.jpg)
75
Activity (cont’d)
Create a validation plan for a quantitative assay to be performed in your laboratory.
![Page 76: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/76.jpg)
76
Now that you have completed this module, you should be able to:
Identify test classifications Define what each validation experiment details for
testing methods Discuss what is recommended to perform each of the
validation experiments for testing methods Recognize how to evaluate data obtained from each of
the validation experiments
In Closing
![Page 77: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/77.jpg)
77
A rapid HIV test would likely be classified as a:
A. High complexity, modified assayB. Moderate complexity, unmodified assayC. FDA-approved, modified assayD. Waived, FDA-approved, unmodified assay
Post-Assessment Question #1
![Page 78: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/78.jpg)
78
The precision of a test method gives information related to the method’s:
A. Systematic errorB. Comparison of results to a reference methodC. ReproducibilityD. Likelihood of being affected by hemolysis, lipemia and
icterusE. Both A and B
Post-Assessment Question #2
![Page 79: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/79.jpg)
79
When transferring reference intervals of 20 specimens used, what is the minimum number that must fall within manufacturer’s reference intervals?
A. 20B. 18C. 16D. 15
Post-Assessment Question #3
![Page 80: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/80.jpg)
80
Which linear regression equation component gives information regarding constant bias?
A. yB. xC. m (slope)D. b (intercept)
Post-Assessment Question #4
![Page 81: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/81.jpg)
81
DAIDS Good Clinical Laboratory Practice (GCLP) Guidelines. www.westgard.com. Validation of Qualitative Methods. 42 CFR § 493.1253. College of American Pathologists Commission on Laboratory Accreditation,
Accreditation Checklists, April 2006. Westgard, James O. Basic Method Validation 2nd Edition. Madison, WI: Westgard
QC, Inc., 2003. Clinical and Laboratory Standards Institute. User Protocol for Evaluation of
Qualitative Test Performance; Approved Guideline. NCCLS document EP12-A. Clinical and Laboratory Standards Institute, Wayne, PA USA, 2002.
Clinical and Laboratory Standards Institute. Evaluation of Precision. Performance of Quantitative Measurement Methods. NCCLS document EP5-A2.
Clinical and Laboratory Standards Institute, Wayne, PA USA, 2004. Clinical and Laboratory Standards Institute. User verification of Performance for
Precision and Trueness. CLSI document EP15-A2. Clinical and Laboratory Standards Institute, Wayne, PA USA, 2005.
References
![Page 82: This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649c765503460f94929e82/html5/thumbnails/82.jpg)
Wrap Up
82