STATISTICAL APPROACH TO MEDICAL DEVICE VERIFICATION AND VALIDATION

Yes, there is going to be some math (but not much)

Medical Device Verification and Validation In the medical device world, verification and

validation are the backbone of design control This is how you prove you have produced

what you said you would and what the customer wants! It’s also required

What The FDA Says About Design Verification/Validation 21CFR820: QUALITY SYSTEM REGULATION Sec. 820.3 Definitions Verification means confirmation by examination and

provision of objective evidence that specified requirements have been fulfilled

Design verification. Each manufacturer shall establish and maintain procedures for verifying the device design. Design verification shall confirm that the design output meets the design input requirements. The results of the design verification, including identification of the design, method(s), the date, and the individual(s) performing the verification, shall be documented in the DHF

What The FDA Says About Design Verification/Validation 21CFR820: QUALITY SYSTEM REGULATION Sec. 820.3 Definitions Validation means confirmation by examination and

provision of objective evidence that the particular requirements for a specific intended use can be consistently fulfilled

Design validation means establishing by objective evidence that device specifications conform with user needs and intended use(s)

What The FDA Says About Design Verification/Validation 21CFR820: QUALITY SYSTEM REGULATION Sec. 820.3 Definitions Design validation. Each manufacturer shall establish and

maintain procedures for validating the device design. Design validation shall be performed under defined operating conditions on initial production units, lots, or batches, or their equivalents. Design validation shall ensure that devices conform to defined user needs and intended uses and shall include testing of production units under actual or simulated use conditions. Design validation shall include software validation and risk analysis, where appropriate. The results of the design validation, including identification of the design, method(s), the date, and the individual(s) performing the validation, shall be documented in the DHF

This is NOT the same as Test Method or Process Validation

Where Does V&V Fit Into the Design Control Picture?

A Few Quick Points

Design V&V studies are NOT experiments! Y0u should know the outcome before

you start.

Require predefined acceptance criteria Which is why they are not

experiments

Before you start V&V, you should have enough characterization data to understand your system (product)

Ok… Enough Introduction

But before we get into the nerdy stuff a few hints to make it easier: Ensure you have clearly defined and measurable

product requirements Use standards when available (more in a minute) Hire a good statistician!!

Standards and Guidelines There are several good guidelines

and standards published to help CLSI publishes a variety of

guidelines for statistical analysis useable for verification studies Normally protocols based on CLSI

guidance is accepted by the FDA Common Technical Specifications

(CTS) for in vitro diagnostics medical devices These CTS requirements are

associated with the IVD Directive 2009/108/EC of the European Parliament and the Council of 3rd February 2009 on in vitro diagnostic medical devices

Standards and Guidelines

A few words on CLSI EP Protocols � Provide estimates for individual parameters (limit

of detection, precision, bias …) � CLSI does not have a standard to combine parameters

CLSI standards in many cases proscribe sample sizes and outliers �routines

Two Requirements for CLSI EP Protocols “Doable” Don’t need a Nobel prize to figure out how to do it �

“Right” As in right enough. If CLSI method deviates from a

more complicated gold standard method after the 7th decimal place, that’s ok

An EP rule The least burdensome of several alternatives will

always be used – if the minimum sample size is 40, 40 will be the only sample size used

Speaking of Sample Size…

Three ways to get sample size From a standard This is the easiest

Math Good enough

Most common questions I get (how many???)

Number of Replicates

Math for Replicate Number: SD is the standard deviation estimate based on

characterization delta is the allowable error limit n is the number of samples α is the significance level, usually 0.05 1-β is the power of the test z1-α/2 and z1-β/2 are the z values corresponding to the

probability of 1-α/2 and 1-β/2 respectively on a normal distribution table (z table)

𝒏𝒏 ≥ ��𝒛𝒛𝟏𝟏−𝜶𝜶/𝟐𝟐 + 𝒛𝒛𝟏𝟏−𝜷𝜷/𝟐𝟐� ∗ 𝑺𝑺𝑺𝑺/𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅�𝟐𝟐

Number of Replicates

Good Enough for Replicate Number: You will find that unless you have unusual

circumstances, the calculation on the last slide will normally give you a small number of replicates

Three (3) is the minimum number you need to see a possible outlier and get a good mean

Power calculations will tend to give you an answer which is more than you have time or money for

Unless y0ur system has a lot of rep to rep variability, a smaller number of reps will work Variability can be determined through good

characterization and used as justification for number of reps.

Speaking of Sample Size…

The same type of logic applies for how many samples are needed: From a standard Again, this is the easiest

Math Good enough

Speaking of Sample Size…

Math: Some definitions: Margin of Error (Confidence Interval) No sample will be perfect, so you need to decide how much

error to allow. The confidence interval determines how much higher or lower than the population mean you are willing to let your sample mean fall

Confidence Level How confident do you want to be that the actual mean falls

within your confidence interval? The most common confidence intervals are 90% confident, 95% confident, and 99% confident

Speaking of Sample Size… Math:

Confidence Level Your confidence level corresponds to a Z-score. This is a

constant value needed for this equation. Here are the z-scores for the most common confidence levels 90% – Z Score = 1.645 95% – Z Score = 1.96 99% – Z Score = 2.576

You will find this will give you pretty large sample numbers

You also have to know the SD

𝒏𝒏 =𝒛𝒛𝟐𝟐 ∗ 𝑺𝑺𝑺𝑺 ∗ (𝟏𝟏 − 𝑺𝑺𝑺𝑺)𝑴𝑴𝒅𝒅𝑴𝑴𝑴𝑴𝑴𝑴𝒏𝒏 𝒐𝒐𝒐𝒐 𝑬𝑬𝑴𝑴𝑴𝑴𝒐𝒐𝑴𝑴𝟐𝟐

Speaking of Sample Size… Good Enough “Large” vs. “Small” Sample Numbers Larger sample numbers are more likely to approach

a closer model to the total population than smaller samples numbers

Larger sample numbers are more likely to approach a “normal” distribution than small sample numbers

A rule of thumb is to have 16 degrees of freedom (D.F.) for any average standard deviation used in building a model "Understanding Industrial Designed Experiments"

(Schmidt & Launsby) This would be 17 samples

Speaking of Sample Size… "The closeness of the approximate

probability 1- alpha to the exact probability depends on both the underlying distribution and the sample size.....as the underlying distribution becomes 'less normal' (i.e., badly skewed or discrete), a larger sample size might be required to keep a reasonably accurate approximation. But, in all cases, an n of at least 30 is usually quite adequate.“ Probability And Statistical Inference 2nd Edition"

(Hogg & Tanis)

Speaking of Sample Size…

As a rule of thumb, you can consider n=30 observations or more to constitute a ‘large' sample

So… If n=30 is a large sample, and n-17 is a reasonable number to get enough degrees of

freedom…

Therefore, by using n=20 there is a compromise in dropping the theoretical value (n=30) to a value that is more practical to use in industrial settings

Speaking of Sample Size…

A few final notes on sample size If SD is known, then use z distribution.

If SD is not known: If n is large, then use z distribution. If n is small, then use t-distribution

Design Verification Examples

There are so many different types of product requirements and potential ways to verify… You really don’t want to be here all night, do you?

Here some suggestions of product requirement verifications and what elements you should have in your protocols and reports

Precision and Accuracy

These are sometimes confused You need to be sure you are measuring the

correct parameter And that parameter meets you needs and

requirements

Evaluation of Imprecision and Inaccuracy Imprecision Refers to random analytic error Lack of repeatability and/or reproducibility

Inaccuracy Refers to systematic analytic error Lack of trueness Constant Proportional

Total Error Combined error for a single result

Precision and Accuracy

How Much Error is Ok??

Performance is acceptable when Observed error < allowable error

Performance is not acceptable when Observed error > allowable error

How much error is acceptable? Based on medical and or clinical need User needs traced to product requirements which

are what you verify/validate

Protocols Should contain (at a minimum) Purpose including design input, design output being

verified Rationale for the verification design Measurable acceptance criteria. Materials required Verification Method/Procedure. Include procedure or

reference for evaluation of unexpected test results (e.g. no-tests, invalids)

Boundary values, must be assessed and included Data Analysis including validity criteria, if applicable Statistical methods with references to published

methods, if applicable Include method for handling exceptions and outliers.

Reports Should contain (at a minimum) Materials used (including part number, lot number,

expiration date, storage condition, as applicable) Equipment used and calibration status Revisions of any reference procedures Date and location of testing Individual(s) performing the verification Results including or referencing raw data Summary of procedural errors and investigation of failures

or unexpected results, if any Summary of excluded data, if any Disposition, i.e., whether or not the acceptance criteria was

met. If acceptance criteria are not met, a discrepancy report should be

initiated

Design Validation

I did not spend a lot of time (or any) on design validation In many cases, the number of sites, samples and

statistics required are dictated by Regulatory agencies Cost of external studies Availability of clinical samples Time (mainly cost and management driven)

You still have to justify what you do and have the data analyzed statistically

In Summary

You always need to have some sort of statistical justification for your studies

You need to be able to defend your justification

There are many guidelines and standards available to reference

HIRE A GOOD STATISTICIAN

Almost Last Slide

That’s All Folks!

My Contact Information: Alan Golden Manager QA, Operations

Support Abbott Molecular Alan.golden@abbott.com 224-361-7159