Handling OOS, OOT and

Unexpected Results

Karen Ginsbury PCI Pharmaceutical Consulting Israel Ltd for IFF, October 2017

What is an OOS

What is an OOT

What is an Unusual, Questionable, Atypical Result

Objective: of this course…

and BE objective when assessing OOS, OOT

Understand what an OOS, OOT, Unusual result is

Understand why Data Integrity is a BURNING hot issue

Understand what you can and can’t do and WHY

5

There is no place for emotions – put them aside

Write Down Keywords relating to OOS

Review and Comment

KG Review

Warning Letter 19 October, 2016 China

You do NOT want to go there…

You do NOT want to go there…

You do NOT want to go there…

Unofficial Testing

b. Your firm frequently performs “unofficial testing” of samples, disregards the results, and reports results from

additional tests. For example, during stability testing, your firm tested a batch sample six times and subsequently

deleted this data.

You performed “trial” sample HPLC analyses prior to acquiring the “official” analyses. The “trial” results were

subsequently discarded. “Trial” HPLC analyses were apparently run as part of the 12-month long-term stability

studies on batch #15069 for related substances. Your employee ran an HPLC analysis sequence and subsequently deleted the raw data files. Your quality control staff named the samples using the last three digits

of the batch numbers to link the "trial" injections with the official assay analyses. Your Senior Quality Control (QC)

Officer confirmed these were analyses of batch samples. Furthermore, we found that this batch was analyzed

for unknown impurities and results reported as within specifications. However, the data showed that the "trial"

injection for this batch failed the unknown impurities specification in several test runs.

Your Senior QC Officer confirmed that QC laboratory employees had frequently practiced the use of “trial”

injections at your facility. Significantly, in addition to the example above, our inspection found 5,301 deleted

chromatograms on a computer used to operate two HLPC instruments in your QC laboratory. Many of these

files were “trial” injections of batches.

Important Disclaimer

Karen Ginsbury is a consultant

Your company has approved SOPs and a quality system

Anything that Karen says is for consideration to stimulate thought and discussion and may

result in review or revision of SOPs and procedures.

AS LONG AS SOPs ARE NOT UPDATED under change control, you and your company

PERSONNEL FOLLOW THE APPROVED PROCEDURE

“Karen said” is NOT an excuse!

14

US CGMP’s

21 CFR 211.192

“Any unexplained discrepancy of the failure of a batch or any of its contents to meet any

of its specifications shall be thoroughly investigated, whether or not the batch has already

been distributed.”

“The investigation shall extend to other batches of the same drug product and other drug

products that may have been associated with the specific failure or discrepancy.”

“A written record of the investigation shall be made and shall include the conclusions and

follow-up.”

15

EU GMP’s Chapter 6: Quality Control

Laboratory Documentation should include:

A procedure for the investigation of Out of Specification and Out Of Trend results;

6.9 Some kinds of data (e.g. tests results, yields, environmental controls) should be recorded in

a manner permitting trend evaluation. Any out of trend or out of specification data should be

addressed and subject to investigation.

16

Guidance

MHRA OOS Guidance

FDA

DI Guidance

MHRA GMP

WHO

FDA Q&A

MHRA GxP

PIC/s

EMA Q&A

January 2015

September 2015

April 2016

July, 2016

August 10, 2016

August 11, 2016

ALCOA+

Accurate

Legible

Contemporaneous (real time)

Original

Attributable

19

✓Accurate

✓Complete

✓Consistent

✓Secure

STRATEGY – DEFINE, EDUCATE, COMMUNICATE

Controversial TopicsBE TRANSPARENT: NEVER delete, alter or hide data

Invalidation of results

because of laboratory error: under what conditions

Reintegration

Remeasurement, reinjection

Repreparation

Retest

Resample

Reportable result

20

Controversial Topics

Use of averaging?

Definition of reportable values?

Number of retests?

Second analyst?

Use of outlier testing?

Defining testing into compliance?

21

FDA Guidance

22

The Barr Court Case

From the New York Times

February 6, 1993, Saturday

(AP); Financial Desk

COMPANY NEWS; Judge Rules On Barr Labs

A generic drug manufacturer must recall batches of some of its medicines and stop distributingothers until the company completes studies of itsmanufacturing process, a Federal judge ruled onThursday. But United States District Judge Alfred M. Wolin refused a request by Federal pharmaceuticalregulators to order a complete shutdown

23

Barr and OOS

Faced with potential closure, the company

took FDA to court

The judge went into great details as to the

meaning and implications of OOS results

The outcome: FDA draft guidance: 1998

FDA final guidance: 2006

24

Why the Guide?

FDA’s current thinking on how to evaluate out-of-specification (OOS) test results

The term OOS results includes all test results that fall outside the specifications or

acceptance criteria established in drug applications, drug master files (DMFs), official

compendia, or by the manufacturer

The term also applies to all in-process laboratory tests that are outside of established

specifications

25

Applicability of Guide

Applies to chemistry-based laboratory testing of drugs regulated by CDER

Directed toward traditional drug testing and release methods

Can we use it for:

Microbiological

Biological

Physical

Qualitative

tests?

26

The Guide

Laboratory tests are performed on active pharmaceutical ingredients, excipients and

other components, in-process materials, and finished drug products to the extent that

current good manufacturing practice (CGMP) regulations (21 CFR parts 210 and 211) and

the Federal Food, Drug, and Cosmetic Act (the Act) (section 501(a)(2)(B)) apply

Laboratory testing, required by CGMP regulations (211.160 and 211.165) is necessary to

confirm that components, containers and closures, in-process materials and finished

products conform to specifications, including stability specifications

Testing also supports analytical and process validation efforts

27

Why does your company perform Tests?

What is the purpose of testing?

Can you assure a correct result?

What is a correct result?

28

Why Does your company Perform Tests?

To determine if a critical quality attribute conforms or does not conform with its

specification

You cannot assure (at a 100% confidence level) that you will obtain a correct result

The correct result is that which is identical to the true result

but

You never know the true result because if you did… you wouldn’t need to test

29

Uncertainty

Since you cannot assure a correct result there is always some level of UNCERTAINTY

associated with any reported laboratory test result

Uncertainty represents risk

What is the risk associated with reporting an incorrect test result?

30

The risk

Declare that a result conforms with the specification when the TRUE result actually does

NOT conform

Declare that a result does not conform with the specification when the TRUE result actually

DOES conform

(False positive / False negative)

31

Some Definitions

ReMeasurement:

ReTest

ReSample

ABORTED TEST

INVALID TEST

INVALIDATED

TEST RESULT

Reinject same sample preparation

Additional test(s) on additional

aliquot of

same original sample

New sample

You stop the test e.g. spillage

Test acceptance criteria fail

(tells you nothing about the results)

Found conclusive evidence of

laboratory error: method, reagents,

performance of test, equipment etc.32/

Reportable Result – The Guide

“The term reportable result as used in this document means a final analytical result

This result is appropriately defined in the written approved test method and derived from

one full execution of that method, starting from the original sample”

33

USP Definition of a Reportable Value

A reportable value is often a summary value for several individual determinations

It is the end result of a completed measurement method, as documented

It is the value compared with the acceptance criteria

In most cases, the reportable value is used as documentation for internal and external

users

34

What is an OOS Results

A reportable result (obtained after applying the entire method as described and

performing any calculations and rounding off the significant figures) which when

compared to the specification is outside the limit / range or declared value

35

Ever heard this….

“The Lab don’t know how to test”

“ Give it to Sheila…

he knows how to do

that test…

Bruce always gets bad

results!”

36

What Could be the Cause of An OOS

1. _____________________________

2. _____________________________

3. _____________________________

Which is most likely?

37

How is Testing Performed

Collect a sample from the batch

Take part of sample (an aliquot) and test e.g. for a liquid: measure out a portion; inject

into HPLC

or

For a powder, take part of the sample (weigh it); dissolve in diluent and inject to HPLC

or

For a tablet: crush the tablet to make a powder; add diluent to dissolve the powder; inject

38

How is Testing Performed

Can take one aliquot and prepare for testing

Can take duplicate aliquots (two weighings) and prepare for testing

Can make duplicate injections (two injections)

39

How is Testing Performed

In which case, the method will describe the reportable result as…

Either the average result of the two weighings and two injections with a standard

deviation

or

The average result of each individual preparation (i.e. each preparation / weighing) and

not more than a certain RSD between the two (or more) results

40

Simplest Scenario (1):

One measurement, one preparation , one injection

41

Scenario (2):

3 preps; 3 injections, ONE reportable result

42

Scenario (3):

3 preps, 3 injections; one reinjection of preparation;

repreparation from weighed / crushed powder?

43

Apparent Implications

The individual determinations do not have to meet the criteria of the reportable result

Determinations are not generally reported out of the QC laboratory

The variability of determinations is similar to a system suitability issue

Set a limit on the standard deviation

44

Apparent Implications

All reportable results must be documented

Do not average OOS with in spec to get an in spec results to release the batch

Do not average reportable results for QA to make a batch release decision

QA must see all reportable results obtained during testing

If after QA makes a decision, and a value is needed for a COA, then average them

45

OOS Guide

The need to provide all reportable results to the QCU is

reinforced

“In addition, when investigation by a contract laboratory

does not determine an assignable cause, all test results

should be reported to the customer on the certificate of

analysis”

46

Before Barr – Current Practice

Prior to 1993 and the court decision – it was

COMMON practice to retest once

or

in exceptionally good companies twice

and to release the batch if the retest result was

within the specification

Companies had not really thought about the

practice

But then…nor had the regulators

47

Barr: What happened in court

The judge heard experts on

behalf of FDA and Barr regarding

the practice of retesting

FDA wanted retesting to be banned

under all circumstances

After a long hearing at which five industry experts, an

FDA investigator, and several company employees

testified, Judge Alfred M. Wolin, U.S. District Judge for

the District of New Jersey, issued a 79-page opinion 48

The Barr Court Case 1993 – Data Integrity?

Reported problems include

misplaced records

test data recorded on scrap paper

failure to control manufacturing steps such as

those governing products' physical properties

release of products not meeting their

specifications

inadequate investigation of failed products

49

Barr: “Testing into Compliance”

Barr had numerous failures

Performed retests with

no investigations

no regard for process and product history

Tested until results met specifications

Then irrespective of previous OOS results

for the batch, released product

reporting only the passing results

50

Q: How do you report passing OOS’s on COA?

Reading the Judgment

The Barr court judgment is like reading FDA’s guidance

Judge Wolin preferred "out-of-specification" (OOS) laboratory results rather than "product failure" used by FDA's investigators

Ruled an OOS result identified as laboratory error by investigation or an outlier test, or overcome by retesting is not a product failureBUT

Limited situations where laboratory error could be used 51

Guide to Inspection: QC Labs

Issued July 1993 (must have been working on

it while the court case was ongoing)

Addresses OOS results; tells inspectors to be alert

“Evaluate the company's system to investigate laboratory test

failures

These investigations represent a key issue in deciding whether a

product may be released or rejected and form the basis for retesting, and resampling “

52

Guide to Inspection: QC Labs

OOS results fall into three categories:

laboratory error

non-process related or operator error

process related or manufacturing process error

Evaluate the company's retesting SOP for compliance with scientifically sound and appropriate procedures

53

Guide to Inspection: QC Labs

A very important ruling in one recent court decision sets forth a procedure to govern the retesting program

This district court ruling provides an excellent guide to use in evaluating some aspects of a pharmaceutical laboratory, but should not be considered as law, regulation or binding legal precedent

The court ruled that a firm should have predetermined testing procedure and should consider a point where testing ends and product is evaluated. If results are not satisfactory, product is rejected

54

Collect Data – Be Objective

It is a capital mistake to theorize before one has data. Insensibly one begins to

twist facts to suit theories, instead of theories to suit facts.

Sir Arthur Conan Doyle (1859 – 1930)

Author of the Sherlock Holmes detective series

55

The FDA Guidance

III. IDENTIFYING AND ASSESSING OOS TEST RESULTS — PHASE I: LABORATORY

INVESTIGATION

The source of the OOS result should be identified either as an aberration of

the measurement process or an aberration of the manufacturing process

Even if a batch is rejected based on an OOS result, the investigation is

necessary to determine if the result is associated with other batches of the

same drug product or other products

Batch rejection does not negate the need to perform the investigation

The regulations require that a written record of the investigation be made,

including the conclusions and follow-up (§ 211.192)

56

Investigation Retesting

Hypothesis Testing

If you have a hypothesis, you can test it

That is NOT what is generally called retesting and usually should not be performed to

obtain a passing result but to recreate the FAILING result

The FDA Guidance

III. IDENTIFYING AND ASSESSING OOS TEST RESULTS — PHASE I: LABORATORY

INVESTIGATION

To be meaningful, the investigation should be thorough, timely, unbiased,

well-documented, and scientifically sound

The first phase of such an investigation should include an initial assessment

of the accuracy of the laboratory's data

Whenever possible, this should be done before test preparations (including

the composite or the homogenous source of the aliquot tested) are

discarded.

This way, hypotheses regarding laboratory error or instrument malfunctions

can be tested using the same test preparations.

58

Initial Assessment of OOS Result

Bear in mind prior:

Product history

Process history

Test history

Reliability of equipment

Reliability of the analyst

Precision of the test (validation)

59

The Guide

The first responsibility for achieving accurate laboratory testing results lies with the analyst

who is performing the test

The analyst should be aware of potential problems that could occur during the testing

process and should watch for problems that could create inaccurate results

In accordance with the CGMP regulations in § 211.160 (b)(4), the analyst should ensure

that only those instruments meeting established performance specifications are used and

that all instruments are properly calibrated

60

Responsibility of Analyst

To follow test procedure as written

To be alert to errors and STOP test BEFORE obtaining the result if error is suspected,

recording what happened e.g. spill

Analyst responsible for ensuring that

instruments meet performance specifications

and are properly calibrated [KSG: maintained?]

Once an OOS result is obtained to review all

records relative to the test to identify possible

laboratory error

61

Supervisory Role: USE a checklist

Visit the crime scene

Supervisors / team leaders / laboratory head:

Should be experienced analysts

Frequently audit while tests ARE BEING

performed

in order to be able to objectively investigate

OOS results

62

Responsibility of Supervisor

Objective assessment without

preconceived assumptions as to cause of OOS

Immediate assessment may include:

Re-examination of:

actual solutions

Test units

Glassware

used in the original measurements and preparations

This could provide more credibility for laboratory error hypotheses

63

Supervisor (The Guide)

1. Discuss the test method with the analyst; confirm analyst knowledge of and performance of the correct procedure.

2. Examine the raw data obtained in the analysis, including chromatograms and spectra, and identify anomalous or suspect information.

3. Verify that the calculations used to convert raw data values into a final test result are scientifically sound, appropriate, and correct; also determine if unauthorized or unvalidatedchanges have been made to automated calculation methods.

4. Confirm the performance of the instruments.

5. Determine that appropriate reference standards, solvents, reagents, and other solutions were used and that they met quality control specifications.

6. Evaluate the performance of the test method to ensure that it is performing according to the standard expected based on method validation data and historical data.

7. Fully document and preserve records of this laboratory assessment.

64

Responsibility of Supervisor

To INVESTIGATE:

Review notebook / worksheet with analyst :

Was method was followed: with a copy of themethod in your hand, have the analyst describeexactly how they performed each step:confirm that the method was understood & followed

Review raw data: Perform calculations againincluding checking dilution schemes

Unauthorised changes to automated calculations

Examine reagents, (reference) standards, solutions

Examine glassware

Performance of instruments

65

The FDA Guidance

III. IDENTIFYING AND ASSESSING OOS TEST RESULTS — PHASE I: LABORATORY

INVESTIGATION

If this initial assessment indicates that no meaningful errors were made in

the analytical method used to arrive at the data, a full-scale OOS

investigation should be conducted

66

Invalidate result

Perform new test on same sample

Convincing evidence

of

Laboratory Error

Correct if possible

or

Reject Batch

Production Error

Resample

Double sample

Revise sampling procedures

Sampling Error

QA decision

regarding

batch disposition

All within

specifications

Reject Batch

One result

OOS

Retest

??? further aliquots

from original sample

Inconclusive

No evidence of production error

QA Investigation

Inconclusive

Laboratory Investigation

Report to QA (copy in batch file?)

OOS resultOOS Flow Chart

67

Out of Specification Results

If the laboratory investigation is conclusive

Document findings

INVALIDATE original test

Perform NEW test on same sample

Report original result with investigation as well as new result in batch record for QA review prior to release

COA carries new result only

If the laboratory investigation is NOT conclusiveinform QA (or customer for contract lab) 68

Out of Trend Results

If the laboratory investigation is conclusive orinconclusive, consult with QA

In most cases, DO NOT perform any additionaltesting or sampling

Make product disposition judgment based on:

Original result

Product history (e.g. stability data – statistical analyses of particular use here)

Batch history (e.g. review indicates that there were noprocessing errors / there were errors)

Other investigational findings

69

FDA: Full-Scale OOS

Investigation

Use a pre-defined procedure

Production/ process review and / or additionallaboratory work

Identify root cause and implement CAPA

QA / QCU responsibility, includes CMOs if used

Documented in the batch record

Involves all aspects of manufacture, qualitycontrol and sampling

Describes corrective actions and endpoint

Is performed PRIOR to ANY retesting

70

FDA: Full-Scale OOS

Investigation

If cause of OOS is identified, batch is rejected

In this case need CAPA on process / product

May not identify cause and may need additionallab testing:

Retest additional portion of original sample

Resample

71

FDA: Retesting and Resampling

Use another analyst? Where possible

The maximum number of retests should be

specified in advance in an SOP

May, on rare occasions, deviate from SOP but

with documented rationale and protocol

The number may vary depending upon the

variability of the test method and NOT

depending on the results obtained

Resampling raises questions as to sampling

procedure validity

72

OOS Guidance

Is there a fundamental process flaw?

Product or process redesign is addressed with a new paragraph.

“OOS results may indicate a flaw in product or process design. … In such cases, it is

essential that redesign of the product or process be undertaken to ensure reproducible

product quality.”

73

A horrible question – but needs to be

asked…AND answered…honestly

Retesting

Retesting has an additional requirement.

“The maximum number of retests to be performed on a sample should be specified in advance in a written standard operating procedure (SOP)

Any deviation from this SOP should be rare

In such cases, before starting additional retesting, a protocol should be prepared that describes the additional testing to be performed and specifies the scientific and/or technical handling of the data”

74

Retesting – How many repeats?

While the guidance does not give

recommendations for the sample size for

retesting, the example given uses seven

retests

Seven was the suggestion in a footnote in

the Barr Case judgment

The sample size question is still unresolved

75

Resampling

“The original sample from a batch should be sufficiently large to

accommodate additional testing in the event an OOS result is

obtained

In some situations, however, it may be appropriate to collect a new

sample from the batch.”

76

Be very cautious about collecting a new sample

How to Identify OOT Results

Out of Trend or unusual results are generallyresults that:

MEET the product specification(may be borderline)

ARE outside the control limits of the process,where control charts are used – iMOH requiring this routinely…or

ARE different to results usually obtainede.g. spec: 95.0 – 110.0usual results: 98.5 – 101.0OOT result: 96.4

77

FDA: Concluding the Investigation

Evaluate results and determine batch quality

Release decision by QCU / QA

An initial OOS does not necessarily mean

that the batch fails and must be rejected.

Where the suspect result is invalidated, the

result should not be used to evaluate the

quality of the batch or lot

For inconclusive results – give full

consideration to the OOS result

78/87

FDA: Concluding the Investigation

Example given shows seven retest results

which gives the only indication in the guide

regarding numbers of retests

The example given is also extreme:

89.5% OOS

99.0, 98.9, 99.0, 99.1, 98.8, 99.1, 99.0%

Consider method precision and validation data

in making release / reject decision

79/87

FDA: Cautions

Where a series of assay results (to produce

a single reportable result) have some

individual results OOS and some in spec and

all within the known method variability,

passing results no more likely to represent

the true value than the OOS result.

In this case the company must err on the side of caution and reject the batch

A result that is low but in specification should

also be a cause of concern

80/87

Cautions

The first paragraph continues a discussion of reportable results stating that “… a firm should

err on the side of caution …”

a low assay result should raise concern and “One cause of the result could be that the batch was not formulated properly. Batches must be formulated with the intent to provide not less than 100 percent of the labeled or established amount of active ingredient”

81

Take Away Information

Companies must have an up-to-date SOP

Write the SOP at the 8th grade level

Use simple declarative sentences.

Use the wording directly out of the guidance where possible; copyright free!

Require the operational definition of the reportable result to be in the SOP, analytical test method, retest protocol and reports

82

Quality Metrics: are you ready?

report number of invalidations for lab error

Revisit your investigations

With an open mind

Ask:

Why did we get an OOS result in the first place

Could it have been laboratory error? YES

Could it have been production error? YES

Did we ELIMINATE the possibility of production error?

Did we CONCLUSIVELY demonstrate lab error?

83

Thank you for your Attention

Questions?

84/

OOS Keywords

Data governance

Data integrity

ALCOA +

OOS

OOT

Questionable / unusual / atypical result

Reportable result

Retest

New test

Invalid test

Aborted test

New test

Resample

Batch release

Investigation

Laboratory error

Production error

Sampling error

Data integrity

Data quality

Averaging

Training, qualification, education

COA

Reporting Results