capability assessments for continued process … assessments for continued process verification:...

Capability Assessments for Continued

Process Verification: Overcoming

potential obstacles and common

pitfalls

SIV 2015

Presenter: Peter M. Saama, Ph.D.

Bayer HealthCare LLC

Agenda/

Content Page 1

Continued Process Verification (Phase III)

Overview: FDA Guidance, Process Capability

Small Sample Size

Non-Normality

Auto-correlation

Mean Shifts

Chronological Order

Round-off Errors

Random Fluctuations

Requirements for FDA Validation Guidance

• FDA Guidance for Industry: Process Validation: General Principles and Practices,

published January 2011 distinguishes three stages of validation

• Stage 1 – Process Design: The commercial manufacturing process is defined during

this stage based on knowledge gained through development and scale-up activities.

• Stage 2 – Process Qualification: During this stage, the process design is evaluated to

determine if the process is capable of reproducible commercial manufacturing.

• Stage 3 – Continued Process Verification: Ongoing assurance is gained during

routine production that the process remains in a state of control.

• Should understand sources of variation

• Detect the presence and degree of variation

• Understand the impact of variation on the process and, ultimately, product

attributes

• Control the variation in a manner commensurate with the risk it presents to the

process and product

• Control Chart depicts inherent variation in a process Stable (In-Control)

• Process Capability Index (PCI) assesses whether or not a process is capable of meeting

customer requirements (a.k.a. PCR)

• Cp, Cpk, Cpm Process is stable (In-Control)

Where with ; from Shewhart’s Tables; is the standard

deviation and T is the target value. Here Cpm is the so-called the Taguchi Index.

• Ppk Process with known and predictable special causes

• Where

Overview: Process Capability

Scope:

• Assume that:

• Critical Quality Attributes (CQA) have been identified for the product

• Parameter Criticality has been established for the process

• Critical Process Parameters (CPP) are linked to CQA

• Alert and action limits have been set for CPP

• Risk analysis was used to determine the monitoring and sampling requirements of

CPP’s and CQA’s

• CPV Strategy is fully implemented

• Examine Capability Assessments for commercialized products

• Small Sample Size (n < 30)

• Process Complexity

Product X Dataset: n = 50

Serial

No.

BULK

Batch

Batch

Disposition

Manufacturing

Date

% Label Claim

(90-110)%

1 PX9999R REL 8-Oct-2012 98.8

2 PX99020 REL 15-Oct-2012 98.4

3 PX990WR REL 24-Oct-2012 98.6

4 PX990WT REL 15-Nov-2012 98.4

5 PX990WV REL 16-Nov-2012 97.4

6 PX990WW REL 10-Dec-2012 99.2

7 PX990QW REL 12-Dec-2012 99.3

8 PX990NV REL 14-Dec-2012 99.9

9 PX990RW REL 17-Dec-2012 99.8

10 PX991WW REL 21-Dec-2012 100.4

11 PX991WX REL 10-Jan-2013 98.4

12 PX991V0 REL 11-Jan-2013 98.2

13 PX991V1 REL 25-Mar-2013 98.4

14 PX991V2 REL 1-Apr-2013 98.6

15 PX992WN REL 11-Apr-2013 96.8

16 PX993N1 REL 12-Apr-2013 98.2

17 PX994RW REL 18-Apr-2013 98.1

18 PX994RR REL 25-Apr-2013 98.4

19 PX9954M REL 12-Jun-2013 96.8

20 PX995WR REL 19-Jun-2013 96.6

21 PX99676 REL 20-Jun-2013 97.3

22 PX99677 REL 9-Aug-2013 97.8

23 PX997VM REL 9-Aug-2013 97.0

24 PX997VN REL 21-Aug-2013 98.4

25 PX997VP REL 30-Aug-2013 98.2

Serial

No.

BULK

Batch

Batch

Disposition

Manufacturing

Date

% Label Claim

(90-110)%

26 PX998T0 REL 16-Oct-2013 101.1

27 PX998MV REL 11-Nov-2013 100.7

28 PX998T1 REL 14-Nov-2013 100.3

29 PX998T2 REL 21-Nov-2013 100.6

30 PX999NR REL 27-Jan-2014 98.1

31 PX99WRV REL 27-Jan-2014 98.9

32 PX999NP REL 3-Feb-2014 98.5

33 PX99WR9 REL 10-Feb-2014 100.1

34 PX99WRW REL 10-Feb-2014 98.2

35 PX9996M REL 17-Feb-2014 97.2

36 PX99RM8 REL 19-Feb-2014 98.6

37 PX99Q35 REL 10-Apr-2014 96.9

38 PX99Q36 REL 11-Apr-2014 96.9

39 PX99O0W REL 17-Apr-2014 97.7

40 PX99O0X REL 21-Apr-2014 97.9

41 PX99O7R REL 28-Apr-2014 98.4

42 PX99OV9 REL 9-Jun-2014 97.2

43 PX99OV8 REL 16-Jun-2014 97.4

44 PX99OXM REL 19-Jun-2014 99.2

45 PX99NQ9 REL 7-Jul-2014 94.4

46 PX99PM0 REL 9-Jul-2014 97.8

47 PX99N76 REL 28-Jul-2014 98.7

48 PX998MT REL 27-Aug-2014 102.2

49 PX99R9X REL 15-Sep-2014 100.8

50 PX99PXR REL 13-Oct-2014 99.1

Normal Capability Analysis for Product X: n=50

Normal Capability Analysis for Product X: n=48

(w/o Special Cause Variation)

Small Sample Size: n < 30

• SPC Assumptions Normality, Independence/Exchangeability, Chronological order,

Specifications based on consumer’s risk, zero autocorrelation, n ≥ 30.

• Impact: Conventional SPC methods (Cp, Cpk) do not apply

• First 25 batches (Product X)

Serial

No.

BULK

Batch

Batch

Disposition

Manufacturing

Date

% Label Claim

(90-110)%

1 PX9999R REL 8-Oct-2012 98.8

2 PX99020 REL 15-Oct-2012 98.4

3 PX990WR REL 24-Oct-2012 98.6

4 PX990WT REL 15-Nov-2012 98.4

5 PX990WV REL 16-Nov-2012 97.4

6 PX990WW REL 10-Dec-2012 99.2

7 PX990QW REL 12-Dec-2012 99.3

8 PX990NV REL 14-Dec-2012 99.9

9 PX990RW REL 17-Dec-2012 99.8

10 PX991WW REL 21-Dec-2012 100.4

11 PX991WX REL 10-Jan-2013 98.4

12 PX991V0 REL 11-Jan-2013 98.2

13 PX991V1 REL 25-Mar-2013 98.4

14 PX991V2 REL 1-Apr-2013 98.6

15 PX992WN REL 11-Apr-2013 96.8

16 PX993N1 REL 12-Apr-2013 98.2

17 PX994RW REL 18-Apr-2013 98.1

18 PX994RR REL 25-Apr-2013 98.4

19 PX9954M REL 12-Jun-2013 96.8

20 PX995WR REL 19-Jun-2013 96.6

21 PX99676 REL 20-Jun-2013 97.3

22 PX99677 REL 9-Aug-2013 97.8

23 PX997VM REL 9-Aug-2013 97.0

24 PX997VN REL 21-Aug-2013 98.4

25 PX997VP REL 30-Aug-2013 98.2

10810510299969390

LSL USL

LSL 90

Target *

USL 110

Sample Mean 98.3024

Sample N 25

StDev (Within) 0.617612

StDev (O v erall) 0.974206

Process Data

C p 5.40

C PL 4.48

C PU 6.31

C pk 4.48

Pp 3.42

PPL 2.84

PPU 4.00

Ppk 2.84

C pm *

O v erall C apability

Potential (Within) C apability

PPM < LSL 0.00

PPM > USL 0.00

PPM Total 0.00

O bserv ed Performance

PPM < LSL 0.00

PPM > USL 0.00

PPM Total 0.00

Exp. Within Performance

PPM < LSL 0.00

PPM > USL 0.00

PPM Total 0.00

Exp. O v erall Performance

Within

Overall

Process Capability of % Label Claim


-Suggested Solutions-

• For CPP/CQA with Two-Sided limits, estimate Two-Sided 95% or 99% or 99.9% Tolerance

Intervals

• Note: Risk Assessment is used to define, apriori, the uncertainty (99.9% or 99% or

95%) as part of the PROCESS CONTROL STRATEGY.

• 99% (p=.99) of the population with confidence level of 95% (α=0.05; γ=0.95)

consistent with standard coverage limits for SPC

• Assume 99% uncertainty and 95% confidence

• Estimate Tolerance Factor, k2, according to Guenther (1977)

, i.e.,

• Tolerance Interval is:

• Use Lower Tolerance Bound (LTB) and Upper Tolerance Bound (UTB) estimates

from 99% Tolerance Interval.

• For a single sample, use the Cpk metric:



• For CPP/CQA with a one-sided limit, estimate a one-sided 95% or 99% or 99.9%

Tolerance Intervals

• Assume 99% uncertainty (p=.99) and 95% confidence (α=0.05; γ=0.95)

• Estimate Tolerance Factor, k1, according to Natrella (1963):

, i.e.,

• Tolerance Interval is:

• Use Lower Tolerance Bound (LTB), , and Upper Tolerance Bound (UTB)

, , estimates from 99% Tolerance Interval.

• For a single sample, use the CpL and CpU metrics:



• Generate 15 ‘batches’ from the Normal Distribution, N(1.004, 0.01332): Product U

• Generate 30 ‘batches’ from the Normal Distribution, N(1.004, 0.01332): Product W



• Capability assessment for Product U using small-sample approximation,

• Capability assessment for Product U using the standard analysis:



• Capability assessment for Product W using small-sample approximation,

• Capability assessment for Product W using the standard analysis:



• For multiple samples, use the Ppk metric

• As noted previously, use half-width of the Tolerance Interval in the divisor(s) for the

Process Capability Index (PCI)

• 99% Population Proportion

• 95% Confidence Level

• For Risk Assessment (FMEA): RPN=v, use α=c=.05, Tolerance Interval= [LTB, UTB]

Process Complexity (Issue 1): Non-Normality

• Evidence of Non-normality in Histogram plot

• Impact: Unreliable estimates of Process Capability Indices



• Probability Plot:

where is the 99.865 percentile (see below)

• Johnson Transformation followed by ‘Normal’ Process Capability

where is a standard normal variate, are unknown, and

Lognormal parameter estimates:

where 100αth percentile is α(n+1)th-ranked value from n observations.



• Box-Cox Transformation followed by Normal Process Capability

where is obtained by Maximum Likelihood Estimation

• ‘Non-Normal’ Process Capability

• Log-Normal

• Error does not depend on measurement

• Constant Standard Deviation

• Process is centered

• Log-Transformation followed by ‘Normal’ Process Capability



• Performance under slight, moderate, and severe departures from Normality

• PCI values (Cpu) from lognormal distribution with µ=0.0 and σ2=0.5 (n=100).

• PCI values (Cpu) from lognormal distribution with µ=0.0 and σ2=0.3 (n=100).

• Box-Cox and Johnson transformations are superior (n ≥100)

• Not appropriate for small sample sizes; unintentional process shift is induced.

• Probability Plot is suitable when there is a mild departure from normality for the underlying

process.

Process Complexity (Issue 2): Auto-Correlation

• High association between adjacent data points. Caused by:

• Production Lot

• Production (Weeks)

• Raw Material Lot (early in production cycle)

• Impact: Profound effect on Control Chart and Process Capability.

• Cpk underestimate total process variation

• Positive autocorrelation more signals in Control Chart

• In-control ARL becomes shorter


-Suggested Solution(s)-

• Short-term limits = 6 * short-term Sigma

• Long-term limits= 6 * long-term Sigma

• Compare Cpk to Ppk

• Ppk (Long Term ) more representative of Process Capability

• Monitor autocorrelation structure using Sample autocorrelation function (ACF), e.g.

Product X

13121110987654321

1.0

0.8

0.6

0.4

0.2

0.0

-0.2

-0.4

-0.6

-0.8

-1.0

Lag

Au

toco

rre

lati

on

Autocorrelation Function for % Label Claim(with 5% significance limits for the autocorrelations)



• Less frequent sampling can remove autocorrelation

• Product X every 2nd observation (sub-sampling): CAF

654321

1.0

0.8

0.6

0.4

0.2

0.0

-0.2

-0.4

-0.6

-0.8

-1.0

Lag

Au

toco

rre

lati

on

Autocorrelation Function for % Label Claim(with 5% significance limits for the autocorrelations)

Process Complexity (Issue 3): Mean-Shifts

• Inherent due to:

• Campaign effects

• Raw Material Changes

• Seasonal effects on the manufacturing process

• Scale-up (e.g. Larger Tank or Increase in Batch Size)

• Affects Critical Process Parameters (CPP): e.g. Mixing Speeds

• Affects Critical In-Process Controls: Net Yield, Yield %

• Increased Risk of Operator Error



• Compare Cpk to Ppk

• Ppk (Long Term) more representative of Process Capability

• Scale-up/Scale-down

• Data Sub-groups Compounding Tank / Batch Size

• Capability Analysis w/n sub-group (n ≥ 30)

• If Cpk ≥ 1.33 in all sub-groups Process is Capable

• CUSUM / EWMA charts to detect mean shifts

PX99PXR

PX99R9X

PX998MT

PX99N76

PX99PM0

PX99NQ9

PX99OXM

PX99OV8

PX99OV9

PX99O7R

PX99O0X

PX99O0W

PX99Q36

PX99Q35

PX99RM8

PX9996M

PX99WRW

PX99WR9

PX999NP

PX99WRV

PX999NR

PX998T2

PX998T1

PX998MV

PX998T0

PX997VP

PX997VN

PX997VM

PX99677

PX99676

PX995WR

PX9954M

PX994RR

PX994RW

PX993N1

PX992WN

PX991V2

PX991V1

PX991V0

PX991WX

PX991WW

PX990RW

PX990NV

PX990QW

PX990WW

PX990WV

PX990WT

PX990WR

PX99020

PX9999R

99.5

99.0

98.5

98.0

97.5

BULK Batch

EW

MA

__X=98.486

UCL=99.433

LCL=97.538

EWMA Chart of % Label Claim

PX99PXR

PX99R9X

PX998MT

PX99N76

PX99PM0

PX99NQ9

PX99OXM

PX99OV8

PX99OV9

PX99O7R

PX99O0X

PX99O0W

PX99Q36

PX99Q35

PX99RM8

PX9996M

PX99WRW

PX99WR9

PX999NP

PX99WRV

PX999NR

PX998T2

PX998T1

PX998MV

PX998T0

PX997VP

PX997VN

PX997VM

PX99677

PX99676

PX995WR

PX9954M

PX994RR

PX994RW

PX993N1

PX992WN

PX991V2

PX991V1

PX991V0

PX991WX

PX991WW

PX990RW

PX990NV

PX990QW

PX990WW

PX990WV

PX990WT

PX990WR

PX99020

PX9999R

0

-10

-20

-30

-40

-50

-60

BULK Batch

Cu

mu

lati

ve

Su

m

0UCL=3.79

LCL=-3.79

CUSUM Chart of % Label Claim

Process Complexity (Issue 4): Chronological Order

• Process data should be date-ordered a) Missing dates, b) Data Entry errors, c) Data not

sorted by date

• a) Missing dates Pattern of missing dates

• Missing At Random (MAR)

• Skipped due to delay in raw material testing results Lack of raw materials

• Skipped due to unplanned events such as inclement weather

• Missing Completely at Random (MCAR): e.g. SRS taken during investigations

• Missing Not At Random (NMAR)

• Intentional, e.g. Batch was scrapped

• Skipped due to sudden change in market demand for product change in

schedule


• Product X: Missing At Random (MAR), i.e. random sample of 40 without replacement

10810510299969390

LSL USL

LSL 90

Target *

USL 110

Sample Mean 98.367

Sample N 40



Process Data

C p 3.92

C PL 3.28

C PU 4.56

C pk 3.28

Pp 3.12

PPL 2.61

PPU 3.63

Ppk 2.61

C pm *



PPM < LSL 0.00

PPM > USL 0.00

PPM Total 0.00


PPM < LSL 0.00

PPM > USL 0.00

PPM Total 0.00


PPM < LSL 0.00

PPM > USL 0.00

PPM Total 0.00


Within

Overall



• b) Data Entry Errors

• Dates entered incorrectly.

• c) Data not sorted by date

• Dates are recorded Process Data Not in chronological order

• Control Chart (Product X)

PX99WRW

PX994RW

PX99O0W

PX99N76

PX99PM0

PX991V0

PX99Q36

PX990RW

PX998MT

PX990NV

PX999NP

PX99O0X

PX992WN

PX99WRV

PX997VM

PX991V2

PX99020

PX991WW

PX99NQ9

PX990QW

PX9999R

PX997VN

PX99R9X

PX990WT

PX998T2

PX990WW

PX998MV

PX99Q35

PX998T0

PX99OV9

PX995WR

PX99OXM

PX999NR

PX998T1

PX9954M

PX991WX

PX99OV8

PX991V1

PX9996M

PX99RM8

PX990WR

PX99WR9

PX994RR

PX99676

PX99PXR

PX99677

PX997VP

PX993N1

PX99O7R

PX990WV

110

105

100

95

90

BULK Batch

Ind

ivid

ua

l V

alu

e

_X=98.49

UCL=102.90

LCL=94.07

110

90

I Chart of % Label Claim

Note: Data are not in chronological order


• Process Capability (Product X): Random shuffle

10810510299969390

LSL USL

LSL 90

Target *

USL 110

Sample Mean 98.4856

Sample N 50



Process Data

C p 2.26

C PL 1.92

C PU 2.61

C pk 1.92

Pp 2.42

PPL 2.05

PPU 2.78

Ppk 2.05

C pm *



PPM < LSL 0.00

PPM > USL 0.00

PPM Total 0.00


PPM < LSL 0.00

PPM > USL 0.00

PPM Total 0.00


PPM < LSL 0.00

PPM > USL 0.00

PPM Total 0.00


Within

Overall


In Chronological Order



• Identify patterns/reasons for missing dates and recode where applicable.

• Increased awareness of missing date patterns

• Understand the distribution of missing dates

• Missing Date Imputation not an option

• Mitigation of missing dates via preventive or corrective actions

• Reduces compliance risk associated with inadequate disclosure

• Less likely to conclude that Process is Not Capable when it is Capable: Reduces Type

II Error

Process Complexity (Issue 5): Round-Off Errors

• Measurements are used to extrapolate to batch processe

• Quality of measurements degraded by round-off errors

• Round-off errors introduced during production

• Round-off errors introduced during data analysis

• Large round-off errors compromised data


• Intra-class Correlation ((ICC)

• Increase in measurement error Decrease in ICC

• Rule I Data value outside ±3SD

• Rule I; ICC = 0.5 88% chance of detection



• In order for process behavior to be predictable uncertainty due to measurement error

needs to be minimized

• Minimize rounding during preliminary steps in data analysis

• When possible, round-off at the end of all calculations

• ICC can be used to quantify uncertainty due to round-off errors

• Impact of round-off error on prediction of process performance < other causes of

uncertainty.

Process Complexity(Issue 6): Random Fluctuations

• “Imprecise” lacking precision without being meaningless. Applicable when specification

limits cannot always be represented by precise numbers (Kaya and Kahraman 2008; Chen, Lai

et al. 2008; Parchami et al. 2014), e.g., Release limits were estimated from a process with

inherent random fluctuations.

• When Specifications are “Imprecise”, PCI is also “Imprecise”. Classic PCI do not apply.

• Example, specification limits are triangular numbers. This applies to certain quantitative quality

measurements, e.g. microelectronics manufacturing process and glass manufacturing.

Process Complexity (Issue 6): Random Fluctuations


• Consider a PCI (FPCI), that is a generalization of the loss-based Taguchi PCI ( Hsu

and Shu 2008)

• Reduces to a classical test with a binary decision

• Computer a 100(1-α)% confidence interval for FPCI

• As increases, the tends to a sharper triangular number.

Process Complexity: Conclusions (1 of 2)

• Strategies for addressing common pitfalls in implementing Continued Process Verification

have been proposed:

• Small Sample Size

• Surrogate Process Capability Index based on Tolerance Intervals

• Can perform capability assessments for recently commercialized products

• Non-Normality

• Box-Cox transformation followed by ‘Normal’ Capability analysis

• Auto-correlation

• Sub-sampling if dataset is large and interval between batches is short.

• Split dataset if different batch sizes represented in the data

• Mean Shifts

• Address root-cause

Process Complexity: Conclusions (2 of 2)

• Chronological order

• Ensure that data are in chronological order

• Round-off Errors

• Carry out any intermediate calculations on the original scale

• Round-off final estimates

• Intra-class Correlation can be used to quantify uncertainty

• Random Fluctuations

• Loss-based Process Capability Index, Taguchi Index, is suggested

Thank you!

Forward-

Looking

Stateme

nts

This presentation may contain forward-looking statements

based on current assumptions and forecasts made by Bayer

Group or subgroup management.

Various known and unknown risks, uncertainties and other

factors could lead to material differences between the actual

future results, financial situation, development or

performance of the company and the estimates given here.

These factors include those discussed in Bayer’s public

reports which are available on the Bayer website at

www.bayer.com.

The company assumes no liability whatsoever to update

these forward-looking statements or to conform them to

future events or developments.

capability assessments for continued process … assessments for continued process verification:...

Documents