ibc biological assay development & validation 2011 gra presentation

Developing Bioanalytical Methods Balancing the Statistical Tightrope

“Lee: can I use this number?”

Process Development

GSK, 1997

2

“it’s about 40”

“about 40?”

“probably...”

3

Enlightenment?

5

Unconscious

Conscious

Incompetent Competent

Consciousness

Blooms Taxonomy

the 4 stages of competence

6

Me

A Statistical God

Using Statistics

1. Potency assays are key in making medicines

2. Bioassays are very variable

3. Statistics will help you understand your data

4. Understanding your data will reveal if control exists

5. Your level of control allows you to judge RISK

6. Regulators globally require it

Why? Six Reasons

9

The Regulator & Assay Control

1. Pharmaceutical cGMPs for the 21st

Century

2. PAT

3. ICH Q2: Validation of Analytical

Procedures

4. ICH Q8: Pharmaceutical Development

5. ICH Q9: Quality Risk Management

6. ICH Q10: Quality Pharmaceutical

Systems

Regulators have been asking for this for years! QbD

10

Statistics

The complete solution?

Or this?

Your assay? 12

Or this?

or your assay?

13

Statistics - an Amazing Transition

14

Bioassays will always be variable You can improve it - by understanding it - Focusing effort in right places - This brings control - You can manage expectations - This is understood by regulators

15

Why assay variation matters?

product variation +

assay variation +

inaccuracy

Many satisfactory OOS batches likely to fail (potentially costing £Ms)

because of combination of assay method & process inaccuracy & variation

A few unsatisfactory

batches may even

pass specification

due to a combination

of assay method and

process variability

16

Our Control Strategy

What does the scientist need to achieve?

i.e. selectivity, accuracy, precision linearity

Measure

Analyse

Improve

Control

Define

Identify & prioritise analytical CNX parameters

eXperimental

parameters

e.g., DoE

Regression

Noise

parameters

e.g., MSA,

Precision

Control

parameters

Fix & control

Method

Robustness

Method

Ruggedness

Method Control Strategy & reduce Risk prior to

Validation → Routine Use & Continuous Improvement

Input into

17

Generating Bioassay

Data

18

1. Speak with your statistician before

generating data

2.See Rule 1

The Rules

19

Lot’s data ≠ Value

20

21

Statistics are a tool 22

Which Tools?

Design

QC

Precision Accuracy Linearity etc.

TIME

UCL

LCL

Stage 1:

Qualification Tool

Fishbone, Minitab

Stage 2:

Development Tools DX8, JMP, Minitab

Stage 3:

Validation Tools Nested, CELLULA

Stage 4

QC Tools CELLULA, Shewhart chart,

CUSUM

Technology

Transfer YES

NO

What’s Appropriate Knowledge?

• Learning takes time

• Will you use it often enough?

• It’s not an academic pursuit

• Activities must add value do what’s necessary

24

Scope &

Design

26

Define & Scope

How is the assay performing? Prec/TOL2-sided = 6 x 16.76

100

= 1.01

Parameters (e.g. 15) pDNA

NaCl

pH

Tube Length

Time

Seeding Density

Ratio of Transfection

Temperature

Agitation and level

Vector – type, conc

Addition Order

Q. How Many parameters? Q. Which parameters? Q. What ranges?

A. Existing knowledge A. Common sense A. Practical limits

29

Define & Scope

Drill down - map out assay - build understanding & scope

Assay Flow

30

Define & Scope

Drill down & map out assay to build understanding & scope

Attention is focused

toward key steps

and the parameters

involved in these

steps

Cause & Effect Diagram (Fishbone) helps think your assay through

Identify & prioritise analytical CNX parameters

Scope & Screen

31

Scope ranges with simple experiments

Scoping Experiments

Explore mildest

to most forcing

conditions

32

Revealing the Big Hitters

Temptation

34

OFAT

pH

pDNA

NaCl

Provides estimates

of effects at set

conditions of the

other factors and

no interaction

effects.

Building Understanding

35

Building Understanding

Factorial Design

Estimates effects at

different conditions to

estimate interactions

Design of Experiments

DOE

250

1300

900

300 500

1800

350 600

2400 2600

36

work towards a

Robust Optimum

Optimise the parameters that

survived the initial screening

Optimisation

37

The tools allow you to simulate scenarios using the data you’ve built up

Simulations

Visual simulation of expected performance relative to specification

Is the Model Correct?

38

39

Ideal Settings

Control Space

Design Space

Method stretch…what if?

The evaluation of robustness should be considered during the development phase and should show the reliability of an analysis with respect to deliberate variations in method parameters ICH Q2B, 1994

Validate & Verify

Assay Control: control the parameters inside boundaries

40

Even if you go outside the control boundaries, the assay will have enough flexibility to deal with it without an OOS

41

Working within the control boundaries will keep the assay under control

Summary - Data Driven Development

Scope

Explore mildest

to most forcing

conditions

Optimize

Estimate & utilize

interactions to move

towards optimum

conditions

Verify

Rattle the cage to

deliver a design

space

QC/TT

Transfer to QC to

validate on batches

& bring into routine

use

Identify few potential

key parameters

Focus on vital few &

narrow ranges

Screen

43

Precision

It may be considered at three levels:

1. Repeatability

2. Intermediate precision

3. Reproducibility

ICH Q2A, 1994

Repeatability

1 analyst in 1 laboratory on 1 day injecting 6 times

Summary Statistics

Number of

Values Mean

Standard

Deviation

Coefficient

of Variation

Lower 95% CI

for Mean

Upper 95%

CI for Mean

t30 PS 6 223.27 6.43 2.88% 216.52 230.02

45

Intermediate Precision

As well as sample variation, this study still provides information on repeatability

46

• 1 analyst in 1 laboratory on

• 1 day

• injecting 6 samples

• each tested 6 times

So we compare the mean values for each sample (over replicate results per sample)

Intermediate Precision

Variance Components

Factor df Variance % Total

Sample 5 27.8535 21%

Repeat 30 102.6361 79%

35 130.4896 100%

Standard

Mean Deviation RSD

216.24 11.4232 5.28%

Variance Components

Factor df Variance % Total

Sample 5 27.8535 21%

Repeat 30 102.6361 79%

35 130.4896 100%

Standard

Mean Deviation RSD

216.24 11.4232 5.28%47

and the others…..?

Precision within a laboratory but with different analysts, on different days, with different equipment…reflects the real conditions within one laboratory

ICH Q2A 1995

48

49

Y

52000

52500

53000

53500

54000

54500

55000

55500

56000

0 5 10 15 20 25

Sample

Pe

ak

Are

a

Data collect using several analysts using several instruments

over several days:

Intermediate Precision

50

Y

52000

52500

53000

53500

54000

54500

55000

55500

56000

0 5 10 15 20 25

Sample

Pe

ak

Are

a

Potentially misleading: large analyst-to-analyst variation

present:

Analyst 1 Analyst 2 Analyst 3

Intermediate Precision

51

better examined looking at multiple sources of variation within an assay

Intermediate Precision

want to understand

major sources of

variation such as

sample, prep,

analyst etc.

52

Intermediate Precision

Can also perform Unbalanced designs

Intermediate Precision

One operator performs multiple injections on single

preparation;

Two operators perform single injections on multiple

preparations

53

54

…. sent to and analysed by other lab

C

B

A

C B A

Samples from

laboratory:

multiple laboratories; typically run as an inter-laboratory cross-over study, with each participating lab sending samples to every other lab and analysing all samples (including own)

Reproducibility

Can use analysis of variance (ANOVA) to look for differences or biases between labs

Alternatively look for “analytical equivalence”

Reproducibility

Risk Management The level of effort, formality and documentation.. ..should be commensurate with the level of risk

ICH Q9

Evaluation of the risk to quality should be based on scientific knowledge & ultimately link to the protection of the patient

Is the bioassay fit for purpose and under control?

56

57

Before & After

How is the assay performing? P/TOL2-sided = 6 x 16.76

100

= 1.01

58

Better P/TOL2-sided = 6 x 6.99

100

= 0.42

Before & After

59

Risk Management

Method Understanding, Control and Capability (MUCC)

Understand impact of variation

upon risk…

Capability

& Precision

Capable?

Control?

Risk

Management

Loop

Understanding?

Statistical

Process Control

(SPC) Charts

60

Understanding?

Capability

& Precision

Capable? Understanding?

100

= 1.01

P/TOL2-sided = 6 x 16.76

Capable? Control?

Risk Management

61

> 5% 0%

NoYes

0.0%

on the I chart.

The process mean is stable. No data points are out of control464136312621161161

225

210

195

180

Observation

t30

PS

_X=199.87

UCL=220.77

LCL=178.96

464136312621161161

30

20

10

0

Observation

Mo

vin

g R

an

ge

__MR=7.86

UCL=25.68

LCL=0

Comments

I-MR Chart of t30 PS

Summary Report

Is the process mean stable?

Evaluate the % of out-of-control points.I Chart

Investigate out-of-control points.

MR Chart

Investigate out-of-control points.

P/TOL2-sided = 6 x 6.99

100

= 0.42

Risk Management

1.Build a good basic understanding of

stats but don’t need to become guru

2.Involve a statistician, at least at the

beginning

3.Build understanding of your bioassay

(QbD) – it’s a must

4.Get to grips with Bioassay Variability

Summary

62

“Lee: can I use this number?”

63

“Yes – it’s 42 ”

0.05 with 95% Confidence

for the statisticians in the audience

64

…

Acknowledgments

Dr. Paul Nelson – Prism TC Ltd

Pictures from “The Cartoon Guide to Statistics” Larry Gonick & Woollcott Smith

65