Developing Bioanalytical Methods Balancing the Statistical Tightrope
“Lee: can I use this number?”
Process Development
GSK, 1997
2
“it’s about 40”
“about 40?”
“probably...”
3
Enlightenment?
5
Unconscious
Conscious
Incompetent Competent
Consciousness
Blooms Taxonomy
the 4 stages of competence
6
Me
A Statistical God
Using Statistics
1. Potency assays are key in making medicines
2. Bioassays are very variable
3. Statistics will help you understand your data
4. Understanding your data will reveal if control exists
5. Your level of control allows you to judge RISK
6. Regulators globally require it
Why? Six Reasons
9
The Regulator & Assay Control
1. Pharmaceutical cGMPs for the 21st
Century
2. PAT
3. ICH Q2: Validation of Analytical
Procedures
4. ICH Q8: Pharmaceutical Development
5. ICH Q9: Quality Risk Management
6. ICH Q10: Quality Pharmaceutical
Systems
Regulators have been asking for this for years! QbD
10
Statistics
The complete solution?
Or this?
Your assay? 12
Or this?
or your assay?
13
Statistics - an Amazing Transition
14
Bioassays will always be variable You can improve it - by understanding it - Focusing effort in right places - This brings control - You can manage expectations - This is understood by regulators
15
Why assay variation matters?
product variation +
assay variation +
inaccuracy
Many satisfactory OOS batches likely to fail (potentially costing £Ms)
because of combination of assay method & process inaccuracy & variation
A few unsatisfactory
batches may even
pass specification
due to a combination
of assay method and
process variability
16
Our Control Strategy
What does the scientist need to achieve?
i.e. selectivity, accuracy, precision linearity
Measure
Analyse
Improve
Control
Define
Identify & prioritise analytical CNX parameters
eXperimental
parameters
e.g., DoE
Regression
Noise
parameters
e.g., MSA,
Precision
Control
parameters
Fix & control
Method
Robustness
Method
Ruggedness
Method Control Strategy & reduce Risk prior to
Validation → Routine Use & Continuous Improvement
Input into
17
Generating Bioassay
Data
18
1. Speak with your statistician before
generating data
2.See Rule 1
The Rules
19
Lot’s data ≠ Value
20
21
Statistics are a tool 22
Which Tools?
Design
QC
Precision Accuracy Linearity etc.
TIME
UCL
LCL
Stage 1:
Qualification Tool
Fishbone, Minitab
Stage 2:
Development Tools DX8, JMP, Minitab
Stage 3:
Validation Tools Nested, CELLULA
Stage 4
QC Tools CELLULA, Shewhart chart,
CUSUM
Technology
Transfer YES
NO
What’s Appropriate Knowledge?
• Learning takes time
• Will you use it often enough?
• It’s not an academic pursuit
• Activities must add value do what’s necessary
24
Scope &
Design
26
Define & Scope
How is the assay performing? Prec/TOL2-sided = 6 x 16.76
100
= 1.01
Parameters (e.g. 15) pDNA
NaCl
pH
Tube Length
Time
Seeding Density
Ratio of Transfection
Temperature
Agitation and level
Vector – type, conc
Addition Order
Q. How Many parameters? Q. Which parameters? Q. What ranges?
A. Existing knowledge A. Common sense A. Practical limits
29
Define & Scope
Drill down - map out assay - build understanding & scope
Assay Flow
30
Define & Scope
Drill down & map out assay to build understanding & scope
Attention is focused
toward key steps
and the parameters
involved in these
steps
Cause & Effect Diagram (Fishbone) helps think your assay through
Identify & prioritise analytical CNX parameters
Scope & Screen
31
Scope ranges with simple experiments
Scoping Experiments
Explore mildest
to most forcing
conditions
32
Revealing the Big Hitters
Temptation
34
OFAT
pH
pDNA
NaCl
Provides estimates
of effects at set
conditions of the
other factors and
no interaction
effects.
Building Understanding
35
Building Understanding
Factorial Design
Estimates effects at
different conditions to
estimate interactions
Design of Experiments
DOE
250
1300
900
300 500
1800
350 600
2400 2600
36
work towards a
Robust Optimum
Optimise the parameters that
survived the initial screening
Optimisation
37
The tools allow you to simulate scenarios using the data you’ve built up
Simulations
Visual simulation of expected performance relative to specification
Is the Model Correct?
38
39
Ideal Settings
Control Space
Design Space
Method stretch…what if?
The evaluation of robustness should be considered during the development phase and should show the reliability of an analysis with respect to deliberate variations in method parameters ICH Q2B, 1994
Validate & Verify
Assay Control: control the parameters inside boundaries
40
Even if you go outside the control boundaries, the assay will have enough flexibility to deal with it without an OOS
41
Working within the control boundaries will keep the assay under control
Summary - Data Driven Development
Scope
Explore mildest
to most forcing
conditions
Optimize
Estimate & utilize
interactions to move
towards optimum
conditions
Verify
Rattle the cage to
deliver a design
space
QC/TT
Transfer to QC to
validate on batches
& bring into routine
use
Identify few potential
key parameters
Focus on vital few &
narrow ranges
Screen
43
Precision
It may be considered at three levels:
1. Repeatability
2. Intermediate precision
3. Reproducibility
ICH Q2A, 1994
Repeatability
1 analyst in 1 laboratory on 1 day injecting 6 times
Summary Statistics
Number of
Values Mean
Standard
Deviation
Coefficient
of Variation
Lower 95% CI
for Mean
Upper 95%
CI for Mean
t30 PS 6 223.27 6.43 2.88% 216.52 230.02
45
Intermediate Precision
As well as sample variation, this study still provides information on repeatability
46
• 1 analyst in 1 laboratory on
• 1 day
• injecting 6 samples
• each tested 6 times
So we compare the mean values for each sample (over replicate results per sample)
Intermediate Precision
Variance Components
Factor df Variance % Total
Sample 5 27.8535 21%
Repeat 30 102.6361 79%
35 130.4896 100%
Standard
Mean Deviation RSD
216.24 11.4232 5.28%
Variance Components
Factor df Variance % Total
Sample 5 27.8535 21%
Repeat 30 102.6361 79%
35 130.4896 100%
Standard
Mean Deviation RSD
216.24 11.4232 5.28%47
and the others…..?
Precision within a laboratory but with different analysts, on different days, with different equipment…reflects the real conditions within one laboratory
ICH Q2A 1995
48
49
Y
52000
52500
53000
53500
54000
54500
55000
55500
56000
0 5 10 15 20 25
Sample
Pe
ak
Are
a
Data collect using several analysts using several instruments
over several days:
Intermediate Precision
50
Y
52000
52500
53000
53500
54000
54500
55000
55500
56000
0 5 10 15 20 25
Sample
Pe
ak
Are
a
Potentially misleading: large analyst-to-analyst variation
present:
Analyst 1 Analyst 2 Analyst 3
Intermediate Precision
51
better examined looking at multiple sources of variation within an assay
Intermediate Precision
want to understand
major sources of
variation such as
sample, prep,
analyst etc.
52
Intermediate Precision
Can also perform Unbalanced designs
Intermediate Precision
One operator performs multiple injections on single
preparation;
Two operators perform single injections on multiple
preparations
53
54
…. sent to and analysed by other lab
C
B
A
C B A
Samples from
laboratory:
multiple laboratories; typically run as an inter-laboratory cross-over study, with each participating lab sending samples to every other lab and analysing all samples (including own)
Reproducibility
Can use analysis of variance (ANOVA) to look for differences or biases between labs
Alternatively look for “analytical equivalence”
Reproducibility
Risk Management The level of effort, formality and documentation.. ..should be commensurate with the level of risk
ICH Q9
Evaluation of the risk to quality should be based on scientific knowledge & ultimately link to the protection of the patient
Is the bioassay fit for purpose and under control?
56
57
Before & After
How is the assay performing? P/TOL2-sided = 6 x 16.76
100
= 1.01
58
Better P/TOL2-sided = 6 x 6.99
100
= 0.42
Before & After
59
Risk Management
Method Understanding, Control and Capability (MUCC)
Understand impact of variation
upon risk…
Capability
& Precision
Capable?
Control?
Risk
Management
Loop
Understanding?
Statistical
Process Control
(SPC) Charts
60
Understanding?
Capability
& Precision
Capable? Understanding?
100
= 1.01
P/TOL2-sided = 6 x 16.76
Capable? Control?
Risk Management
61
> 5% 0%
NoYes
0.0%
on the I chart.
The process mean is stable. No data points are out of control464136312621161161
225
210
195
180
Observation
t30
PS
_X=199.87
UCL=220.77
LCL=178.96
464136312621161161
30
20
10
0
Observation
Mo
vin
g R
an
ge
__MR=7.86
UCL=25.68
LCL=0
Comments
I-MR Chart of t30 PS
Summary Report
Is the process mean stable?
Evaluate the % of out-of-control points.I Chart
Investigate out-of-control points.
MR Chart
Investigate out-of-control points.
P/TOL2-sided = 6 x 6.99
100
= 0.42
Risk Management
1.Build a good basic understanding of
stats but don’t need to become guru
2.Involve a statistician, at least at the
beginning
3.Build understanding of your bioassay
(QbD) – it’s a must
4.Get to grips with Bioassay Variability
Summary
62
“Lee: can I use this number?”
63
“Yes – it’s 42 ”
0.05 with 95% Confidence
for the statisticians in the audience
64
…
Acknowledgments
Dr. Paul Nelson – Prism TC Ltd
Pictures from “The Cartoon Guide to Statistics” Larry Gonick & Woollcott Smith
65
