©2016, Genentech

Recommendations to Challenges for

Appropriate Selection and Characterization

of Calibrator Material

Kyra J. Cowan, Ph.D.Biomarker Theme, AAPS NBC 2016:

Characterization of Protein Biomarker Reference Material

©2016, Genentech

Background

Consensus to date

Questions from Scientists

Biomarker Assay Protein calibrators Subteam

Scope of work for cross-industry team: BAPS

Approach to date

Recommendations and Solutions for Challenges

Across protein biomarker assay calibrators

Fit-for-purpose (FFP) approach

Commutability

Summary

Acknowledgements

Outline 2

©2016, Genentech

Publications Cross-Industry Meetings/Conferences 2013 FDA Draft Guidance

Written Consensus papers:

Lee et al. 2005, 2006, 2009

Biomarker assay validation

O’Hara et al. 2012

Critical Reagents characterization

King et al. 2014

GBC Harmonization white paper on

critical reagents for LBAs

Bower et al. 2014

Commentary paper on

reference standards and

reagents in BMV

Building Industry Consensus:

AAPS NBC 2014, 2015, 2016

- Biomarker themes, topics

Crystal City V 2013

FDA draft guidance discussion

Crystal City VI 2015

Biomarker assay discussion

Lowes, Ackermann 2016

2nd Upcoming publication

WRIB 2014, 2015, 2016

Biomarker assay discussions

Gap Identified: Best practices for the selection and characterization

of calibrator material.3

©2016, Genentech

Key Difference: PK vs. Biomarker Assay Calibrators

Calibrator Analyte

PK Assay example

•Unlikely that recombinant material will be the same as the endogenous analyte for biomarker assays.

•Recombinant material for protein biomarker assays often not well characterized, or readily available

Recombinant Calibrator Analyte

Biomarker Assay example

Glyc

OR

Glyc Glyc

Monomer No PTM

Glyc

Difficult to express Misfolded, aggregates

4

©2016, Genentech

What characteristics are we looking for in a recombinant material for our assays?

Challenge is to match recombinant material with endogenous

Potential post-translation modifications, depending on disease-state, matrix, treatment regimen, genetics, environment…

Are we measuring what we think we’re measuring?

Specificity vs. Interference

Is the reagent we’re using reliable as a calibrator for our assay?

Parallelism – must be assessed early on

Lot-to-lot variability, stability

Commutability

What is the “best” calibrator material for protein

biomarker assays? 5

©2016, Genentech

BAPS: Biomarker Assay Protein calibrators Subteam

LBABFG Biomarker Discussion Group Need recommendations on calibrator selection, characterization

Need for an AAPS subteam to establish consensus to address challenges

Scope of Subteam Recommend best practices for bioanalytical scientists

FFP approach for selection, characterization, and assay acceptance.

Based on experience, challenges faced, important case studies, and relevant literature.

Define the important characteristics that a scientist should look for in identifying a reliable recombinant material.

Evaluate the choices for different types of calibrator protein depending on the challenges faced during assay development (from full-length proteins to truncated forms or peptides).

Introduce and incorporate appropriate commutability approaches used in clinical chemistry and approaches for available NIST standards into industry-wide recommendations.

6

©2016, Genentech

Kyra Cowan (TL), Genentech

Lakshmi Amaravadi, Sanofi

Mark Cameron, Beckman-Coulter

Damien Fink, Janssen

Darshana Jani, Pfizer

Medha Kamat, Eurofins

Lindsay King, Pfizer

RJ Neely, Bristol-Myers Squibb

Yan Ni, Bristol Myers-Squibb

Paul Rhyne, Q2 Solutions

Renee Riffon, CiToxLAB North America

Yuda Zhu, Genentech

BAPS 7

©2016, Genentech

List of challenges each team member has faced

Develop recommendations for:

Challenges common to all protein biomarkers

Challenges unique to commonly tested, available proteins

Challenges unique to novel targets/uncommon proteins

FFP Table

Recommendations across proteins for characterization

Risks and caveats

Commutability approach for bioanalytical scientists

Apply to recombinant material for biomarker assays

Approach 8

©2016, Genentech

Know your endogenous protein!• Amino acid sequence; Tertiary, quaternary structure?

• Is the protein monomeric/dimeric/oligomeric in nature?

• Does the protein of interest undergo cleavage or alternative splicing?

• How is that represented in the “total” protein?

• Does it have a binding partner, or bind with drug?

• Are “misfolded” proteins reactive in the assay? Are they bioactive?

• Endogenous and exogenous match, but do the modifications or post-translational modifications match? Glycosylation? Isoforms? His/Flag/etc. tagged.

How much must the exogenous protein have similarity to the endogenous protein?

Further Understanding:• Look Into Protein Database Resources, published biology

• Take home message: Calibrator material should be as close to the endogenous form of the protein as possible, to provide confidence in true quantitation

• Also, should know your assay and what it can detect.

Case Study: State of endogenous “heterodimer” versus calibrator material

Solutions/Recommendations Across Proteins 9

©2016, Genentech

BIOLOGY:

IL-23 is a heterodimer protein consisting of a p19 and p40 subunit.

Shares p40 subunit with IL-12 cytokine

KIT:

Capture Ab is IL-23-specific, does not bind IL-12.

Calibrator material is a p19 subunit fused to the p40 subunit.

Company used calibrator material to generate the capture Ab.

Results from samples:

Kit gave higher than expected levels of IL-23 (based on previous publications and in-house data).

Case Study: IL-23 Luminex Commercial Kit

RJ Neely, BMS

10

©2016, Genentech

Troubleshooting: Compared Luminex kit to a later run, in-house BMS IL-23 assay

- Substantially different levels (<BLQ)

- Established 3rd party vendor assay agreed with the in-house, BMS assay.

Likely Issues:Luminex kit used a fused p19/p40 heterodimer as calibrator material - used to generate capture Ab

In-house and third party vendors used ex vivo formed

p19/p40 heterodimers in assay

- Fused p19/p40 heterodimer most likely different from endogenous

- Ex vivo material better mimics endogenous

Case Study: IL-23 Luminex Commercial Kit

Jennifer Postelnek, RJ Neely, BMS

Lessons Learned

•Calibrator material should be as close to the

endogenous form of the protein as possible

•Make sure the analyte biology supports the results generated.

Sample

LuminexOOR =

out of range

BMS

pg/mL

3rd

Party

Vendor

pg/mL pg/mL

1 283 <9.6 0.047

2 44.5 <9.6 0.111

3 7004 13.4 0.084

4 OOR < <9.6 0.033

5 102 <9.6 0.035

6 396 <9.6 0.065

7 60 <9.6 0.116

8 1529 <9.6 0.011

9 715 <9.6 0.247

10 OOR < <9.6 0.071

11

©2016, Genentech

• Vendors provide:

• Concentration of the protein

• Purity (such as a percentage, based on a silver stain or HPLC analysis)

• Source or origin (E. coli derived, for example)

• An accession number (providing basic information about the protein)

• Formulation (phosphate buffered saline, for example)

• Storage and stability information.

What Is Standard Practice for Protein Manufacturers?

• Vendors may, or may not, provide the method(s) used to assign a concentration to their protein product.

• These types of assays are typically plate-based colorimetric assays or absorption spectroscopy.

• As part of pre-purchase due diligence, information about how the vendor has assigned a concentration value to the protein produce should be gathered.

• Recommendation:

• Contact technical support key to understand this information, including why the vendor selected a particular method over another.

12

©2016, Genentech

FFP Table for calibrator material of protein biomarker assays• To define recommended extent of characterization depending on the intended use

of the biomarker assay data.

• To provide a consistent approach to aid in the selection and characterization of a calibrator

• To define what characterization criteria are "must" vs ”good to have” from vendors

• To provide a framework for how much characterization is "enough”

FFP Approach for Characterization Team recommendations for selection and characterization of biomarker assay protein calibrator material• Need to define “well-characterized”, for in-house and commercially obtained:

• What is the characterization of the endogenous, and what are acceptance/rejection criteria?

• What, if any, additional characterization should be done in-house if purchased?

• Risks associated with varying degrees of characterization.

Table recommends the bare minimum for what we recommend depending on the purpose of the data, regardless of whether the assay/material is purchased from a vendor or generated in-house.

• Caveat for table: highly dependent on whether the endogenous protein is well characterized.

13

©2016, Genentech

FFP Table: Identity

Recommendations for Biomarker Calibrator Material Characterization

Assay Type/Inten

ded Purpose

Relative quantitative assays used in exploratory

setting*

All quantitative assays used to justify dose, toshow safety or efficacy, and/or to support drug

registration**

Identity CoA from Producer

(external or internal)CoA plus Additional

Internal Characterization1

1 Including, but not limited to: Sequencing confirmation,

Peptide mapping, Mass Spec, or comparison to other family

members*“Exploratory” on next slides**“Treatment Decision-Making” on next slides

14

©2016, Genentech

FFP Table: Quantity and Purity

Recommendations for Biomarker Calibrator Material Characterization

Assay Type/Intended

PurposeExploratory Treatment Decision-Making

Quantity/Concentration

Carrier-free calibrators - A2802, BCA, Bradford, Performance in

Assay3, or CoA from Producer4

Carrier-protein containing calibrator - Performance in assay3, or CoA from Producer4

Purity, impurities and

contaminantsCoA from Producer4 Electrophoresis, or SEC, or Mass

Spec

2 Ensure proper extinction coefficient used3 Given there is one available; performance in orthogonal method optional4 Consider internal characterization (eg. SDS-PAGE/Electrophoresis) for new

vendors and for incomplete characterizations.

15

©2016, Genentech

FFP Table: Physicochemical, Expression

Recommendations for Biomarker Calibrator Material CharacterizationAssay

Type/Intended Purpose

Exploratory Treatment Decision-Making

Physicochemical properties5 CoA from Producer4 Electrophoresis, SEC, Mass Spec,

Maldi-TOF, DLS, and/or DSCExpression

SystemsMammalian or other6 Preferably mammalian expression

in appropriate cell types6

4 Consider internal characterization (eg. SDS-PAGE/Electrophoresis) for new

vendors and for incomplete characterizations.5 Molecular weight confirmation, post-translational modification, oligomerization,

folding, etc.6 Alternative expression systems may be deemed suitable depending on the

specific biomarker.

16

©2016, Genentech

FFP Table: Activity, Stability

Recommendations for Biomarker Calibrator Material Characterization

Assay Type/Intende

d PurposeExploratory Treatment Decision-Making

Biological activity

/Tertiary structure

Confirm activity7 or binding to Ab reagents

Confirm activity in an existing method7

Stability of calibrator

Short term (i.e. Freeze/Thaw, 4°C, Room Temperature, etc.)

Short & Long Term (multi-year -70°C)

7 If available; performance in orthogonal method if available.

17

©2016, Genentech

FFP Table: Parallelism, Lot-Lot Var.Recommendations for Biomarker Calibrator Material

CharacterizationAssay

Type/Intended Purpose

Exploratory Treatment Decision-Making

Parallelism Calibrator and endogenous

should pass with fewer samples tested (n > 3)

Calibrator should match endogenous analyte using more

samples (n > 10) 8, disease-

state samples required

Lot-to-lot variability

Determine comparability of new lot to previous lot. Normalize (if

needed) new material to old material or obtain a replacement

lot.

Test multiple lots from same producer to determine

variability. Consider value assignment.

8 If 10 samples are not available, scientists should consider the risks if fewer samples are tested. This number also applies to pre-clinical samples.

18

©2016, Genentech

Risk: Calibrator Mismatch to Endogenous

Risks/Impact Associated With Unacceptable Biomarker Calibrator Material

Assay Type/Intende

d PurposeExploratory Treatment Decision-Making

Impact

Calibrator performance significantly different than

endogenous biomarker, impacting data interpretation

and decisions based on data,

plus timelines, resources.

Data predicts incorrect dose. Patient safety impacted.

Incorrect efficacy

assessment. Therapeutic

approval affected.

19

©2016, Genentech

Additional Notes

Goal of FFP Table:

• Set threshold recommendations

• List risks associated with missing characterization of calibrator material, depending on intended use of biomarker assay data.

Must understand the biology of the analyte:

• To help ensure recombinant material will match endogenous analyte

Parallelism should always be performed early on• During method development

• Bridge surrogate matrix/recombinant protein vs. matrix with endogenous protein.

• Parallelism means calibrator and endogenous analyte behave in the same manner in the assay.

20

©2016, Genentech

BAPS: Biomarker Assay Protein calibrators Subteam

LBABFG Biomarker Discussion Group Need recommendations on calibrator selection, characterization

Need for an AAPS subteam to establish consensus to address challenges

Scope of Subteam Recommend best practices for bioanalytical scientists

FFP approach for selection, characterization, and assay acceptance.

Based on experience, challenges faced, important case studies, and relevant literature.

Define the important characteristics that a scientist should look for in identifying a reliable recombinant material.

Evaluate the choices for different types of calibrator protein depending on the challenges faced during assay development (from full-length proteins to truncated forms or peptides).

Introduce and incorporate appropriate commutability approaches used in clinical chemistry and approaches for available NIST standards into industry-wide recommendations.

21

©2016, Genentech

Commutability Approach

Definition: Any mathematical relationship between the results of different assays for a biomarker calibrator and for representative samples.

• Commutability is a property of the calibrator with respect to a defined set of assays and samples.

• Used in Clinical Chemistry settings to compare calibrators with samples.

• Any assay calibrator material from different sources, or even a different lot from same source, may not be commutable.

Importance of commutability:

• To manage a biomarker assay over years to support clinical trials

• To support assay on different platforms as needed

22

©2016, Genentech

Commutability MethodologyWe wanted to define:

• The appropriate mathematical relationships

• How to assess these relationships

Deming Residuals1. Plotted sample data with calibrator:

●

●●

●

●

●

●

●

●●

●

●●

●●

●

●●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

● ● ●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●●

●

●

●

●●●

●

●

●

0

5

10

15

0 5 10 15

Assay.1

Assay.2 Type

●

●

Reference

Sample

6 8 10 12 14

46

810

12

14

Deming Regression Fit

Method1

Me

tho

d2

Pearson's r = 0.845

The 0.95−confidence bounds are calculated with the analytical method.

Deming RegressionFit (n=100)

−0.68 + 1.05 * Method1identity

2. Fit Deming regression, and

compute Deming residuals:

23

©2016, Genentech

Commutability Methodology

● ●

●●

●●

●●

●●

●● ●

●●

●●

●

●●

●

●

●

●

●

●

●●

●●

●●

●●

●●

●●

● ●●●

● ●●●

●

●●● ●

● ●

●●

●●

●●

●

●

● ●●●

●

●

●

●●

●

●●●

●

●●

●●●

●●

●

●●●

●●

● ●●● ● ●

●

●●

●

●●

●

●●●

●

0.0

0.2

0.4

0.6

0.0 0.5 1.0 1.5 2.0

Residuals

de

nsity Data

●

●

Ref. Material

Samples

Outlier Screening via Deming residuals

DEMING RESIDUALS:

Using residuals, we can

determine a 99%

quantile based on

samples and examine

the location of the

calibrator material’s

Deming residuals.

Calibrator, at different

levels, fits within the

range of samples.

Calibrator is

commutable.

3. Estimate 99% cutpoint Deming residuals:

24

©2016, Genentech

Summary of RecommendationsRecommendations across proteins:

Know your endogenous protein

– Binding partners, PTM, multi-merization, etc.

FFP Table

– Baseline selection and characterization

– Dependent on intended use of data

– Conduct parallelism early

Commutability

– Suggested statistical approach for long-term

support of clinical assays: Deming residuals

25

©2016, Genentech

Acknowledgements

Cross-Industry:

Lakshmi Amaravadi

Mark Cameron

Damien Fink

Darshana Jani

Medha Kamat

Lindsay King

RJ Neely

Yan Ni

Paul Rhyne

Renee Riffon

Yuda Zhu

AAPS NBC Session Moderators

Yan Ni

Steve Piccoli

AAPS NBC Session Speakers

Dave Bunk

Hubert Vesper

Genentech BioAnalytical

Sciences:

Eric Wakshull

An Song

Patty Siguenza

26