aseptic site risk assessment tool” - pqripqri.org/wp-content/uploads/2015/11/baumstein2.pdf•...

Challenges and Solutions for the Quantification and Mitigation of Risk When

Assessing Global Aseptic Manufacturing Operations

“Aseptic Site Risk Assessment Tool” Presented by:

Michael Baumstein

Global Quality Operations

Microbiology and Aseptic Support (MAS) PQRI: September 17, 2014

Overview

• This slide deck is meant to provide a high-level overview

of the Aseptic Site Risk Assessment Tool (ASRAT)

process:

– Purpose & Scope

– ASRAT Process Flow

– Fundamental principals including benefits and limitations

– An explanation of the risk ratings and data analysis

• The MAS team currently is responsible for facilitating

both data collection and subsequent report generation:

– The MAS Team facilitates the process and provides

support to address any specific questions or comments

related to the program

What is the Purpose & Scope?

• Develop a tool to assess general aseptic manufacturing risk by site /

area.

• Designed to be simple, easy to use, applied consistently across the

network and account for site variability.

• Any tool should be able to assess risk based upon available internal

metric data.

• Even though questions posed may be retrospective (i.e. trailing

indicators) they should reflect prospective future risk.

• Can be used for indicated risk mitigation by sites with support by

MAS and other technical functions as necessary.

• This is a supplementary tool to enhance risk mitigation information

already available to sites through their current Quality Assurance /

Control process and procedures.

• ASRAT metrics are only one measure to consider the state of

site compliance to internal as well as external expectations.

Some Key Process Fundamentals …

• The risk algorithm employed was discussed and endorsed by the ASRAT Steering Team.

• Important to gain clarity and consensus on scope, risk scoring, and assignment of overall risk status.

• Process for data gathering needed to be efficient: 1. MAS request data from the sites (~30 days after close of the quarter)

2. Sites provided ample time to populate quarterly data (1-3 weeks)

3. MAS reviews and summarizes data (~1 week)a

4. MAS generates final report to Quality Leadership (< 1 week)b

• MAS shares draft quarterly data with the ASRAT Steering Teama.

• Final quarter-over-quarter progress report distributed to and results are shared with both site quality leaders and the Quality Operations Leadership Team (QOLT)b.

• Mitigation plan / action items are defined and discussed with ASRAT Steering Team.

What are the Key Process Activities?

• MAS posts updated spreadsheet file that is designed for shared access by a site data entry designee.

• Site designee enters data values into the file which automatically calculates program category metrics and applies risk threshold levels.

• Site specific data entries and calculations may be reviewed anytime during the open data entry period.

• MAS generates a summary report based upon data entered into the spreadsheet file.

• MAS reviews and finalizes the compiled summary report with input from the ASRAT Steering Team.

• Finalized report and any suggested mitigation(s) are approved by the ASRAT Steering Team.

How Does the Process Flow Look?

The Site’s Data Entry Form …

The Data Entry Form … continued

What Does the Data Look Like?

How are Results Defined?

• Not all aseptic sites are equivalent…for example: – Different products, processes, equipment and facilities are found

throughout the network reflecting differing levels of potential risk.

• Ratings are not an absolute measure of site risk, rather they

are relative risk rankings.

• Risk thresholds have been set in a balanced manner in order

to better accommodate site differences.

• Metric data typically includes the total number of samples

collected and the number of deviation (i.e. Action Level)

events. – EM, Personnel Monitoring, WFI Testing and Pre-Filtration Bioburden

• ASRAT emphasizes trended / extended information not

specific deviations which are handled through established site

quality operation process / procedures.

So What are the Assessed Categories?

1. Process Performance – Media Fills

– Sterility Testing

– Pre-Filtration Bioburden

– Water (WFI) Testing

2. Environmental Performance – Microbiological Environmental Monitoring within Grade A & B areas

– Microbiological Environmental Monitoring within Grade C & D areas

– Surface

– Passive Air

– Active Air

– Microbiological Personnel Monitoring

There are two main topic indicators that are considered

and assessed using metric data within the ASRAT:

Additional ASRAT Process Details …

• Site-level risk ratings (i.e. Acceptable, Management Awareness,

or Management Alert) are assigned each quarter.

• Ratings are based in-part on the previous quarter’s overall ranking

as well as current results in each defined program category.

• Program categories are grouped by weighted risk as follows:

• Higher Risk Activities: Media Fill failures, Sterility Test failures

• Medium Risk Activities: Grade A/B EM Action Level excursions, Personnel

Monitoring Action Level excursions, WFI Test failures

• Lower Risk Activities: Pre-Filtration Bioburden Test failures, Grade C/D EM

Action Level excursions

• Higher Risk Activities have absolute risk thresholds (e.g. whole

number value assignments).

• Medium and Lower Risk Activities have normalized proportional

(%) thresholds based on historical performance in the network.

An Explanation on Risk Thresholds …

Risk Thresholds and how the ASRAT Deals with Site-

to-Site Variability:

• Formulated risk ratings are not an absolute measure of site risk.

– These are a combination of absolute risk from representative

program categories, normalized risk (percentage change for

medium and lower risk program categories) and the relative ranking

from previous quarter(s).

• Normalized risk thresholds are calculated based on 85th and 95th

data percentiles and updated (~2 years).

• Overall site risk rating is based on program category risk value and

weight as well as historical quarterly performance results.

• Data is reviewed quarterly by the dedicated ASRAT Steering Team

to provide additional details on the findings to support final site risk

rating assignments.

Assigned Risk Level Examples …

RISK LEVELS

PROGRAM Category Acceptable Management

Awareness

Management

Alert

Media Fill (Failures) 0 Failures 1 Failure >1 Failure

Media Fill (Positive Units) 0 Positives 1 Positive >1 Positive

Sterility Test (Failures) 0 Failures 1 Failure >1 Failure

Environmental Monitoring, Grades A/B Air

& Surface (% exceeding Action Level) <0.34% 0.34 to 0.7% >0.7%

Environmental Monitoring, Grades C/D Air

& Surface (% exceeding Action Level) <0.67% 0.67 to 1.7% >1.7%

Environmental Monitoring, Personnel

Monitoring on exit (% exceeding Action

Level)

<0.47% 0.47 to 1.1% >1.1%

Water (WFI) Testing (% exceeding Action

Level) <0.62% 0.62 to 2.0% >2.0%

Pre-Filtration Bioburden (% exceeding

Action Level) <0.6% 0.6 to 3.2% >3.2%

Further Detail on Risk Rating Rationale

• A Management Awareness (yellow) or Management Alert

(red) rating in a single medium or lower risk activity program

category will not cause the overall site level rating to move to

the Management Alert (red) risk level, regardless of the

previous quarter’s rating.

• However, a failure in a higher risk activity program category

(e.g. Media Fill with >1 positive unit or >1 Sterility Test

failure) or a combination of multiple medium and lower risk

activity program categories at Management Awareness

(yellow) or Management Alert (red) can result in an overall

Management Alert site level rating, even if the previous

quarter was Acceptable.

Example #1: Result Interpretation

• Overall rating from previous quarter was Acceptable (green).

• One Management Awareness (yellow) in a lower risk activity (Grade C/D EM).

• Overall rating for this quarter is Acceptable (green).

0

0

0

0.2%

1.1%

0.5%

0.0%

0.0%

Media Fill (Failures)

Media Fill (Positive Units)

Sterility Test (Failures)

Environmental Monitoring, Grades A/B Air & Surface (% exceeding Action Level)

Environmental Monitoring, Grades C/D Air & Surface (% exceeding Action Level)

Environmental Monitoring, Personnel Monitoring (% exceeding Action Level)

Water (WFI) Testing (% exceeding Action Level)

Pre-Filtration Bioburden (% exceeding Action Level)

0

0

0

0.1%

0.5%

0.3%

0.0%

0.0%


• Overall rating from previous quarter was Acceptable (green).

• Two Management Awareness (yellow) in a higher risk activity (Media Fill).

• Overall rating for this quarter is Management Awareness (yellow).









0

0

0

0.2%

0.7%

0.1%

0.0%

0.0%

1

1

0

0.2%

0.3%

0.1%

0.0%

1.2%


• Overall rating from previous quarter was Management Awareness (yellow).

• One Management Awareness (yellow) in a higher risk activity (Sterility Test), two Management Alerts

(red) in medium risk activities (Grade A/B EM and Personnel Monitoring) and two Management Alerts

(red) in higher risk activities (Media Fill).

• Overall rating for this quarter is Management Alert (red).









0

0

2

2.7%

0.0%

20.6%

5.0%

0.0%

2

220

1

5.6%

0.0%

10.5%

0.0%

0.0%

Report - Site Summary Results

Report – Category Result Summary

Report – Overall Site Rankings

Site

2QYear3 Data

Review

1QYear3 Data

Review

4QYear2 Data

Review

3QYear2 Data

Review

2QYear2 Data

Review

1QYear2 Data

Review

4QYear1 Data

Review

3QYear1 Data

Review

Site 1

Acceptable. Management Awareness. No

improvement in C/D EM results

and increased WFI test failure

rate.

Acceptable. Improved A/B EM

results. Watch C/D EM results.

Management Awareness. High

A/B and C/D EM results.

Acceptable. Watch C/D EM data

for return to poor trend.

Acceptable. No action. C/D EM

results have improved.

Acceptable. No action. Watch

C/D EM data for poor trend in next

Q.

Acceptable. No action. Watch

C/D EM data for poor trend in next

Q.

Site 2 Acceptable. Acceptable. Pre-filtration

bioburden continues increasing

trend.

Acceptable. Watch pre-filtration

bioburden for increasing trend.

Acceptable Acceptable Management Awareness. Media

fill failure.

Acceptable. Follow up on MSQA

observation.

Acceptable

Site 3 Acceptable. No data. Shutdown

for upgrade until 2013.

Acceptable. No data. Shutdown


Acceptable. No data. Shutdown


Site 4

Acceptable. Acceptable. Watch pre-filtration

bioburden for increasing trend.

Management Awareness. High

personnel monitoring results.

Acceptable Management Awareness.

Continued high personnel

monitoring results.

Management Awareness. Sterility

test failure and high personnel

monitoring results.

Acceptable. Dona to follow up on

PM and A/B EM data.

Uncertain. Need info on current

processes and specs. Sterile or

low bioburden? Regulatory

commitments? Dona to follow up.

Site 5

Management Awareness.

Increased personnel monitoring

results.

Acceptable. No sterility test

failures this quarter.

Management Alert. Finished

product sterility test failures (1).

In-process sterility test failures (5).



In-process sterility test failures (3).

Management Awareness. WFI

test failures this quarter.

Management Alert. Continued

sterility test failures, but trend is

improving.


Concern with sterility test results.

Bill to contact Gustavo Scaffa, cc:

Chris Smith.


Concern with sterility assurance.

Bill to contact Gustavo Scaffa, cc:

Chris Smith.

Site 6 Acceptable. Acceptable. Acceptable. Acceptable. High C/D EM results. Acceptable. Management Awareness. Two

media fill failures this quarter.

Acceptable Management Awareness. Paolo

working with site to resolve media

fill positives.

Site 7

Management Alert. High A/B EM

and personnel monitoring results.



High A/B EM and personnel

monitoring results.

Acceptable. However EM results

not available by grade. Will be

corrected with implementation of

MODA system. Watch high in-

process sterility test failures.

Management Awareness. Finished

product sterility test failure (1).In-

process sterility test failures (287).

EM results not available by grade.

Will be corrected with

implementation of MODA system.




MODA system.




MODA system.

Uncertain. Paul will follow up on

missing EM and PM data.

Management Awareness. Paul

working with site to resolve issues.

Site 8 Acceptable. Acceptable. Acceptable. Acceptable. Acceptable. Acceptable Acceptable Acceptable

Site 9 Management Awareness.


results.



results.

Acceptable. Acceptable. Acceptable. Management Awareness.

Continued media fill failures.

Management Awareness. Dona to

review media fill failure.

Acceptable

Site 10 Acceptable. Acceptable. Acceptable. Acceptable. Acceptable. Acceptable Acceptable Acceptable

Site 11 Acceptable. Acceptable. Acceptable. Acceptable. Acceptable. Uncertain. No data. Uncertain. All zero results raise

doubts. Stew to review MSQA

audit.

Uncertain.

Site 12 Acceptable. Acceptable. Acceptable. Acceptable. Acceptable. Uncertain. Uncertain. Uncertain.

Site 13

Acceptable. Acceptable. Acceptable. Acceptable. Acceptable. Uncertain. All zero results raise

doubts. Bill to contact Uma.

Review EM program?

Uncertain. All zero results raise


Review EM program?



Review EM program?

Site 14

Acceptable. Acceptable. Acceptable. Management Awareness. Positive

unit (1) in finished product media

fill and high C/D EM results.

Acceptable. Acceptable Acceptable. Follow up on existing

programs satisfactory.


doubts. Stew to review EM

program.

Site 15

Management Awareness. Positive

unit (1) in finished product media

fill.

Acceptable. Acceptable. Acceptable. Acceptable. Acceptable. Watch media fill

results next quarter.

Acceptable Acceptable

Use of Data Trending Visual Cues ...

Trend

Indicator:

J = Improving

K = No Change

L = Worse

PROGRAM Category Site A Site B

Media Fill (Failures) 0 K 0 K

Media Fill (Positive Units) 0 K 0 K

Sterility Test (Failures) 0 K 3 L

Environmental Monitoring, Grades A/B Air & Surface

(% exceeding Action Level) 0.22% J 0.44% L

Environmental Monitoring, Grades C/D Air & Surface

(% exceeding Action Level) 0.82% J 1.47% L

Environmental Monitoring, Personnel Monitoring on

exit (% exceeding Action Level) 0.82% L 0.09% J

Water (WFI) Testing (% exceeding Action Level) 2.05% L 0.05% K

Pre-Filtration Bioburden (% exceeding Action Level) 0.77% J 0.27% J

Some Final Thoughts on ASRAT …

• The program’s initial development to actual

implementation took about a year.

• Has served as a novel way to assess prospective future

risk beyond simple reactionary limits (i.e. Alert/Action

levels).

• Provides additional management awareness to site

performance and overall site “health”.

• Adaptability via risk threshold baseline adjustments.

• Active site engagement makes this process at times

challenging, certainly value added and most importantly

provides for continuous site improvement

Any Questions … ?

Thank You for Your Attention

E-mail: [email protected]

aseptic site risk assessment tool” - pqripqri.org/wp-content/uploads/2015/11/baumstein2.pdf•...

Documents