microbial risk assessment: lessons learned and future directions greg paoli decisionalysis risk...
TRANSCRIPT
Microbial Risk Assessment: lessons learned and future
directions
Greg PaoliDecisionalysis Risk Consultants, Inc.
Ottawa Canada
One View of the Processing Stages in Risk Assessment
• Problem Selection• Model DM’s Values• Outcome Selection• Scope Selection• Tool Selection• Evidence Acquisition• Evid. Characterization• Model Development
• Diagnosis and Experimentation
• Validation-Seeking• Auditing• Documentation• Peer Review• Communication with
Risk Managers• Dissemination
Hazard Identification
• Managerial Hazards
• Scope Hazards
• Evidence Hazards
• Computational Hazards
• Characterization Hazards
• Communication Hazards
Managerial Hazards
• Silence regarding Values
• Valuation of Knowledge Gains
• Linear Processes
• Limited Tool Development
Scope Hazards
• “Keep it simple!”• “This is preposterously simple. You must include
the complexity or it can have no credibility!”• Scope decisions are qualitative risk assessments• Under-valuation of multi-use mitigations• Model resolution balance• Risk-risk tradeoff considerations
Evidence Hazards
• Numerical Hazards (Mean Log Example)
• Evidence regarding Process Deviations
• Regional and Temporal Issues
• Human Factors
• Cross-contamination
• Model Uncertainty
Mean Log Example
10000
1000
10000
1000
100000
10
10
10
22000100030
Risks Still Differ by a Multiple of 5
Characterizing the Extremes
• Illnesses result from combinations of rare events• We can expect to characterize most everyday
processes reasonably well.• The devil is in the tails
– no data on the magnitude and probability of deviations.
• Need to prove that this as a common and dominant phenomenon
Cross-Contamination: The Final Frontier
• Is it ‘unmodellable’ for the population?
• How big and black can a black box be?
• How do we incorporate and work with model elements for which our state of knowledge can be best described as ignorance?
Uncertainty Characterization
• If we express our uncertainty for most, but not all, variables or model assumptions, can we still say we have captured uncertainty.
• How much uncertainty is enough?
• Simple Answer: all of it:
• What’s the Real-Life Answer?
Computational Hazards
• Risk Estimate Stability (Rare Events)
• Auditability and Error-Proneness
• Transparency (Strict vs. Real)
• Time consumption (e.g. 2-D Models)
• Inability to Provide Real-Time Decision Support or Managerial Learning
• Lack of Diversity in Approach
Characterization Hazards
• Choice of Measures– Risk to Susceptibles– Population Risk– Risk per Serving/Preparation/Kilogram/Batch
• How Much Uncertainty has to be Included to Pass the “Uncertainty Test”
• Oversimplified Sensitivity Analysis
Communication Hazards
• Dissemination of Models for Review
• Expressing the Magnitude of Uncertainty
• How many audiences can we serve?
• Can stakeholders be meaningfully engaged in complex risk assessments?
Solution Sets
• Methodological Research– Tool Diversification– Modular System Characterization
• Linkages with Epidemiology
• Food Safety Objectives
Tool Diversification
• When all you have is a hammer …• Process Risk Models• Analytical Models• Bayesian Network Models• Expert Systems• Model Learning• Causal Models• Qualitative Risk Assessment• Module Librairies
Tool Diversification
• Within PRM Approaches …– Diversify software, and/or– Perform Needs Assessment– Performance Comparisons– Good Modelling Practice
Qualitative Risk Assessment
• Prone to Inferential Sloppiness– A literature search with conclusions?
• It is possible to impose structure, but its not always welcome
• Absence of an Inferential Trail
• We should be cautious about conferring the label “Risk Assessment” to anything that uses the right terminology.
Risk Modules not Risk Assessments
• Need to build microbial risk assessment ‘infrastructure’– Development away from the ‘bright lights’
– Carefully documented and computationally sound
– Examples of appropriate implementations
– Documented with limitations and caveats
– Shared library and shared experiences
• With this infrastructure, risk assessment can only get better, easier, more reliable
Linkages with Epidemiology
• Attributable Risk Problem
• Integration of Case-Control findings for sporadic cases
• Approaches for model validation
• Epidemiology re-thinking causality criteria
• Epi needs biological plausibility
• QRA needs epidemiological evidence
FSOs, Risk Measures
• Surrogate Variables
• Performance Indices
• Concrete Linkages to Objectives
• Qualitative Measures of Risk
• Farm-Level Measures
Process Characterization
• Distribution of Concentration across Units
• Prevalence of Contamination
• Indicator Levels• Competitive Flora• Homogeneity• Distribution of Strains
Present
• Information Provided with Food to Affect Later Handling
• Downstream Processing• Extent of Pooling• Growth Inhibitors• Packaging and Insulation• Information to Affect
Traceback and Recall• Target Consumer
A Tiered Approach to Methodology Development
• Tier 1:
• systems modelling– Multi-pathogen, indicators, sampling results,
responses to deviations
• Modular abstraction to simpler forms
• Modular integration tools
Tiered Approach (Tier 2)
• risk assessment– farm to fork– plant to fork– farm to plant– Supplier to purchaser
Tiered Approach … (Tier 3)
• risk-based expertise capture– A Risk-Based Process Inspector
• Formal, structured qualitative analysis– Advice replicates quantitative findings through
querying the problem in qualitative terms as well as some quantitative data.
In the Defense of Risk Assessment
• Bending over backwards to meet the demands for:– the best and up-to-date data– the best model and modelling technique– the best software implementation– good documentation of model– high-quality report– technical appendices for peer review– non-technical summaries– peer-reviewed publications– all at once, without a safety net
In the Defense of Risk Assessment: Are we Shooting the Messenger?
• Systems are too complex for human reasoning• Currently, Microbial Risk Assessors are
Methodological Researchers• Carefully formalizing some of the reasons why we
have not been successful in the past– We don’t acknowledge the complexity– We’ve never really understand the systems well enough
to control them reliably.
• Value of Information analysis sorely needed.