performance metrics for evaluating lng vapor …psc.tamu.edu/files/symposia/2009/presentations/2...
TRANSCRIPT
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 1 -
Performance MetricsPerformance Metrics For EvaluatingFor Evaluating
LNG Vapor Dispersion ModelsLNG Vapor Dispersion Modelsby
Frank A. Licari, PE, CSPUnited States Department of Transportation
Pipeline and Hazardous Materials Safety AdministrationPipeline Safety Office
Washington, DC
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 2 -
AgendaAgenda
• Historical Perspective of Metrics
• Novel Performance Metric - MSWC
• Methodology to Calculate MSWC
• Example Calculations
• Error Analyses & Their Importance
• Conclusions
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 3 -
Traditional Metrics ValidateTraditional Metrics Validate Vapor Dispersion Model PerformanceVapor Dispersion Model Performance
Model Comparisons – Hanna et al [2]
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 4 -
Historical Perspective of MetricsHistorical Perspective of Metrics
1980s Havens & Spicer DEGADIS
1993 Hanna et al Comparative Study
2001 Carissimo et al SMEDIS Validation
2004 Chang & Hanna Model Performance
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 5 -
Historical Perspective of MetricsHistorical Perspective of Metrics
• Are Valuable Tools To Validate Dispersion Models
• Traditional Statistical Methodologies:
– characterize strengths of models and
– identify their best applications
• Yet, Past Metrics Don’t Describe:
– extra separation distance that protects the public or
– additional confidence in a model’s predictions
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 6 -
New Metric DescribesNew Metric Describes ModelModel’’s Inherent Safety & Confidences Inherent Safety & Confidence
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 7 -
Novel Performance Metric Novel Performance Metric -- MSWCMSWC
• Margin of Safety With Confidence
– is a statistical tool
– quantifies model performance
– allows models to be compared
– describes model’s minimum margin of safety
– accurately describes confidence level of model predictions for 30+ data pairs
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 8 -
Novel Performance Metric Novel Performance Metric -- MSWCMSWC
• Margin of Safety for One Prediction Is
• Margin of Safety of a Vapor Dispersion Model Is a Range of Values Due to:
– atmospheric conditions
– local terrain
– test error
– modeling assumptions
– computational error
i
i
OP
iMs
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 9 -
Methodology to Calculate Methodology to Calculate MSWCMSWCLNG Test &
Atmospheric Stability (Pred/ObsRatio)
Burro 8 - E 0.716
1.462
0.798
0.683
Burro 9 - C 1.885
1.629
1.669
Maplin 29 - D 0.775
0.803
1.137
0.972
1.231
1.424
1.204
Maplin 39 - D 0.541
1.147
1.139
2.111
1.554
1.672
2.319
Table 1 – Excerpt of Havens 1992 Gas Concentration Ratios [5]
iMs • DEGADIS Dispersion Model Predictions
• Gas Concentration Data from LNG Field Tests
• 21 Data Ratios
• Range = .541 to 2.319
• = 1.28
x
x
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 10 -
Methodology to Calculate Methodology to Calculate MSWCMSWC
Histogram of in Table 1iMs
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 11 -
Methodology to Calculate Methodology to Calculate MSWCMSWCLNG Test &
Atmospheric Stability (Pred/ObsRatio)
Burro 8 - E 0.716
1.462
0.798
0.683
Burro 9 - C 1.885
1.629
1.669
Maplin 29 - D 0.775
0.803
1.137
0.972
1.231
1.424
1.204
Maplin 39 - D 0.541
1.147
1.139
2.111
1.554
1.672
2.319
Table 1 – Excerpt of Havens 1992 Gas Concentration Ratios [5]
iMs
= .49
= 1.0
= -.57
Confidence Level = 72%
desiredMs
MSWC = 1.0 with 72% Confidence
1
)Ms( 2i
1
n
xS
n
iMs
Ms
desired
SxMs
scoreZ
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 12 -Figure 1 - Determines and Confidence Level
= 1.281.0 =
desiredMs
desiredMs
desiredMs
scoreZ
72%
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 13 -
MSWC MSWC Explains HowExplains How
desiredMs
• Model’s Inherent Margin of Safety Is 1.0 or More
• 72 % of Gas Concentration Predictions Equal or Exceed LNG Field Trial Observations
• Models May Be Evaluated By Comparing Their
– inherent margins of safety (or safety buffers)
– confidence level (bias to over or under predict)
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 14 -
MSWCMSWC Explains ModelExplains Model’’s Accuracys Accuracy & Shapes Evaluation Decision& Shapes Evaluation Decision
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 15 -
MSWCMSWC Example for Distance PredictionsExample for Distance Predictions
• Explains Distance Prediction Concepts for Siting LNG Facilities
• Describes Importance of Societal Risk Preferences
• Calculates MSWC for 2 Geographic Regions
• Compares Regional Decisions to Accept DEGADIS Distance Predictions
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 16 -
Distance Prediction ConceptsDistance Prediction Concepts
100% LFL
50% LFL
• Site property line so hazards of flammable gas during an LNG spill remain in facility
• Gas concentration at 100 % of the lower flammability limit (LFL) is min. distance
• 50% LFL is NFPA 59A required distance
Distance to LNG Facility’s Property Line
Property Line
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 17 -
MSWCMSWC Example Example -- Societal Risk PreferencesSocietal Risk Preferences
• Property Line of LNG Facility May Extend to 100% LFL
• Region A Prefers Safety Buffer & = 1.5
• Region R Prefers No Safety Buffer & = 1.0
• Each Region Expects DEGADIS Predictions to Have High Confidence Levels
AdesiredMs
RdesiredMs
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 18 -
MSWCMSWC AA ExampleExample of Distance Predictionsof Distance PredictionsLNG Test & Atmospheric
Stability
Flammability Limit (LFL)
Observed
Distance (m)
Predicted
Distance (m)
iMs (Pred/Obs
Ratio)
Burro 8 - E 50% 700 550 0.786 Burro 9 - C 50% 480 700 1.458
Maplin 29 - D 50% 280 300 1.071 Maplin 39 - D 50% 230 400 1.739
Burro 8 - E 100% 360* 360* 1.000* Burro 9 - C 100% 240* 450* 1.875*
Maplin 29 - D 100% 150* 180* 1.200* Maplin 39 - D 100% 125* 220* 1.760*
MSWCA = 1.5 with 37% Confidence
Table A.1 – for Distances at 50 & 100 Percent LFLiMs
*data extrapolated from Figures 3 through 6 [5]
x MsS AdesiredMs= 1.36 = .40 = 1.5 = .34A
scoreZ
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 19 -
MSWCMSWC R R ExampleExample of Distance Predictionsof Distance PredictionsLNG Test & Atmospheric
Stability
Flammability Limit (LFL)
Observed
Distance (m)
Predicted
Distance (m)
iMs (Pred/Obs
Ratio)
Burro 8 - E 50% 700 550 0.786 Burro 9 - C 50% 480 700 1.458
Maplin 29 - D 50% 280 300 1.071 Maplin 39 - D 50% 230 400 1.739
Burro 8 - E 100% 360* 360* 1.000* Burro 9 - C 100% 240* 450* 1.875*
Maplin 29 - D 100% 150* 180* 1.200* Maplin 39 - D 100% 125* 220* 1.760*
MSWCR = 1.0 with 81% Confidence
Table A.1 – for Distances at 50 & 100 Percent LFLiMs
*data extrapolated from Figures 3 through 6 [5]
x MsS RdesiredMs= 1.36 = .40 = 1.0 = -.89 R
scoreZ
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 20 -Figure 2 - & Shape an Evaluation DecisionRMSWC AMSWC
= 1.5
1.36
of 37%
1.081%
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 21 -
FindingsFindings From Distance Prediction ExampleFrom Distance Prediction Example
desiredMs
• Siting a LNG Facility Property Line at 100% LFL Reduces Safety Buffer to Zero
• Desired, Minimum Margin of Safety Shapes Region’s Acceptance Decision
• Region A May Reject Model; It Overpredicts by Factor of 1.5 with 37% Confidence
• Region R May Accept Model, If Its Constituents Believe Safety Buffer is Unnecessary
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 22 -
What Is Size What Is Size of Modelof Model’’s Safety Buffer?s Safety Buffer?
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 23 -
MSWCMSWC DD of Exclusion Zone Predictionsof Exclusion Zone Predictions
desiredMs
• Describe Margin of Safety for an Exclusion Zone Prediction
• Calculate MSWC for Exclusion Zone Predictions
• Compare MSWC for Exclusion Zone Predictions to Distance Predictions
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 24 -
Exclusion Zone Creates Safety BufferExclusion Zone Creates Safety Buffer
desiredMs
100% LFL
50% LFL
During LNG Spill, Exclusion Zone at 50% LFL Separates Public from
Hazards of Flammable Gas at 100% LFL
Property Line
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 25 -
Part 193 Requires Margin of SafetyPart 193 Requires Margin of Safety
desiredMs
• 49CFR Part 193 & NFPA 59A (2001 edition) Establish LNG Facility Property Line at 50% LFL
• Safety Buffer Protecting Public Is Inherent Margin of Safety of 50 vs. 100% LFL
• Margin of Safety for DEGADIS Prediction of Exclusion Zone Distance Is:
LFLi
LFLi
OP
%100
%50DiMs
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 26 -
MSWC MSWC DD Example Example -- Exclusion Zone DistancesExclusion Zone Distances
MSWCD = 1.5 with 88% Confidence
Table A.1 – for Distances at 50 & 100 Percent LFLDiMs
*data extrapolated from Figures 3 through 6 [5]
x MsS AdesiredMs= 2.41 = .78 = 1.5 = -1.17A
scoreZ
LNG TestsAtmospheric
Stability
100% LFL Observed
Distance (m)*
50% LFL Predicted
Distance (m)(Pred/Obs
Ratio)
Burro 8 E 360 550 1.528
Burro 9 C 240 700 2.917
Maplin 29 D 150 300 2.000
Maplin 39 D 125 400 3.200
DiMs
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 27 -
MSWC MSWC DD Compared To Previous ExamplesCompared To Previous Examples
desiredMs
Examples Prediction
Inherent Margin
of SafetyConfidence Level (%)
Safety Buffer
MSWC D Exclusion Zone Distance 2.41 1.5 88 robust
MSWC R Distance 1.36 1.0 81 none
MSWC A Distance 1.36 1.5 37 inadequate
x
Societal Preferences Shape Model’s Acceptance
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 28 -
What Is ErrorWhat Is Error In In MSWCMSWC’’’’s Confidence Level?s Confidence Level?
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 29 -
MSWCMSWC Error Analyses & Their ImportanceError Analyses & Their Importance
desiredMs
• Sample Sizes in Previous Examples Are Small
• Calculations Contain Some Statistical Error
• Large Datasets with 30+ Minimize Error
• Error Analyses Characterize MSWC’s Accuracy
iMs
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 30 -
Analysis of Analysis of EE GG, Standard Error of Mean, Standard Error of MeanLNG Test &
Atmospheric Stability (Pred/ObsRatio)
Burro 8 - E 0.716
1.462
0.798
0.683
Burro 9 - C 1.885
1.629
1.669
Maplin 29 - D 0.775
0.803
1.137
0.972
1.231
1.424
1.204
Maplin 39 - D 0.541
1.147
1.139
2.111
1.554
1.672
2.319
Table A.3 – Excerpt of Havens 1992 Gas Concentration Ratios [5]
= 1.28 = .49
= 1.0 = -.57
= 1.725 at 90% = 21
= .184
or = = 1.10 or 1.46GG Ex min max
)(2/ GMsGG
n
StE G
Gt 2/
GMsSGxGdesiredMs G
scoreZ
Gn
GiMs
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 31 -
GMSWC
= 1.28= .18
1.0 =
= 1.46
1.1
of 83%
of 58%
Figure A.6 - With Min. & Max. Confidence Limits
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 32 -
Confidence Level Accuracy for Small SampleConfidence Level Accuracy for Small Sample
desiredMs
• Error in Confidence Level Is Estimated:
-.20 or -.95
• & Respectively Indicate Confidence Levels of 58 & 83%
• MSWCG Is 1.0 with 72% Confidence with Approximate Errors of -14 and +11%
GMs
GG
s SE
scoreG
max scoreG
min core Zor ZZ
Gmin coreZs
Gmax scoreZ
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 33 -
Effective Performance Metrics Effective Performance Metrics Ensure Prudent Evaluation DecisionsEnsure Prudent Evaluation Decisions
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 34 -
ConclusionsConclusions
• Uncertainties Must Be Reconciled As Predictions & Observations Are Correlated
• Model Predictions Vary By Factor of 2 Due to “Natural & Stochastic Variability” [2]
• Screening Predictions & Observations Guides Model Validation Process [3]
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 35 -
ConclusionsConclusions
• Geometric Mean Bias & Geometric Variance Graphs Readily Compare Model Performance [2]
• Fractional Results ( % between .5 & 2) Identify Best Applications for Models [4]
• Larger Validation Datasets Favor New Performance Metrics Like MSWC
Together All Metrics Balance Evaluation Decisions
iMs
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 36 -
Thank You!Thank You!
Frank A. Licari, PE, CSPPhone: (202) 366-5162Email: [email protected]://www.phmsa.dot.gov/pipeline
Questions
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 37 -
Backup SlidesBackup Slides
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 38 -
Why Is Confidence Level 37%?Why Is Confidence Level 37%?
• Inherent Margin of Safety Is Zero
• Model’s Bias to Overpredict Is Low
• Hanna Concluded Good Models Are Within Factor of .5 to 2
Mary Kay O’Connor Process Safety Center2009 International Symposium
College Station, Texas
- 39 -
Confidence Level Accuracy For Large SampleConfidence Level Accuracy For Large Sample
desiredMs
• Confidence Level Error Is Estimated By:
• Standard Error of Is:
• For 30+ , &
Yields Confidence Level Error for Large Sample
nZ
SMs
Ms
21 2/
max
nZ
SMs
Ms
21 2/
min
iMs
minin ty variabili scoreZ
Ms
MsdesiredMsMs
maxMs
MsdesiredMs
or
MsS
Msx MsMsin ty variabili score
Z