lable at ScienceDirect

Journal of Loss Prevention in the Process Industries 23 (2010) 745e752

Contents lists avai

Journal of Loss Prevention in the Process Industries

journal homepage: www.elsevier .com/locate/ j lp

Performance metrics for evaluating liquefied natural gas, vapordispersion models

Frank A. Licari a,b,*aUnited States Department of Transportation, Pipeline and Hazardous Materials Safety Administration, Pipeline Safety Office, Washington, DC, USAbMary Kay O’Connor Process Safety Center, 2009 International Symposium, College Station, TX, USA

a r t i c l e i n f o

Article history:Received 4 March 2010Accepted 3 May 2010

Keywords:Performance metricsLNG plant sitingVapor dispersion modelsExclusion zonesMargin of safetyConfidence level

* United States Department of Transportation, PipeSafety Administration, Pipeline Safety Office, Washing5612; fax: þ1 202 366 4566.

E-mail address: licarifa@usa.com1 In this paper, margin of safety is an occupatio

expressed as a ratio.2 DEGADIS is a dense gas, vapor dispersion mo

collaboration with the Gas Research Institute and theUnited States Department of Transportation adoptedsiting regulations within Part 193 of the 49 Code of F

0950-4230/$ e see front matter Published by Elseviedoi:10.1016/j.jlp.2010.05.002

a b s t r a c t

New performance metrics are necessary to quantify the inherent margins of safety1 in vapor dispersionmodels for liquefied natural gas (LNG) spills. Currently, vapor dispersion model calculations in the 49Code of Federal Regulations, Part 193 as well as Standard 59A of the National Fire Protection Association(2001 edition) reduce the lower flammability limit (LFL) of methane in air by a safety factor of two (to50% LFL) to ensure that flammable vapors do not extend beyond an LNG facility’s property line during anLNG spill. Yet, neither document explicitly states the additional distance or the additional confidencelevel this existing safety standard creates to separate the public from LNG vapors at 100 percent LFLwithin the facility vs. 50 percent LFL at the facility property line.

Although researchers have successfully validated how vapor dispersion models calculate conservativebuffer (exclusion) zones, their collective work did not readily explain to the general public the inherentmargins of safety in these models. Havens and Spicer developed correlations to demonstrate how wellDEGADIS2 predictions compared with field testing measurements in the late 80s (Havens & Spicer, 1985).Their research also confirmed that peak gas concentrations exceeded time averaged measurementsduring some field trials as well as DEGADIS predictions. Then Hanna, Chang, and Strimaitis (1993)explained how several vapor dispersion models could be compared by calculating geometric meanbias and geometric variance and shared these validation results with the public. The works of the Havensand Hanna teams were also influential in explaining why the maximum concentration of methane in airat the property limits of an LNG facility should be 50 percent of its lower flammability limit during anLNG spill. Eleven years later, Chang and Hanna discussed how the relationships between fractional bias,geometric mean bias, geometric variance, and normalized mean square error could explain vapordispersion model over and under prediction (Chang & Hanna, 2004). Despite these successful efforts,there has been reluctance to embrace vapor dispersion model results, because exclusion zones are notdescribed as creating margins of safety (i.e. additional separation distance) or higher confidence levels(i.e. a likelihood of being correct) that protect the public.

This paper proposes an improved performance metric to evaluate the validity of vapor dispersionmodels and a statistical methodology to determine the confidence level and the inherent margin ofsafety in calculating vapor dispersion exclusion zones. Descriptions of the new metric and methodologyare presented in this document for the DEGADIS vapor dispersion model, together with examplecalculations.

Published by Elsevier Ltd.

line and Hazardous Materialston, DC, USA. Tel.: þ1 202 366

nal safety phrase, and it is

del that was developed inUniversity of Arkansas. TheDEGADIS in its LNG facilityederal Regulations.

r Ltd.

1. Introduction

A new performance metric and a corresponding statisticalmethodology are presented in this paper, which may be helpful inevaluating dispersion models and explaining the results of a valida-tion exercise. However, this new metric and its methodology bythemselves do not constitute a comprehensive model validation orevaluation. They, like all performance metrics, are purely anothermeans to judge whether an LNG vapor dispersion model predicts

3 The 49 CFR Part 193 selectively incorporates portions of the 2001 edition of theNational Fire Protection Association Standard 59A (NFPA 59A), for calculating vapordispersion exclusion zones at LNG facilities.

4 The histogram in Fig. A.5 indicates the data set in Table 1 resembles a normaldistribution.

F.A. Licari / Journal of Loss Prevention in the Process Industries 23 (2010) 745e752746

exclusion zone distances (or gas concentrations) with an adequatemargin of safetyand anacceptable level of confidence. The remainderof this paper describes howperformancemetrics for vapordispersionmodels have evolved in recent years and how a new metriccomplements prior research in this field.

2. Historical perspective of performance metricsfor vapor dispersion models

In the late 1980s, Havens and Spicer began the first efforts tovalidate LNG vapor dispersion models by developing statisticallymeaningful correlations between DEGADIS predictions and fieldtest observations (Havens & Spicer, 1985). They evaluated resultsfrom LNG tests conducted by the American Gas Association andShell, and in their work to document the strength of DEGADIScorrelations, they also illustrated how peak, gas concentrationmeasurements exceeded time averaged results inmany tests. Thesefindings acknowledged the challenges in creating correlations withhigh confidence levels as well as the potential uncertainties forpredicting flammable gas concentrations.

Then in 1993, Hanna et al. performed comparative validations offifteenmodels (Hanna et al., 1993). Their research constructed plotsof the geometric mean variance vs. geometric mean bias for gasconcentrations of vapor dispersion models and created visualrepresentations of each model’s variance within a 95 percentconfidence level and its range of over and under prediction. Thesemodels were then grouped by instantaneous, passive, and contin-uous releases to depict the strengths of each model’s predictionsagainst observed gas concentrations during field tests. Hanna et al.clearly illustrated how each model’s performance in predicting gasconcentrations typically falls within a factor of two compared toactual field observations. Using the ratios of gas concentrationpredictions to observed measurements, residual plots of HGSYTEMwere created to illustrate trends related to distance, wind speed,and atmospheric stability. In their conclusions, the authors appro-priately acknowledged that the performance of vapor dispersionmodels could vary by 50 percent from one site to another due to“natural or stochastic variability in the atmospheric diffusionphenomena.” Another milestone in the application of performancemetrics and the validation of vapor dispersion models had beenachieved.

When Europe completed its SMEDIS (Scientific Model Evalua-tion of Dense Gas Dispersion Models) project, Carissimo et al.published its evaluation of twenty eight vapor dispersion models in2001 (Carissimo et al., 2001). This report utilized the performancemetrics which Hanna et al. had previously established and alsoproposed that vapor dispersion models be evaluated by theirfractional results e the percentage of gas concentration predictionsbetween 50 and 200 percent of observed test measurements(a factor of ½ to 2 of observed results). Using these performancemetrics, the authors concluded integral models, like DEGADIS,performed well with no complex effects [boundary conditions likesevere terrain and obstacles] to mitigate or aggravate normalatmospheric dispersion. Furthermore, all models were better atpredicting arcwise results [rather than pointwise] corresponding tocenterline, maximum gas concentrations. Thus, SMEDISresearchers confirmed conventional wisdom that simple integralmodels would be valuable tools in calculating vapor dispersionexclusion zones without complex effects.

In 2004, Chang and Hanna completed an extensive study ofstatistical techniques for evaluating air quality models (Chang &Hanna, 2004). They confirmed commonly held views that goodvapor dispersion models have 50 percent of their vapor dispersionpredictions within a factor of ½ and 2 of observed test measure-ments. Chang and Hanna recognized thatmodels may be tailored to

specific vapor dispersion applications and explained at length howstatistical techniques should be selected to complement thecorrelation of test results to air quality model predictions. Properlyunderstanding and characterizing the relationships betweenpredictions and observed test measurements were critical toselecting an appropriate performance metric for the task. In theirview, screening the predictions and observed measurements andperforming exploratory data analyses were essential tasks inidentifying a reasonable performance metric. Knowing whetherobserved results were within a factor of two, five, or ten ofpredictions was an elemental step in the model validation process.Using data sets with at least twenty data points was equallyimportant to achieve statistically valid conclusions.

To date, performance metrics for evaluating vapor dispersionmodels have been valuable scientific tools for validating how wellthese models predict gas concentrations measured during fieldtrials. Researchers have developed statistical and analytical meth-odologies which accurately characterize the strengths of numerousmodels and describe how such models should be used. However,regulatory agencies and the public require a slightly differentperformance metric to explain how well vapor dispersion modelssafeguard communities from LNG vapors at 100 percent LFL withinthe facility vs. 50 percent LFL at the facility property line. A newperformance metric and statistical methodology is needed todescribe the additional distance and/or confidence level that anLNG vapor dispersion exclusion zone creates to separate the publicfrom the hazards of flammable LNG vapors.

3. Novel performance metric for evaluating LNGvapor dispersion predictions

Considering the limitations of existing performance metrics forLNG vapor dispersion models, this paper proposes a new metricand statistical methodology for calculating vapor dispersionpredictions and exclusion zones at an LNG facility. To his credit,Havens developed ratios of predicted to observed gas concentra-tions for the Burros and Maplin Sands LNG tests (Havens, 1992).However, Haven’s work did not determine the minimummargin ofsafety and corresponding confidence level that NFPA 59A3 and 49CFR Part 193 create, when a vapor dispersion exclusion zoneextends beyond 100 percent LFL to 50 percent LFL at an LNGfacility’s property line. The notional concept of a minimum marginof safety with confidence (MSWC) for a vapor dispersion model andits predictions is explained here and applied in several examples.

In principle, a margin of safety (Msi) for a dispersion modelprediction may be defined by the simple ratio of the prediction(distance or gas concentration) to the observedmeasurement duringa field trial (reference Eq. (1) in the Appendix). However, numerousLNG spill tests have confirmed that atmospheric conditions, localterrain, and testing error create a significant variance in the accuracyofmeasuring a flammable gas concentration at an observed distance.Furthermore, the inherent modeling and computational errors thatare intrinsic to both source term and vapor dispersion modelsprovide additional error. Consequently, the concept of a dispersionmodel’smarginof safety is no longer a single value. On the contrary, itbecomes a range ofMsi values similar toTable 1 (refer toHavens,1992paper andTable 10) thatmaybe illustratedby the normaldistributioncurve in Fig. 1.4

Table 1Excerpt of Havens (1992) gas concentration ratios.

LNG test & atmospheric stability Msi (Pred/Obs ratio)

Burro 8 e E 0.7161.4620.7980.683

Burro 9 e C 1.8851.6291.669

Maplin 29 e D 0.7750.8031.1370.9721.2311.4241.204

Maplin 39 e D 0.5411.1471.1392.1111.5541.6722.319

Fig. 1. Msdesired determines Zscore and confidence level.

F.A. Licari / Journal of Loss Prevention in the Process Industries 23 (2010) 745e752 747

For this example of 21 data ratios (n), the DEGADIS modelunderpredicted actual test measurements seven times and over-predicted fourteen times. Assuming this data set corresponds toa normal distribution, one could estimate the standard deviation(SMs) and sample mean ðxÞ for this data set, and they would be 0.49and 1.28 respectively.5 Calculating the Zscore for this data setdetermines that the desired minimum margin of safety (Msdesired)is Zscore standard deviations from the sample’smean ðxÞ, as depictedin Fig. 1. If a community deemed an Msdesired of 1.0 to be adequate,then Zscore is �0.57.

By estimating the shaded area beneath the standard normaldistribution curve in Fig. 1, the confidence level associated withMsdesired of 1.0, may be determined for the Msi data. Msdesired witha corresponding confidence level could then be deemeda minimum margin of safety with confidence (MSWC) for the dataset. For the above data and a Zscore of �0.57, there is a 72 percentconfidence level that the minimum margin of safety of a DEGADISgas concentration prediction will exceed 1.0 times a correspondingtest observation. In more concise terms, theMSWC for the DEGADISpredictions in Table 1 is 1.0 with a confidence level of 72 percent.

In principle, this improved performance metric, MSWC, shouldbe determined during the last step of a validation process. Thiswould allow researchers to perform sensitivity and uncertaintyanalyses of model predictions and to understand the strengths andweaknesses of an LNG vapor dispersion model. This groundworkwould ensure an appropriate data set is chosen to determinea model’s MSWC.

4. MSWC calculations for vapor dispersion modelsand distance predictions

The value of this methodology is further explained in a hypo-thetical example that compares the safety perceptions of twogeographic regions, their decisions to accept an LNG model and itsdistance predictions (reference Table A.1), and the benefit ofestablishing an LNG facility property line at 50 percent LFL ratherthan 100 percent LFL. In this scenario, both regions may establishthe property line of an LNG facility where LNG vapors will ignite at

5 Reference the Appendix for a detailed explanation of the statistical basis forthese calculations.

100 percent LFL. Each region also expects DEGADIS distancepredictions to agreewith observed fieldmeasurements without theinherent margin of safety created in 49 CFR Part 193 (a 50 percentLFL limit at the facility property line).

However, Region A and Region R have different perceptions ofsafety and risk, so the MsdesiredA for Region A is assumed to be 1.5,andMsdesiredR for Region R is 1.0. Accordingly, Z scores are calculatedand indicate that MsdesiredA is 0.34 standard deviations from x, andMsdesiredR is �0.89 standard deviations away. As the shaded area inFig. 2 illustrates, there is a 37 percent confidence level that theminimum margin of safety of a DEGADIS distance prediction forRegion A will exceed 1.5 times a corresponding test observation. Insimpler terms, the DEGADIS distance prediction for Region A hasaMSWCA of 1.5 with a 37 percent confidence level, while Region R’sMSWCR is 1.0 with an 81 percent confidence level.

These findings are important to regulatory agencies and emer-gency response officials, because the minimum margin of safetydesired by each region shapes its respective decision to accept(or reject) a vapor dispersion model and its distance prediction toan LNG facility property line. Allowing flammable vapors at 100percent LFL to extend to a facility property line reduces to zero thesafety buffer (the inherent margin of safety in a DEGADIS exclusionzone calculated per 49 CFR Part 193) between the public and thehazards of flammable gases. For Region R and a MsdesiredR of 1.0, itwould be 81 percent confident that DEGADIS would overpredict bya factor of 1.0 or more the distance to a facility property line whereflammable vapors reach 100 percent LFL. Region R would also be 19

Fig. 2. MSWCR & MSWCA shape an evaluation decision.

F.A. Licari / Journal of Loss Prevention in the Process Industries 23 (2010) 745e752748

percent confident that DEGADIS will underpredict the distance toa facility property line, exposing the public to the hazards offlammable vapors. However for Region A and a MsdesiredA of 1.5, itwould be only 37 percent confident DEGADIS would overpredict bya factor of 1.5 or more the distance to an LNG facility property line.

Considering the disparate safety perceptions in thesegeographic areas, Region R might accept the DEGADIS vapordispersion model and its predictions, if its constituents believea safety buffer to protect the public is unnecessary. Yet Region Amost likely would reject the DEGADIS model and its predictions,because it desires a minimum margin of safety of at least 1.5 ata higher confidence level.

Clearly, Region A would benefit by establishing an LNG facilityproperty line at 50 percent LFL in accordance with 49 CFR Part 193rather than at 100 percent LFL. This conclusion is explained furtherin the next example.

5. Applying the new performance metric toNFPA 59A and 49 CFR part 193

As both the Pipeline and Hazardous Materials Safety Adminis-tration and the National Fire Protection Association only permit theconcentration of methane (the primary component of vaporizedLNG) in air to reach 50 percent LFL at an LNG facility property line,the Appendix explains how the aforementioned methodologywould be applied to vapor dispersion exclusion zones in NFPA 59Aand 49 CFR Part 193. When used consistently with large data sets of30 observed distances or more, the inherent margin of safety andits corresponding confidence level may be determined for vali-dating LNG vapor dispersion models and their predictions of vapordispersion exclusion zones.

For demonstrative purposes, individual margins of safety (MsiD)for DEGADIS exclusion zones, using predicted distances at 50 percentLFL (Pi50%LFL) and observed distances at 100 percent LFL (Oi

100%LFL), aredelineated in Table A.2. From this small data set of four ratios (nD),SMs

D (the standard deviation of the DEGADIS data set or sample) and�xD (the sample’smean) are respectively estimated to be0.78 and 2.41.Calculating the ZscoreD for this data set and a MsdesiredD (the desiredminimum margin of safety for a DEGADIS exclusion zone) of 1.5indicates MsdesiredD is �1.17 standard deviations from �xD. Conse-quently, the DEGADIS exclusion zone prediction at 50 percent LFLappears to have aMSWCD of 1.5with an 88 percent confidence, for anobserved test distance corresponding to 100 percent LFL (referenceFig. A.3).

To emergency response officials, this conclusion is significant,because DEGADIS exclusion zone predictions of property linedistances (where vapors must not exceed 50 percent LFL to complywith 49 CFR Part 193) are at least a factor of 1.5 greater than thedistance between an LNG spill and the point where LNG vaporsignite at 100 percent LFL. For the Burros 8 field observation data inTable A.2, this overprediction means the facility’s property linewould be 540 m from the spill, and the distance between a hypo-thetical LNG spill and flammable vapors at 100 percent LFL wouldbe approximately 360m, creating a 180m safety buffer (or inherentmargin of safety) between the public and flammable LNG vapors.Furthermore, emergency response personnel would be 88 percentconfident this safety buffer would not be less than 180 m, thusunderscoring the benefit of a 50 percent LFL limit at an LNG facilityproperty line.

6. Error analyses and their importance

At best, the above examples demonstrate the value of thisstatistical methodology and its new performance metric fordetermining the minimum margins of safety and their

corresponding confidence levels. The data sets presented here areobviously small and permit error in these calculations. Larger datasets of 30 Msi or more will provide statistical analyses with higherconfidence levels and lower error. This explains why an erroranalysis is an important element of the performance metric,MSWC.

As described in Fig. A.4 and the Appendix, the maximum error(E) of the average margin of safety (x) does accurately characterizehow the minimum and maximum values of the dataset’s mean, m,for a moderate sample size influences the accuracy of confidencelevel estimates (reference Eq. (11)). This error in the MSWC’sconfidence level should then be compared against the standarderror for SMs (reference Eqs. (8e10)) to establish the best estimatesof confidence for an LNG model’s MSWC. When these analyses arecompleted, one should be reasonably confident that a model and itsexclusion zone predictions are either acceptable or suspect.

7. Conclusions

Relevant performance metrics and their correct application areessential elements in evaluating LNG vapor dispersion models. Onlythe best tools should be employed for this work. A brief synopsis ofthese tools is presented here to facilitate additional study in thisfield.

� The process for creating correlations of vapor dispersion modelpredictions and observed field trial results should reconcile theuncertainties associated with test measurements, source termmodels, and computational techniques to ensure statisticallyvalid results.

� “Natural and stochastic variability in the atmospheric diffusionphenomena”, as described by Hanna et al. (1993) inevitablycause dispersion model predictions to vary by a factor of two.

� Screening vapor dispersion model predictions and observedmeasurements reveals whether they agree by a factor of two,five, or ten and guides researchers in their model validation, asproposed by Chang and Hanna (2004).

� Graphs of geometric variance and geometric mean bias readilycompare the performance of vapor dispersion models.

� Calculating the percentage of gas concentration predictionsbetween 50 and 200 percent of observed results (also known asfractional results by Carissimo et al., (2001) is a useful tech-nique for comparing vapor dispersion models and theirapplication.

� Due to the work of Coldrick, Lea and Ivings (2009), larger LNGdata sets are now available and allow researchers, regulators,and LNG operators to utilize different statistical techniques andnew performance metrics, like theMSWC, in their validation ofLNG vapor, dispersion models.

Collectively, these tools provide regulators, LNG operators, andnew entrants to this industry with a balanced approach to vettingand selecting appropriate vapor dispersion models to protect theworkplace, public, and environment from the hazards of flammablegases.

Acknowledgements

This paper represents the author’s views exclusively and doesnot reflect those of the Pipeline and Hazardous Materials SafetyAdministration.

The author wishes to express his sincere gratitude to those indi-viduals and organizations that assisted in the preparation of thispaper. Allen Mayberry and Sherri Pappas of the Pipeline andHazardous Materials Safety Administration (PHMSA) approved andsupported this work. Anay Luketa of Sandia Laboratories provided

Table A.1Msi for distances at 50 & 100 percent LFL.

LNG test &atmosphericstability

Flammabilitylimit (LFL)

Observeddistance(m)

Predicteddistance(m)

Msi (Pred/Obsratio)

Burro 8 e E 50% 700 550 0.786

F.A. Licari / Journal of Loss Prevention in the Process Industries 23 (2010) 745e752 749

technical guidance, and John Jacobi of PHMSA offered editorialassistance. Anthony Breen and Lori Hutwagner of PHMSA assistedwith illustrations. The author also wishes to recognize and thank themany researchers who have contributed to the evaluation of vapordispersion models and the development of suitable performancemetrics for this purpose.

Burro 9 e C 50% 480 700 1.458Maplin 29 e D 50% 280 300 1.071Maplin 39 e D 50% 230 400 1.739Burro 8 e E 100% 360* 360* 1.000*Burro 9 e C 100% 240* 450* 1.875*Maplin 29 e D 100% 150* 180* 1.200*Maplin 39 e D 100% 125* 220* 1.760*

*Data extrapolated from Figs. 3 through 6 (Havens, 1992).

Appendix. New methodology & performance metric toevaluate LNG vapor dispersion models

Statistical basis for determining a margin of safety with confidence

The proper validation of a vapor dispersion model requiresa large data set of both observed test measurements and predictedvalues to obtain statistically valid conclusions with a high confi-dence. When the Health and Safety Laboratory compiled anextensive database of liquefied natural gas (LNG) test informationthis year, focused data sets with well documented test conditionsbecame available to validate LNG vapor dispersionmodels (Coldricket al., 2009). With the advent of these larger, well documented datasets, researchers may now apply a slightly different statisticalmethodology and a new performance metric to confirm how welltheoretical predictions of LNG vapor exclusion zones correspond toactual test measurements. Such analyses would allow LNG opera-tors, government regulators, and the public to understand theadditional confidence level and distance that exists when anexclusion zone ends at 50 percent of the vapor’s lower flammabilitylimit (LFL) rather than 100 percent LFL.

The proposed methodology would provide stakeholders witha performance metric that directly links a vapor dispersion model’sinherent margin of safety with a defined confidence level. Witha validation data set of observed distances (Oi

d) and predicteddistances (Pid), individual margins of safety (Msi) for each data paircould then be calculated for LNG tests with unobstructed testconditions, using the following ratio.

Msi ¼PdiOdi

(1)

These margins of safety, when calculated for all data pairs (n) ina validation data sample, should be similar to the normal distri-bution in Fig. A.1 and should be plotted in a histogram to confirmthis relationship (refer to Fig. A.5). For a modest sample size ofperhaps 30 data pairs, the data set could be described by a standarddeviation (SMs) and average (x) by Eq. (2) (Freund, 1981, chaps.7e9).

Fig. A.1. Msdesired determines Zscore and confidence level.

SMs ¼

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiXni¼1

ðMsi � xÞ2

n� 1

vuuuut(2)

Assuming this data set is like a normal distribution, then one maytransform this data set distribution into a standard normal distri-bution by the formula:

Zscore ¼ Msdesired � xSMs

(3)

where Msdesired is the desired minimum margin of safety for thedata set of predicted exclusion zone distances.

From the Zscore, one concludes that Msdesired is Zscore standarddeviations from the sample’s mean, x, as depicted in Fig. A.1. Byestimating the shaded area beneath the standard normal distri-bution curve, the confidence level associated with Msdesired may bedetermined for all Msi data pairs. This desired minimum margin ofsafety with a corresponding confidence level could then be deemeda minimum margin of safety with confidence (MSWC) for an LNGvapor dispersion model.

Example MSWC calculations for a vapor dispersionmodel and distance predictions

One example of a MSWC using DEGADIS distance predictionsmay be calculated from a data set which Havens developed in 1992(Havens, 1992). Using Table 10 and Figs. 3 through 6, eight datapairs (n) may be extrapolated from his study to calculate a MSWC.Although this data set, depicted in Table A.1, is too small fora conclusive validation of DEGADIS, its data do provide a simpleexample of this new methodology and its corresponding perfor-mance metric, MSWC.

From data pairs of DEGADIS distance predictions and observeddistances, Msi are calculated, and SMs and x are estimated to be 0.40and 1.36 respectively.

For demonstrative purposes, the MSWC of a DEGADIS distanceprediction shall be calculated for two fictitious geographic areas,Region A and Region R. Each region has different perceptions ofsafety and risk, so theMsdesiredA for Region A is assumed to be 1.5, andMsdesiredR for Region R is 1.0. Z scores are calculated and indicate thatMsdesiredA is 0.34 standard deviations from x, and MsdesiredR is �0.89standard deviations away. As the shaded area in Fig. A.2 illustrates,there is a 37 percent confidence level that the minimum margin ofsafety of a DEGADIS distance prediction for Region Awill exceed 1.5times a corresponding test observation. In simpler terms, theDEGADIS distance prediction for Region A has a MSWCA of 1.5 witha 37 percent confidence level, while Region R’sMSWCR is 1.0 with an81 percent confidence level. Different perceptions of safety and riskfor each region directly influence its MSWC.

Fig. A.2. MSWC for Regions A & R. Fig. A.3. MsdesiredD and xD determine MSWCD.

F.A. Licari / Journal of Loss Prevention in the Process Industries 23 (2010) 745e752750

MSWC estimates for a DEGADIS exclusion zone

Because many societies are uncomfortable with the abovemargins of safety or corresponding confidence levels, regulatorsand researchers have recommended that confidence levels beincreased. Since DEGADIS was adopted within 49 CFR Part 193,additional confidence was achieved by requiring DEGADIS exclu-sion zones to extend beyond 100 percent LFL within an LNG facilityto 50 percent LFL at the facility’s property line. Using the abovemethodology, the inherent margin of safety and confidence level ofthis decision may now be estimated and explained.

The data set which Havens developed in Table 10 for LNG testswith unobstructed test conditions is again referenced to calculateindividual margins of safety (MsiD) for DEGADIS exclusion zones,using observed distances at 100 percent LFL (Oi

100%LFL), predicteddistances at 50 percent LFL (Pi50%LFL) in Table A.2, and Eq. (4).

MsDi ¼ P50%LFLi

O100%LFLi

(4)

SMsD ¼

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiXni¼1

�MsDi � �xD

�2

nD � 1

vuuuut(5)

From this small data set of four ratios (nD) and Eq. (5), SMsD and

xD are respectively estimated to be 0.78 and 2.41 for demon-strative purposes. Calculating the ZscoreD for this data set and anMsdesiredD of 1.5 indicates MsdesiredD is �1.17 standard deviationsfrom xD. Consequently, the DEGADIS exclusion zone prediction at50 percent LFL appears to have a MSWCD of 1.5 with an 88percent confidence level (reference shaded area in Fig. A.3), foran observed test distance corresponding to 100 percent LFL.Although this is only an estimate, the confidence level of MSWCD

is significantly better than MSWCA (reference page 5) whenMsdesiredD and MsdesiredA are identical, because DEGADIS margins ofsafety (MsiD) use observed distances at 100 percent LFL and

Table A.2MsiD for a DEGADIS exclusion zone at 50 percent LFL.

LNG tests Atmosphericstability

100% LFLobserveddistance (m)*

50% LFLpredicteddistance (m)

MsiD

(Pred/Obs ratio)

Burro 8 E 360 550 1.528Burro 9 C 240 700 2.917Maplin 29 D 150 300 2.000Maplin 39 D 125 400 3.200

*Data extrapolated from Figs. 3 through 6 (Havens, 1992).

predicted distances at 50 percent LFL, plus their average, xD, issystematically larger than x in Fig. A.2.

Size of data set and sensitivity to error

At best, the above examples demonstrate a useful methodologyfor determining the minimum margins of safety and their corre-sponding confidence levels for validating a vapor dispersion model.The data sets presented here are obviously small and permitundesirable error in these calculations. Larger data sets of 30observed distances or more will provide statistical analyses withhigher confidence levels and lower error. This explainswhy an erroranalysis is an important element of the performance metric,MSWC.

To estimate the error associated with the MSWC of a modestvalidation data set (perhaps 30 to 40 data pairs), the t distribution isused to obtain the maximum error (E) of the average margin ofsafety (x) for this sample, using Eq. (6).

E ¼ ta=2

�SMsffiffiffin

p�

(6)

where ta/2 is the t score for a desired confidence level (presumably90 percent), and SMs and n are respectively the standard deviationof all Msi values in the sample and the number of all data ratios.From E, one concludes that the sample may have a true mean (m)that is between xþ E ¼ mmax and x� E ¼ mmin.

By examining the normal distribution curve for the validationsample, the area under the curve between mmax and mmin can thenbe determined in Fig. A.4 and compared against Msdesired, thedesired minimum margin of safety for the data set of predictedexclusion zone distances. The error in the confidence level forMSWC that corresponds to E may then be calculated by Eq. (7).

Fig. A.4. E determines mmin, mmax, and MSWC confidence level.

Table A.3Excerpt of Havens (1992) gas concentration ratios.

LNG tests Atmosphericstability

MsiG(Pred/Obs ratio)

Burro 8 E 0.716E 1.462E 0.798E 0.683

Burro 9 C 1.885C 1.629C 1.669

Maplin 29 D 0.775D 0.803D 1.137D 0.972D 1.231D 1.424D 1.204

Maplin 39 D 0.541D 1.147D 1.139D 2.111D 1.554D 1.672D 2.319

F.A. Licari / Journal of Loss Prevention in the Process Industries 23 (2010) 745e752 751

Zscore minor Zscore max ¼ Msdesired � ðx� EÞSMs

¼ Zscore � ESMs

(7)

To complete this evaluation of error and its impact on MSWC, thestandard error of SMs must also be considered and estimated.Assuming once again that a data set may be limited to between 30and 40 observed distances (a large sample), the maximum andminimum values of the large sample, standard deviation (sMs)should be calculated using Eq. (8) and Eq. (9), where Za/2 is 1.645 fora 90 percent confidence level.

smaxMs ¼ SMs

1� Za=2ffiffiffiffiffi2n

p(8)

sminMs ¼ SMs

1þ Za=2ffiffiffiffiffi2n

p(9)

Should the variability of sMs be small for this sample, it may beappropriate to assume that the calculated average, xMs, is in factthe mean, mMs, of the Msi population. For this case, the maximumand minimum confidence levels for MSWC may be determinedfrom Eq. (10).

Zscore� variability in sMs¼ Msdesired�mMs

sminMs

orMsdesired�mMs

smaxMs

(10)

These confidence levels should be compared against the resultsobtained in Eq. (7) to establish the best estimates of confidence fora model’s MSWC. At the completion of this error analysis, oneshould be reasonably confident that an LNG vapor dispersionmodelis properly validated.

Example calculation of the error in the average marginof safety and its impact on MSWC

Table 10 of Havens, 1992 study provides an excellent data set tocalculate the error, EG, for the average margin of safety, xG, of gasconcentration ratios, MsiG. An excerpt of this data appears inTable A.3 below, and an error analysis of these ratios would explainhow EG affects the minimum margin of safety with confidence(MSWCG) for DEGADIS predictions of gas concentrations.

First, a histogram of the MsiG should be plotted to gage howwellthe data within Table A.3 correspond to a normal distribution.Fig. A.5 illustrates this relationship and confirms the data above 1.0is similar to a normal distribution. This image would be consistent

Fig. A.5. Histogram of MsiG in Table A.3.

with results from a conservative vapor dispersion model wherea plurality of data is near and above 1.0.

From the 21 data ratios (nG) in Table A.3, SMsG and xG are respec-

tively estimated as 0.489 and 1.280. Calculating the ZscoreG for this dataset and a MsdesiredG of 1.0 indicates ZscoreG is�0.573. This value of ZscoreG

means there is a 72 percent confidence level that the minimummargin of safety of a DEGADIS gas concentration prediction willexceed 1.0 times a corresponding test observation. In more conciseterms, the MSWCG is 1.0 with a confidence level of 72 percent.

AsMSWCGwouldbe deemed low, estimating themaximumerror(EG) in the average (xG) of theMsiG data set would be appropriate. Inthis case, EGis �0.184 when ta/2

G is 1.725 for a 90 percent confidencelevel. Consequently, mmin

G and mmaxG would respectively be 1.10 and

1.46 for the data set in Table A.3. The error in the confidence level forMSWCGmay then be calculated by Eq. (11) and is depicted in Fig. A.6.These maximum and minimum values of

ZGscore min or ZGscore max ¼

ZGscore �EG

SMsGz� :573� :184

:489z� :95 or � :20 ð11Þ

Zscore maxG and Zscore min

G would then correspond to confidence levelsof approximately 83 and 58 percent respectively. Therefore, the

Fig. A.6. MSWCG with minimum & maximum confidence levels.

F.A. Licari / Journal of Loss Prevention in the Process Industries 23 (2010) 745e752752

average confidence level for a MSWCG of 1.0 would be 72 percentwith approximate errors of þ11 and �14 percent.6

References

Carissimo, B., Jagger, S. F., Daish, N. C., Halford, A., Selmer-Olsen, S., Riikonen, K.,et al. (2001). The SMEDIS database and validation exercise. International JournalEnvironment and Pollution, 16(1e6), 614e629.

Chang, J. C., & Hanna, S. R. (2004). Air quality model performance evaluation.Meteorology & Atmospheric Physics, 87, 167e196.

Coldrick, S., Lea, C. J., & Ivings, M. J. (2009). Guide to the LNG model validationdatabase. Health and Safety Laboratory and The Fire Protection ResearchInstitute.

Freund, J. E. (1981). Statistics, a first course. Englewood Cliffs, NJ: Prentice-Hall, Inc.,Chapters 7-9.

Hanna, S. R., Chang, J. C., & Strimaitis, D. G. (1993). Hazardous gas model evaluationwith field observations. Atmospheric Environment, 27A(15), 2265e2285.

Havens, J. A. (1992). An evaluation of the DEGADIS dense gas (atmospheric) dispersionmodel with recommendations for a model evaluation protocol. (South Coast AirQualify Management District).

Havens, J. A., & Spicer, T. O. (1985). Development of an atmospheric dispersion modelfor heavier-than-air gas mixtures, Vol. I. (U.S. Coast Guard).

Nomenclature

E: maximum error in the average margin of safety, xEG: the error in xGLFL: lower flammability limitLNG: liquefied natural gasm: meter(s)Msdesired: desired minimum margin of safety

MsdesiredA : desired minimum margin of safety for a DEGADIS distance prediction in

Region AMsdesired

D : desired minimum margin of safety for a DEGADIS exclusion zone

MsdesiredG :desired minimum margin of safety for DEGADIS predictions of gas

concentrationsMsdesired

R : desired minimum margin of safety for a DEGADIS distance prediction inRegion R

Msi: margin of safety for a vapor dispersion model prediction

MsiD: margin of safety for a DEGADIS exclusion zone prediction

MsiG: margin of safety for a DEGADIS prediction of gas concentration

MSWC: minimum margin of safety with confidence

6 The author reminds readers that confidence levels are not symmetrical aboutxG , because areas beneath the normal distribution curve are unequal for themaximum and minimum values of ZscoreG .

MSWCA: minimum margin of safety with confidence for a DEGADIS distancepredictions in Region A

MSWCD: minimum margin of safety with confidence for DEGADIS exclusion zonepredictions

MSWCG: minimum margin of safety with confidence for DEGADIS predictions ofgas concentration

MSWCR: minimum margin of safety with confidence for DEGADIS distancepredictions in Region R

NFPA 59A: 2001 edition of National Fire Protection Association Standard 59An: number of Msi ratios in the data set or samplenD: number of MsiD ratios in the data set or samplenG: number of gas concentration ratios in MsiG data set or sample

Oid: observed distance during a field trial

Oi100%LFL: observed distance at 100 percent LFL

Pid: predicted distance of a vapor dispersion model

Pi50%LFL: predicted distance at 50 percent LFL

SMEDIS: Scientific Model Evaluation of Dense Gas DispersionSMs: standard deviation of the Msi data set or sample

SMsD: standard deviation of the MsiD data set or sample

SMsG: standard deviation of the MsiG data set or sample

ta/2: t score for a desired confidence level to calculate E

ta/2G: t score for a desired confidence level to calculate EG

x: average of the Msi data set or samplexD: average of the MsiD data set or samplexG: average of the MsiG data set or samplexMs: average of the data set or sample that is similar to mMsZscore: standard units above (or below) x on a normal distribution curve

ZscoreD : standard units above (or below) �xD on a normal distribution curve

ZscoreG : standard units above (or below) xG on a normal distribution curve

Zscore � variability in sMs: maximum and minimum Zscore corresponding to sMs, mMs, andMsdesired

Za/2: desired confidence level to determine the large sample, standarddeviation

49 CFR Part 193: Title 49, Code of Federal Regulations, Part 193, Liquefied Natural GasFacilities: Federal Safety Standards

sMs: large sample, standard deviation of the Msi populationm: mean of the populationmmax: maximum value of the population mean corresponding to Emmin: minimum value of the population mean corresponding to E

mmaxG : maximum value of the population mean corresponding to EG

mminG : minimum value of the population mean corresponding to EG

mMs: mean of the Msi population