1 of 36 the epa 7-step dqo process step 6 - specify error tolerances (60 minutes) (15 minute morning...

1 of 36

The EPA 7-Step DQO Process

Step 6 - Specify Error Tolerances

(60 minutes)(15 minute Morning Break)

Presenter: Sebastian Tindall

DQO Training CourseDay 3

Module 17

2 of 36

Terminal Course Objective

To be able to define, for a specific project:1. The variability for each COPC

2. the decision errors,

3. the consequences of the errors,

4. the null hypothesis,

5. the lower bound of the gray region

3 of 36

Step Objective: To specify the decision

makers’ tolerable limits on decision errors, which are used for limiting uncertainty in the data– Since analytical data can only

provide an estimate the true condition of a site, decisions that are based on such data could potentially be in error

Step 6: Specify Error Tolerances

Step 4: Specify Boundaries

Step 2: Identify Decisions

Step 3: Identify Inputs

Step 1: State the Problem

Step 5: Define Decision Rules

Step 6: Specify Error Tolerances

Step 7: Optimize Sample Design

4 of 36

Decision Error

Tolerances

Bounds of the

Gray Region

Assign probability limits on either side of the gray region

Information IN Actions Information OUT

From Previous Step To Next StepDecision

RulesStep 5

Determine the variability of the environmental variables

Step 6- Specify Error Tolerances

Choose the null hypothesis

Identify the decision errors

Specify the boundaries of the gray region

5 of 36

Decision Error Tolerances The goal of the planning team is to develop

a data collection design that reduces the chance of making a decision error to a tolerable level

Step 6 provides a mechanism for allowing the decision maker to define tolerable limits on the probability of making a decision error

6 of 36

Two Reasons Why Decision Makers Make Decision Errors

Sampling error occurs because the sampling design is unable to capture and control the complete extent of heterogeneity that exists in the true state of the environment

Measurement error occurs because analytical methods and instruments are not absolutely perfect

7 of 36

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5





In order to calculate the number of samples needed

(in DQO Step 7), an estimate of the population standard

deviation is needed for each environmental variable.• Compile a list of the “driver” COPCs• Use existing data (must pass Step 3 data assessments)• Establish the range based on historical information

– Existing data– Process knowledge– Professional judgment

• Estimate of the population standard deviation– Reference source – Method of calculating


8 of 36

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5





In order to calculate the number of samples needed

(in DQO Step 7), an estimate of the population standard

deviation is needed for each environmental variable.• Compile a list of the “driver” COPCs• Use existing data (must pass Step 3 data assessments)• Establish the range based on historical information

– Existing data– Process knowledge– Professional judgment

• Estimate of the population standard deviation– Reference source – Method of calculating

Estimate the standard deviation by using the Deming

approach of dividing the range by 2 or 3, depending

on the frequency distribution.


9 of 36

Estimated Standard DeviationsEstimated Standard Deviations for NaI Example

Environmental VariableRange of

EnvironmentalVariable

Estimate of Population Standard DeviationDR#

AttributeUnit of

MeasureLowerLimit

UpperLimit

StandardDeviation

Source How Estimated?

1

True mean ofCs-137estimated by95% UCL ofsample mean

pCi/g 0.031 1.89 0.4052000HPGedata

From 116-H-1 CVP data

1

True mean ofCs-137estimated by95% UCL ofsample mean

cpm(convertedto pCi/g)

-5.17 99.92

7.48or

7.48/2=3.74

1999 to2000 NaIdata

From 116-H-1 CVP data

10 of 36

Estimated Standard Deviations CS

DR #Environmental

Variable

Range ofEnvironmental

Variable

Estimate of PopulationStandard Deviationa

AttributeUnit of

MeasureLowerLimit

UpperLimit

StandardDeviation

SourceHow

Estimated

Lead mg/kg 10 27 6.5

Uranium mg/kg 78 112 10.5

TPH mg/kg 0.03 U 4.0 1.5

1-2b

PCBs(Aroclor

1260)

mg/kg 0.03 U 2.0 0.88

RI/FSData

Computedfrom RI/FS

data

a The choice of an estimate of a standard deviation has a large impact on the number of samples required. Avoid underestimating thestandard deviation. Always be conservative when estimating the standard deviation.

b The two rules are applied to two depths of soil: 0-6” (DR 1) and 6”-10’ (DR 2).U = Sample was undetected, value shown is the analytical detection limit.

11 of 36

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5





Define both types of decision error:Determine which one occurs above and which one occurs below the action level.

Two Types of Decision Error:• Cleaning up a clean site• Walking away from a dirty site


12 of 36

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5





For each Alternative Action:• Create a list of possible decision error(s) that may occur if an action

is incorrectly taken• Discuss the consequences of making each decision error• Rate the severity of the consequences of a decision error

(i.e., low, moderate, severe) at a point:– Far below the action level– Below but near the action level– Above but near the action level– Far above the action level

• Indicate which decision error has the most severe consequencenear the action level


13 of 36

Decision Error ConsequencesCS

DR #

AA #Possible Decision

ErrorConsequences of Decision Error

Severity of Consequences of Decision Error When True Parameter Is …

(Rated as Low/Moderate/Severe)Decision Error That Has More

Severe Consequences

Near the Action Level

Far Below the Action

Level

Below But Near the

Action Level

Above But Near the

Action Level

Far Above the

Action Level

1a1

1b1

1: Conduct remedial action

Remediating an uncontaminated site

Expense and schedule impacts of remediating an uncontaminated site; worker safety

Severe Moderate None None

Not remediating a contaminated site

2: Take no further action

Failing to remediate a contaminated site

Leaving a site in place the poses a threat to human health and safety

None None Moderate Severe

2a2

2b2

1: Conduct remedial action

Remediating an uncontaminated site

Expense and schedule impacts of remediating an uncontaminated site; worker safety

Severe Moderate None None

Not remediating a contaminated site

2: Take no further action

Failing to remediate a contaminated site

Leaving a site in place the poses a threat to human health and safety

None None Moderate Severe

1 Applies to the depth of 0-6”.2 Applies to the depth of 6”-10’.

14 of 36

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5




Specify the boundaries of the gray regionProvide rationale for rating the severity of consequences as low or severe


15 of 36

Rationale for Error Consequence Ratings

CS

Table 3b. Justification for Rating Severity of Consequences as “Severe”

DS/DR

AA Justification

#1a,#2b

#1 The decision error for remediating an uncontaminated site where the concentration is farbelow the action limit is rated as “severe.” Because of the greater cost is incurred if thisincorrect decision is made, thus the severity increases.

#1a,#2b

#2 The decision errors for not remediating both shallow and deep soils when theconcentrations are far above the action level are also rated as “severe.” In these cases, soilwith concentrations that may pose serious health risks would remain at the site.

a Applies to decision statements and rules related to depth 0 – 6”.b Applies to decision statements and rules related to depth 6” – 10’.

16 of 36

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5




Specify the boundaries of the gray regionDefine the null hypothesis (baseline condition) and the alternative hypothesis:The decision error that has the most adverse potential consequences should bedefined as the null hypothesis.

The null hypothesis should state the OPPOSITE of what the project hopes to demonstrate.

• Site is assumed to be contaminated until shown to be clean• Site is assumed to be clean until shown to be contaminated


17 of 36

Null Hypothesis

Contaminated:HO : > Action Level

Uncontaminated:HA : < Action Level

CS

DR Null Hypothesis Statement (examples)IndicateSelection

The site is contaminated. #1-2 a

The site is uncontaminated.a Applies to both 0-6” depth and 6”-10’ depth

18 of 36

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5





The gray region is a range of possible parametervalues within which the consequences of a decision errorare relatively minor.


19 of 36

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5





The gray region is bounded on one side by the action level, and on the other side by the parametervalue where the consequences of decision error beginsto be significant. This point is labeled LBGR, whichstands for lower bound of the gray region.


20 of 36

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5

Determine the possible range of the parameter of interest

Choose the null hypothesis.

Identify the decision errors.





It is necessary to specify the gray region because variability in the population and unavoidable imprecision in the measurement system combine to produce variability in the data such that a decision maybe “too close to call” when the true parameter value is very near the action level.


21 of 36

Width of the Gray Region () :UBGR - LBGR or AL - LBGR

= Analytical + Sampling Error– Estimated based on past data and general knowledge

= 1/2 of the AL – For each COPC, calculate and set LBGR

= 10 - 30% of the AL

– For each COPC, calculate and set LBGR

= PDF method– Use PDF for worst COPC to set LBGR

22 of 36

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5





Present the rationale of how the LBGR was calculated or determined.


23 of 36

Lower Bound of the Gray Region

Because the null hypothesis is that the site is contaminated, the upper bound of the gray region is set equal to the action level

The LBGR should be set at a value where the consequences of the decision error begin to be significant

24 of 36

How to Set the LBGR

LBGR set by the Analytical + Sampling Error LBGR set to 1/2 Action Level LBGR set to ~ 50 to 90% of AL (Decision-

Maker “whim”) LBGR set by the Probability Distribution Function

(PDF) method

UBGR - GR = LBGR

AL - = LBGR

25 of 36

The LBGR is often based on unavoidable variability in the concentration data– The GR may be estimated based on the precision

that the analytical methods allow plus an estimate as to the sampling variance

– LBGR = AL – GR (Analytical + Sampling Error) 100 ppm – (10 ppm + 31 ppm) = 59 ppm

MARSSIM suggests the LBGR be set as:– LBGR = AL – GR (1/2 AL)

100 ppm – 50 ppm = 50 ppm

How to Set the LBGR (cont.)

26 of 36

The LBGR is often set at some other value– This is based on the decision makers’ choice and

is not scientifically based– LBGR = AL – GR (20% of AL);

100 ppm – 20 ppm = 80 ppm


27 of 36

Use the Frequency Distribution method– The LBGR may be estimated based the Probability

Distribution Function (PDF)– Place the Action Level on the mean of the PDF– Ask: “Does a substantial amount of contaminant

concentration values exceed the Action Level?”– If yes, begin moving the PDF backwards along the

x-axis towards zero concentration– Pause and ask again– When the answer is no, you have set the LBGR

(e.g., where the mean of the PDF lies on the x-axis is now the LBGR)

• Use probability theory to show/prove this


28 of 36

Methods for Evaluating the Attainment of Cleanup Standards - Volume 1: Soils and Solid Media

EPA, February 1989

PB89-234959

How to set the LBGR

2

121

211

2

5.0

Z

AL

ZZsn

1 is a hypothetical “true mean concentration where the site should be declared clean with a high probability”. (1 = LBGR)

29 of 36

Show Probability Density Function Distribution

Demonstration

30 of 36

Setting the GR for LeadCS

DR Basis of GR Action Levela Gray Regiona LBGRa

1,2 Analytical andSampling Error

250 6.5 243.5b

1,2 ½ of the ActionLevel

250 125 125

1,2 Regulator Input(20% of the AL)

250 50 200

1,2 Setting theLBGR Based onthe PDF Method

250 TBD: FAM/DWP TBD:FAM/DWP

a. All units are mg/Kgb. The observed standard deviation of the past data was used to estimate the analytical + sampling error. The

duplicate results were included. The choice of an estimate of a standard deviation has a large impact on thenumber of samples required. Avoid artificially low estimates of the standard deviation. Always beconservative when estimating the standard deviation.

FAM = Field Analytical MeasurementsDWP = Dynamic Work Plan

31 of 36

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5





Assign probability values that reflect the decision maker’s tolerable limits for making an incorrect decision.

• At the action level (Alpha error)• At the other bound of the gray region (Beta error)• At a point far below the action level• At a point far above the action level


32 of 36

Site is dirtySite is clean

100

True State of Site

Alternative Action

Walk away from site Clean up site

75

Probability of deciding

that the site is dirty

0.0

0.5

1.0

Lower Bound of Gray Region

Four Decision-MakerError Tolerance

Locations

True mean COPC Concentration

Action Level

Null Hypothesis:

Site is dirty.

33 of 36

Step 6 Summary

Determine the variability for each COPC Define the two types of error

– Incorrectly walking away from a dirty site, or– Incorrectly cleaning a clean site

Evaluate severity of the incorrect decisions both below, above, and near the action level

Select the null hypothesis

34 of 36

Step 6 Summary

Establish a LBGR based on one of the four methods shown previously

Provide the basis for selecting the LBGR Remember the closer the LBGR is to the

action level, the more samples are needed Assign probability limits on either side of the

gray region – Specify the error rates (Alpha and Beta) decision

makers are willing to accept and provide rational for the rates

35 of 36

Decision Error

Tolerances

Bounds of the

Gray Region




RulesStep 5






36 of 36

End of Module 17

Thank you

Questions?

We will now take a 15 minute break.

Please be back in 15 minutes.

1 of 36 the epa 7-step dqo process step 6 - specify error tolerances (60 minutes) (15 minute morning...

Documents