copyright © 2004 david m. hassenzahl understanding uncertainty definitions and tools for risk...

Copyright © 2004 David M. Hassenzahl

Understanding Uncertainty

Definitions and Tools for Risk Analysts


Goals for this Lecture

• Understand sources and types of uncertainty

• Evaluate quantitative and qualitative representations of uncertainty– Distributional forms

• Explore some typologies• Consider how people interpret

uncertainty


Overview

• Uncertainty is unavoidable• Uncertainty is information!

– Where to do further research– Worst case scenarios

• Uncertainty is chronically understated– Generally satisfied with statistical analysis– Henrion and Fischhoff (1986), estimates of speed

of light (see Kammen and Hassenzahl 1999 page 125)

Copyright © 2004 David M. HassenzahlFrom Henrion and Fischhoff

(1986)

Expected valuewith standard error

Mea

sure

d sp

eed

of li

ght (

km/se

c)

Year of experiment

1870

299600

299650

299700

299750

299800

299850

299900

299950

300000

1880 1890 1900

Year of experiment

299750

299760

299770

299780

299790

299800

299810

299820

299830

1900 196019501940193019201910

1984value

1970

Recommended valuewith reporteduncertainty


Some thoughts

• “Research efforts in risk analysis should be viewed as tools for understanding uncertainties, not necessarily for reducing them.” (Finkel, 1990)

• “Probability does not exist” (Morgan and Henrion, 1990)– Either you die from cause X or you don’t


Uncertainty Analysis Should Be Thorough

• It should include quantitative measures of uncertainty

• It should include qualitative discussion of uncertainty

• Doing this well takes time and effort (and is not always rewarded…)


Quantitative Measures

• Display the data set

• Use descriptive statistics– Moments, ranges, etc

• Apply sensitivity analysis

• Eschew spurious precision!


Use of Point Estimates?

• “introducing confidence intervals…to deal with uncertainty may be pointless…unless the policy analyst eliminates the point estimate itself.”

• “A specific number has a vividness and simplicity which makes it an inevitable focus of policy debate.”

• Camerer and Kunreuther, 1989


Point Estimate or not?

-100%

0

1

Option

Changein risk

+100%

increase

reduction

2 3 4 5donothing

Figure 10-6 from SWRI page 258


Qualitative discussion

• Discuss known sources of error

• Consider plausible sources of error

• Evaluate the importance of uncertainty


Challenges to ignoring uncertainty

• Regarding a report on ozone depletion “…no attempt was made to estimate the systematic errors in evaluating rates or omission of chemical processes. Without such estimates, decision makers are free to make their own judgments ranging from uncritical acceptance of the current models to complete skepticism as to their having any likelihood of being correct.” Morgan and Henrion 1990.


Historical Mistakes

• Rasmussen Report (WASH 1400) on reactor safety– Significant portions retracted by US government– Still referenced

• Inhaber report on nuclear “safer” than other energy technologies– Major source (Holdren) responded “not so”

• Chauncy Starr– Voluntary/involuntary– Risk as f(benefit)– True…but not precisely known

• Tengs et al?


Briggs and Sculpher (1995)

• Cost-effectiveness analyses from medical literature– Public Health origins of CEA

• Incomplete or inadequate attention to uncertainty analysis in 86%


Starr (1969) interpretation

-11

-4

-5

-6

-7

-8

-9

-10

-3

100 500020001000500200 10000

Average annual benefit/person involved [dollars]

Pf [F

atalit

ies/p

erso

n hr

. exp

osur

e]

R~B 3

R~B 3

Voluntary

Involuntary

Hunting, skiing,smoking

Railroads

Naturaldisasters

Commercialaviation Motor

vehicles

General aviation

Average P fdue to disease

Electricpower

R = RiskB = Benefit

1010

1010

1010

1010

1010

1010

1010

1010

1010

1010


Otway and Cohen (1975) Interpretation

R~B 1.8

R~B 6.3

Voluntary

Involuntary

Hunting, skiing,smoking

Railroads

Naturaldisasters

Commercialaviation Motor

vehicles

General aviation

Average P f dueto disease

-11

-4

-5

-6

-7

-8

-9

-10

-3

100 500020001000500200 10000

Average annual benefit/person involved [dollars]

Pf [F

atalit

ies/p

erso

n hr

. exp

osur

e]

Electricpower

R = RiskB = Benefit

1010

1010

1010

1010

1010

1010

1010

1010

1010


Example: Amitraz on Pear Orchards

• EPA decided to ban Amitraz for use on pear orchards (US EPA 1979)

• Point estimate generated for Cost effectiveness– $2.6 million per life-year saved (Tengs et al

1995)


Expected Value of Ban

• Does Amitraz control pests?– If not, ban has no economic implications– cpyls ≤ $0

• Is Amitraz a carcinogen?– If not, ban has major economic implications– cpyls $ ∞

• E(cpyls) ≠ $2.6 million• E(cpyls) = uniform($0, $∞)• Hassenzahl (2004)


Estimators

• Because we can’t always get the data we want, we need to estimate data

• We can use– Frequencies– Distributions– Curve-fitting


Frequency example: 500 people

• 495 have 10 toes

• 2 have 12 toes

• 1 has 9 toes

• 1 has 5 toes

• 1 has 0 toes

• In next 1000: how many will have 6 toes?


Distributions as estimators

• We often use DISTRIBUTIONAL FORMS to approximate data sets

• We then estimate missing or future values using the distribution– Extrapolate beyond data set– Interpolate within data set


Example: 500 people

• feet number• 0 0• 0.5 0• 1 0• 1.5 0• 2 0• 2.5 0• 3 1• 3.5 1

• feet number• 4 12• 4.5 48• 5 145• 5.5 203• 6 78• 6.5 10• 7 2• 7.5 0


Triangular? Normal?

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0 1 2 3 4 5 6 7 8 9

Height (feet)

Fre

qu

ency


Normal: Characteristics: Mean = 5.6 feet, Standard Deviation = 1.3 feet

Frequency Chart

.000

.005

.011

.016

.022

0

54.5

109

163.5

218

2.21 3.89 5.58 7.27 8.96

10,000 Trials 9,921 Displayed

Forecast: Normal


Triangular? Normal?

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0 1 2 3 4 5 6 7 8 9

Height (feet)

Fre

qu

ency


A Good Estimator Is

• Consistent

• Unbiased

• Efficient

• Sufficient

• Robust

• Practical


Depicting Uncertainty: Distributions

• No consensus on how to depict ignorance!– More later under Monte Carlo Analysis

• Key forms– Uniform / rectangular– Triangular– Normal, lognormal– Many others!


Decision

• When should we hold the Founder’s Day parade?

• Assume we want to avoid tornadoes


Uniform

0.001

0.002

Prob

abili

ty o

f tor

nado

Jan Feb Mar Apr May Jun Jul Aug Sep Nov Dec

Month

Oct

0.003

From SWRI page 106. Figure 3.5 Uniform distribution for daily p(tornado) as described in problem 3-7a.

Kammen and Hassenzahl 1999


Rectangular

From SWRI page 106. Figure 3-6 Rectangular distribution for daily p(tornado) as described in problem 3-7a.

Prob

abili

ty o

f tor

nado

Jan Feb Mar Apr May Jun Jul Aug Sep Nov Dec

Month

Oct

0.004

0.002

0.006

0.008

0.010

0.012



Triangular

From SWRI page 107. Figure 3-7 Triangular distribution for daily p(tornado) as described in problem 3-7a.

Prob

abili

ty o

f tor

nado

Jun Jul Aug Sep

MonthOct

0

0.005

0.010

0.015

0.025

0.020



Policy Decision

• Radon is found in homes across the country

• We might worry at 100 pCi/liter• We can’t measure all the homes in the

country, but we have a decent sized sample

• It matters how we model that distribution


Normal Distribution

10 7432 5 6

2

10

8

6

4

0

14

12

8

Num

ber o

f hom

es

Radon level (pCi/liter)



Lognormal

0

5

10

15

20

Hous

es (%

)

222 Rn (pCi/liter)

>8

0 42 6 8

Nero et al 1987


Exponential

0.0001

0.001

0.01

0.1

1

10

100

1 10 100 1000

USAless6

pwr(1.75) %

pwr(1.25) %

LnNrm3.5%

Rn Concentration

% o

f hou

ses w

ith co

ncen

tratio

n >

x

Goble and Socolow 1990


Fitting distributions to data sets

• Compare data to predictions– Least squares– Maximum Likelihood

• We will explore these in context– Binomial for animal toxicology data


Breaking Down Uncertainty

• Useful typologies for thinking about uncertainty

• Can’t always reduce uncertainty

• Typologies have– Internal overlaps– Missing pieces (no perfect typology)


Four typologies

• Finkel– Parameter Uncertainty– Model Uncertainty– Decision-rule Uncertainty

• Smithson– non-quantifiable/holistic

aspects – uncertainty as one

component of ignorance.”

• Boholm– Situates Uncertainty

as the non-calculable part of risk

– appropriate coping strategies:

• faith• precaution and• avoidance


Typology (after Morgan and Henrion 1990)

1. Random error and statistical variation

2. Systematic error and subjective judgment

3. Linguistic imprecision

4. Variability

5. Randomness and unpredictability

6. Expert Uncertainty

7. Approximation

8. Model uncertainty

Normative Uncertainty


Random error/statistical variation

• We have a well defined set of tools

• These can be misleading!– Often the ONLY thing that is done– Often done…and ignored

• Z-scores, Chi-squared, p-values

• Meaning of 95% confidence interval?


Random Error: “Clusters”


Systematic error and subjective judgment

• Example: speed of light, or energy predictions

• Chronically understated

• Useful approach: bounding


Systematic Error: Predicted Year 2000 Energy Use

1972 19821980197819761974

Year of Publication

0

125

100

50

25

150

75

200

175

Qua

drill

ion

Btu

Per Y

ear

Goldemberg et al 1987


Linguistic imprecision

• Inconsistencies in language and usage can lead to problems

• Is “beyond a reasonable doubt” 95%? What does that mean?

• “Rain is likely”…are you from Las Vegas or Bangladesh?

• “A few thousand deaths”


Variability

• Also called “dispersion”

• Get the right population!

• Describing variability can be a challenge

• Monte Carlo analysis is a useful tool


Variability

• Height of individuals– Deterministic

element– Random element

• Susceptibility to disease– Predisposition

• Known• Unknown

– Life-history and habit

• Theoretical models• Empirical data• Who are we worried

about?– “Average” person– Most susceptible

subset


Randomness and unpredictability

• Inherent randomness is irreducible!

• Practical limitations and chaos


Expert uncertainty

• Multiple interpretations of a single data set

• Norms of analysis

• Motivational bias (decision stakes, reputation)


Expert Uncertainty

• Economists’ Conception– Limited resources– Resource substitution– Adaptability

• Ecologists’ Conception– Stable systems– Long term impact of disruption


Climate Change: Economists “versus” Ecologists

Individual respondents' answers

Loss

of g

ross

wor

ld p

rodu

ct

14 17 3 16 1 2 9 1812156114 510137 8-5

0

10

5

25

20

15

90th percentile

10th percentile

50th percentile

Nordhaus 1994


EMF’s and Expertise

• Biomechanists / physicists– “Impossibility” theorems

• Epidemiologists– Correlation and proposed causation

• Toxicologists– Extrapolation


Approximation

• Never have complete data

• There is a tradeoff between efficient computation and resolution or precision– Sensitivity analysis

• Significant figures are important


Model uncertainty

• Getting the right model– Does the model explain the data?– Is the model consistent with theory?

• Getting the model right– Is the model properly stated?– Is the math done correctly

• Other uncertainties (drawn from the typology above)


Model Uncertainty

0.0001

0.001

0.01

0.1

1

10

100

1 10 100 1000

USAless6

pwr(1.75) %

pwr(1.25) %

LnNrm3.5%

Rn Concentration

% o

f hou

ses w

ith co

ncen

tratio

n >

x

Goble and Socolow 1990


Normative Uncertainty

• Often not asked: what is important to us?

• Arguments about technical information mask the true issues

• Leads to vitriol and claim of “ignorance” and “antiscientific attitudes”


Normative UC and YMP

• Gore, Bush: “let the science decide”

• Secretary Abraham “technically suitable site”

• LV residents “technically unsuitable site”

• Have we defined “suitability?”


Interpreting Uncertainty

• Very limited information on how people interpret uncertainty

• Possible links– Uncertainty and credibility– Uncertainty and trust


References

• Briggs, A. and M. Sculpher (1995) Sensitivity analysis in economic evaluation: a review of published studies. Health Economics 4: 355-371.

• Camerer, C. F. and Kunreuther, H. (1989) “Decision Processes for Low Probability Events: Policy Implications,” J. Policy Analysis and Management 8 (4), 565 - 592.

• Goble, R. and Socolow, R. (1990), “High Radon Houses: Implications for Epidemiology and Risk Assessment,” Cented Research Report No. 5 (Clark University, Worcester, MA).


References

• Goldemberg, J,. Johansson, T., Reddy, A and Williams, R (1987). Energy for a Sustainable World. Washington DC: World Resources Institute.

• Hassenzahl, D. M. (2004). “The effect of uncertainty on ‘risk rationalizing’ decisions.” Journal of Risk Research.

• Henrion, Max and Fischhoff, Baruch (1986) “Assessing uncertainty in physical constants”, American J. of Physics,54, 791 - 798.


References

• Nero, A. V., Schwehr, M. B., Nazaroff, W.W. and Revzan, K.L. (1986) “Distribution of Airborne Radon-222 Concentrations in U.S. Homes,” Science (234) 992 - 997.

• Nordhaus, W. D. (1994) “Expert opinion on climate change”, American Scientist, 82, 45 - 51.

• Otway, H. and J. J. Cohen (1975). Revealed Preferences: Comments on the Starr Benefit-Risk Relationships. Laxenburg, Austria, International Institute for Applied Systems Analysis.


References

• Starr, C. (1969). "Social benefit versus techological risk." Science 165: 1232-1238.

• Tengs, T. O., M. E. Adams, et al. (1995). "Five-Hundred Life-Saving Interventions and Their Cost-Effectiveness." Risk Analysis 15(3): 369 - 390.

• US EPA (1979) Determination pursuant to 40CFR162.11(a)(5) concluding the rebuttable presumption against registration of pesticide products containing amitraz. Federal Register 48: 2678-83.


Additional readings

• Kammen and Hassenzahl, 1999. Should We Risk It? Exploring Environmental, Health and Technology Problem Solving, Princeton University Press, Princeton NJ.

• Finkel, Adam, 1990. Confronting Uncertainty in Risk Management (Resources for the Future, Washington DC)

• Morgan, M. Granger and Max Henrion (1990). Uncertainty Cambridge University Press, NY NY.

copyright © 2004 david m. hassenzahl understanding uncertainty definitions and tools for risk...

Documents

hassenzahl briggs

hassenzahl goals

hassenzahl challenges

uncertainty definitions

unavoidable uncertainty

types of uncertainty

dont slide

hassenzahl point estimate