decision making under deep uncertainty robert lempert director rand pardee center for longer range...

Decision Making Under Deep UncertaintyDecision Making Under Deep Uncertainty

Robert LempertDirector

RAND Pardee Center for Longer Range Global Policy and the Future Human Condition

July 8, 2009

Traditional Planning Methods Can Illuminate Trees Rather Than ForestTraditional Planning Methods Can

Illuminate Trees Rather Than Forest

Traditional analytic methods characterize uncertainties as a prelude to assessing alternative decisions

Predict Act

Climate change confronts decisionmakers with deep uncertainty, where

– They do not know, and/or key parties to the decision do not agree on, the system model, prior probabilities, and/or “cost” function

Decisions can go awry if decisionmakers assume risks are well-characterized when they are not

– Uncertainties are underestimated

– Competing analyses can contribute to gridlock

– Misplaced concreteness can blind decision-makers to surprise

Climate change confronts decisionmakers with deep uncertainty, where

– They do not know, and/or key parties to the decision do not agree on, the system model, prior probabilities, and/or “cost” function

Decisions can go awry if decisionmakers assume risks are well-characterized when they are not

– Uncertainties are underestimated

– Competing analyses can contribute to gridlock

– Misplaced concreteness can blind decision-makers to surprise

Forecasting the Unpredictable Can Contribute to Bad Decisions

Gross national product (trillions of 1958 dollars)

2.22.0

1.81.6

Energy use (1015 Btu per year)

Historical trend

continued1970

1920 19291940

19501960

19731900

20 40 60 80 100 120 140 160

1975 Scenarios• In the early 1970s forecasters made projections of U.S energy use based on a century of data

• In the early 1970s forecasters made projections of U.S energy use based on a century of data

Forecasting the Unpredictable Can Contribute to Bad Decisions

Gross national product (trillions of 1958 dollars)

2.22.0

1.81.6

Energy use (1015 Btu per year)

Historical trend

continued1970

1920 19291940

19501960

19731900

20 40 60 80 100 120 140 160

2000 Actual

19801977

1975 Scenarios2000 Actual

19801977

… they all were wrong

OutlineOutline

• Robust Decision Making (RDM)

• Climate vulnerability and response option analysis for Inland Empire Utilities Agency (IEUA)

• Observations

New Technology Allows Computer to Serve As “Prosthesis for the Imagination”

• Robust Decision Making (RDM) is a quantitative decision analytic approach that

– Characterizes uncertainty with multiple, rather than single, views of the future

– Evaluates alternative decision options with a robustness, rather than optimality, criterion

– Iteratively identifies vulnerabilities of plans and evaluates potential responses

• Robust Decision Making (RDM) is a quantitative decision analytic approach that

– Characterizes uncertainty with multiple, rather than single, views of the future

– Evaluates alternative decision options with a robustness, rather than optimality, criterion

– Iteratively identifies vulnerabilities of plans and evaluates potential responses

Candidate strategy

Identify vulnerabilities

Assess alternatives for ameliorating vulnerabilities

• RDM combines key advantages of scenario planning and quantitative decision analysis in ways that

– Decision makers find credible

– Contribute usefully to contentious debates

• RDM combines key advantages of scenario planning and quantitative decision analysis in ways that

– Decision makers find credible

– Contribute usefully to contentious debates

RDM Has Effectively Addressed Many Types of Decisions Under Deep Uncertainty

Energy, Environment, and Climate Change

• Long-Range Natural Resource Management

• Renewable portfolios standards

• Center on climate change decision making

National Security

• Terrorism Insurance

• Force procurement and deployment

• Pre-conflict shaping strategies

Commercial-Sector Applications

• Electric utilities’ strategies under deregulation

• Product and technology planning in the auto industry

Compare Alternative Approaches to Managing Catastrophic Event with Unknown ProbabilityCompare Alternative Approaches to Managing Catastrophic Event with Unknown Probability

• Consider town on shore of pristine lake

– Lake can switch abruptly to undesirable and potentially irreversible eutrophic state at unknown pollution concentration

• Citizens must decide how much pollution to emit

– Gain small utility from emitting pollution to lake and lose significant utility if lake goes eutrophic

– Deeply uncertain about location of concentration threshold

• Alternative decision approaches include:

– Optimum expected utility

– Precautionary principle

– Robust decision making

• Consider town on shore of pristine lake

– Lake can switch abruptly to undesirable and potentially irreversible eutrophic state at unknown pollution concentration

• Citizens must decide how much pollution to emit

– Gain small utility from emitting pollution to lake and lose significant utility if lake goes eutrophic

– Deeply uncertain about location of concentration threshold

• Alternative decision approaches include:

– Optimum expected utility

– Precautionary principle

– Robust decision making

Robert J. Lempert and Myles T. Collins., 2007: “Managing the Risk of Uncertain Threshold Response: Comparison of Robust, Optimum, and Precautionary Approaches” Risk Analysis 27 (4), 1009–1026

Use Simple Simulation Model of Lake System to Assess Consequences of Town’s Decisions

•Three “policy levers” describe town’s citizens’ adaptive strategy

– Initial pollution emissions (L0)

– Maximum yearly increase in emissions (L)

– Safety margin (S) – buffer between pollution emissions and estimate of critical threshold (Xcrit)

•Over time, citizens learn true value of critical threshold– Observations increasingly accurate as level of pollution approaches unknown

threshold

Nutrients in LakeNatural Emissions

Nutrient sink

Recycling when eutrophicLt = f(L0,L,S)

AnthropogenicEmissions

Learning

Lt = f L0,ΔL,S( )

Lempert and Collins (2007)

Well Characterized Uncertainty Suggests An Optimal Strategy

0.3 0.4 0.5 0.6 0.7 0.8 0.9

Probability Density

0.3 0.4 0.5 0.6 0.7 0.8 0.9

Probability Density

Optimal Strategy w Uncertainty

Linit 0.37

Safety Margin 3.0

L 0.11

Mean PVU 13.

Probability distribution

for critical threshold

Robust Strategies Trade Some Optimal Performance for Less Sensitivity to Assumptions

• Represent uncertainty about critical threshold with set of multiple, plausible distributions

• Define expected regret of strategy s contingent on distribution i

where strategy s regret is

• Compared to optimal strategy, a robust strategy has small weighted average of best and worst expected regret

• Represent uncertainty about critical threshold with set of multiple, plausible distributions

• Define expected regret of strategy s contingent on distribution i

where strategy s regret is

• Compared to optimal strategy, a robust strategy has small weighted average of best and worst expected regret

ρi X( )

R s,i = Rs x( )ρ i x( )dxx

Rs x( )=Maxs'pvU s' x( )[ ]− pvU s x( )

Vs = zR s,best + 1− z( )R s,worst

0 ≤ z <1

Find Vulnerabilities of Optimal StrategyFind Vulnerabilities of Optimal Strategy

Regret of Strategy A over different values of Xcrit

0.3 0.4 0.5 0.6 0.7 0.8 0.9

Regret of Strategy A

Vulnerabilities

Present value utility of optimum strategy

Compare Regret Over Range of FuturesCompare Regret Over Range of Futures

0 20 40 60 80 100 120 140

Expected Regret over Strategy A's Vulnerable Futures (z=0)

Expected Regret over Initial Priors (z=1)A

0 20 40 60 80 100 120 140

Expected Regret over Strategy A's Vulnerable Futures (z=0)

Expected Regret over Initial Priors (z=1)

Optimal Strategy

Town’s Citizens Have More Robust Options Than Strategy A

0.01 0.1 1 10 100

Odds of Strategy A Priors

Expected Regret

0.01 0.1 1 10 100

Odds of Strategy A Priors

Expected Regret

Optimal Strategy

PotentialRobust Strategy

OutlineOutline

– What impacts may climate change have on IEUA’s current plans?

– What should IEUA do in response?

• Observations

– What impacts may climate change have on IEUA’s current plans?

– What should IEUA do in response?

• Observations

Climate Change Poses Significant Planning Challenge for Water Managers

• Climate change will likely have large but uncertain impacts on supply and demand for water

• “Stationarity is dead”

– Most agencies already include climate (often implicitly) in many decisions

– Amidst all the uncertainty one thing we do know for sure -- tomorrow’s climate will not be like the past’s

• Relaxing this assumption poses key challenges– How do you adjust plans based on uncertain climate projections?

– How do you communicate these plans, especially when uncertain long-term benefits require near-term costs?

• Climate change will likely have large but uncertain impacts on supply and demand for water

• “Stationarity is dead”

– Most agencies already include climate (often implicitly) in many decisions

– Amidst all the uncertainty one thing we do know for sure -- tomorrow’s climate will not be like the past’s

• Relaxing this assumption poses key challenges– How do you adjust plans based on uncertain climate projections?

– How do you communicate these plans, especially when uncertain long-term benefits require near-term costs?

Conducted Vulnerability and Options Analysis for Inland Empire Utilities Agency (IEUA)

– IEUA currently serves 800,000 people

• May add 300,000 by 2025

– Water presents a significant challenge

David G. Groves, Debra Knopman, Robert J. Lempert, Sandra H. Berry, and Lynne Wainfan, Presenting Uncertainty About Climate Change to Water Resource Managers, RAND TR-505-NSF, 2007.

– Current water sources include:

• Groundwater 56%

• Imports 32%

• Recycled 1%

• Surface 8%

• Desalter 2%

• May add 300,000 by 2025

Groves et. al. (2007)

– Current water sources include:

• Groundwater 56%

• Imports 32%

• Recycled 1%

• Surface 8%

• Desalter 2%

Focus of IEUA’s 20 year plan

• May add 300,000 by 2025

Performance of plans

IEUAPlans

System data &climate forecasts

We Built a Model to Assess Performance of IEUA Plans in Different Future States of World

– Model projects future water supply and demand for IEUA service area

• Consistent with IEUA management plans and assumptions

• Reflect plausible trends of climate change

– Model projects future water supply and demand for IEUA service area

• Consistent with IEUA management plans and assumptions

• Reflect plausible trends of climate change

Based on WEAP software toolGroves et. al. (2007)

GCMs Project Plausible Temperature and Precipitation Ranges for Southern California

– Derived from forecasts from 21 GCMs with A1B emissions scenario

– Each forecast weighted by ability to reproduce past climate and level of agreement with other forecasts

– Derived from forecasts from 21 GCMs with A1B emissions scenario

– Each forecast weighted by ability to reproduce past climate and level of agreement with other forecasts

(Tebaldi et al.)

-1 0 1 2 3Change in summer temperature (deg C) from 2000 - 2030

Temperature

-30 -20 -10 0 10 20Percent change in winter precipitation from 2000 - 2030

Precipitation

Generate Future Weather Sequences by Resampling Historic Local Climate Records

KNN method produces hundreds of local weather sequences

– Daily and monthly variability that matches historic Chino climate

– Temperature and precipitation trends that match climate model forecasts

KNN method produces hundreds of local weather sequences

– Daily and monthly variability that matches historic Chino climate

– Temperature and precipitation trends that match climate model forecasts

(Yates et al.)

1980 2000 2020 2040 2060Year

Wetter Neutral

Drier Historical

Precipitation

1980 2000 2020 2040 2060Year

Hotter

Warmer

Neutral

Historical

Temperature

ModelPerformance

of plans

IEUA Plans

System data & climate forecasts

Model Assess Performance of IEUA Plans in Many Different Scenarios

Temp: +1.6Temp: +1.6ooC Precip: -10%C Precip: -10%

Scenario BPlan suffers shortages in adverse future climate

100100

150150

200200

250250

300300

350350

400400

20052005 20102010 20152015 20202020 20252025 20302030

YearYear

Recycled

GroundwaterGroundwater

Local SuppliesLocal Supplies

ImportsImports

Dry-year yieldDry-year yieldSurplusSurplus

ShortageShortage

Temp: +0.7Temp: +0.7ooC Precip: +3%C Precip: +3%

Scenario APlan generates surpluses in benign future climate

100100

150150

200200

250250

300300

350350

400400

20052005 20102010 20152015 20202020 20252025 20302030

YearYear

RecycledRecycled

GroundwaterGroundwater

ImportsImports

SurplusSurplus

Local SuppliesLocal Supplies

Many Uncertain Factors Could Impact the Performance of Current IEUA PlanMany Uncertain Factors Could Impact the Performance of Current IEUA Plan

Natural Processes

• Future temperatures

• Future precipitation

• Changes in groundwater processes

Performance of Management Strategies

• Development of aggressive waste-water recycling program

• Implementation of groundwater replenishment

Costs of Future Supplies and Management Activities

• Imported supplies

• Water use efficiency

Planners in S. California, for Instance, Face aRange of Possible Future Climate ConditionsPlanners in S. California, for Instance, Face aRange of Possible Future Climate Conditions

Summer-time temperature change(2000- 2030)

+.1C +2.1C0

Likely range

Results based on statistical summary of 21 of the world’s best Global Climate Models

Winter-time precipitation change (2000 - 2030)

+8%-19% 0

Likely range

No change Hotter

WetterMuch drier

Natural Processes

0 1.0 2.0 3.0 4.0PV shortage cost ($ billions)

PV supply cost

($ billions) Scenario A

Scenario B

• Adverse climate

• $3.4 billion in supply cost

• $1.9 billion in shortage cost

Current IEUA 2005 Urban Water Management Plan

• Benign climate

• $3.3 billion in supply cost

• $0 in shortage cost

“Scenario Maps” Help Decision Makers Visualize How Plans Evolve Over Many Futures

David G. Groves, Robert J. Lempert, Debra Knopman, Sandra H. Berry: Preparing for an Uncertain Climate Future: Identifying Robust Water Management Strategies, RAND DB-550-NSF, 2008.

PV supply cost

($ billions)

Current IEUA Plan

(200 Scenarios)

PV supply cost

($ billions)

Current IEUA Plan

$3.75 billion cost threshold

Current plan generates high costs in 120 of 200 Scenarios

Discover Key Scenarios in Ensembles of Many Model Runs

1. Ran the model 200 times under different combinations of uncertain factors (e.g. temperature and precipitation trends and others)

2. Used statistical algorithms to identify conditions that lead to 2005 UWMP to perform poorly

3. These factors become key driving forces for “policy-relevant” scenarios

1. Ran the model 200 times under different combinations of uncertain factors (e.g. temperature and precipitation trends and others)

2. Used statistical algorithms to identify conditions that lead to 2005 UWMP to perform poorly

3. These factors become key driving forces for “policy-relevant” scenarios

Number of futures

2.5 3 3.5 4 4.5 5 5.5 6 6.5

NPV total costs ($ billions)

UWMP Forever

High Cost(120 runs)

Statistical analysis suggests factors that

contribute most to these undesirable

outcomes

Current IEUA Plan

Statistical Analysis Suggests Key Factors That Create Vulnerabilities for Existing PlanStatistical Analysis Suggests Key Factors

That Create Vulnerabilities for Existing Plan

Natural Processes

These three factors explain 70% of vulnerabilities of IEUA’s current plans

Response Options May Help IEUA Address These Vulnerabilities

Can Quantify Some, But Not All, Of These CostsCan Quantify Some, But Not All, Of These Costs

Costs increase over time

Average Cost

0 200 400 600 800 1000 1200

Saved through efficiency

Recycled

Stormwater Replenishment*

Groundwater

Recycled Replenishment*

Imported (Tier 1)

Imported Replenishment*

Imported (Tier 2)

Desalted Groundwater

Shortages

Cost in 2005 ($/AF)* includes the cost of spreading

Should IEUA Act Now or Later to Reduce Potential Climate Vulnerabilities?

Act now to Act now to augment augment

2005 Plan?2005 Plan?

Monitor, and take Monitor, and take additional action additional action if supplies drop if supplies drop

too lowtoo low

In 2015, 2020, 2025, ….

YESImplement Implement additional additional efficiency, efficiency,

recycling, and recycling, and replenishmentreplenishment

In 2015, 2020, 2025, ….

Monitor, and take Monitor, and take additional action additional action if supplies drop if supplies drop

too lowtoo low

Compare Nine Strategies Over200 Scenarios Reflecting Key Uncertainties

0 40 8060 100 12020

Static options

Update options

Number of Scenarios (PV Costs > $3.75 billion)

Current Plan forever

Current Plan + DYY and recycling

Current Plan + replenishment

Current Plan with updates

Current Plan + replenishment with updates

Current Plan + efficiency

Current Plan + efficiency with updates

Current Plan + DYY and recycling with updates

Current Plan + all enhancements

Just Allowing IEUA’s Current Plan to UpdateReduces Vulnerability Substantially

0 40 8060 100 12020

Static options

Update options

From 120Down to 30

Number of Scenarios (PV Costs > $3.75 billion)

Current Plan forever

Current Plan + DYY and recycling

Current Plan + replenishment

Acting NowReduces Future Vulnerabilities Even More

0 20 403010

Static options

Update options

Number of Scenarios(PV Costs > $3.75 billion)

Acting NowReduces Future Vulnerabilities Even More

0 20 403010

Static options

Update options

Number of Scenarios(PV Costs > $3.75 billion)

Implementation becomes

more challenging

This analysis helped IEUA decide to make more near-term efficiency investments, and to monitor performance and adapt

as needed down the roadGroves et. al. (2008)

OutlineOutline

• Observations

Conducted Elicitations Among IEUA’s Planners and Community to Estimate

Likelihood of Achieving Goals

Conducted Elicitations Among IEUA’s Planners and Community to Estimate

Likelihood of Achieving Goals

Density

40 50 60 70 80Recycling

Density

80 90 100 110 120 130GW

Recycling Replenishment

Goal GoalMissgoal

Missgoal

Probability of meeting UWMP goals

Meet Goals

Miss Goals Groves et. al. (2007)

Natural Processes

Meet recycling goal

Meet replenishment goal

Future climate

New conservation

Percolation decrease

Climate on imports

Miss ExceedMeet

Drier Wetter

-5% +20%

-20% 0%

Weak Strong

Explains 70% of high cost cases

Analysis Suggests Factors That Cause Severe Shortages for IEUA’s 20 Year Plan

Climate-related uncertainties facing IEUA Climate-related uncertainties facing IEUA

RDM Enables Effective Planning Based on Multiple Views of FutureRDM Enables Effective Planning

Based on Multiple Views of Future

• Use many scenarios to imagine the future

– Not a single forecast

• Seek robust strategies that do well across many scenarios assessed according to several values

– Not optimal strategies

• Employ strategies that evolve over time in response to changing conditions

– Not "fixed" strategies

• Use computer as “prosthesis for the imagination”

– Not a calculator

• Use many scenarios to imagine the future

– Not a single forecast

• Seek robust strategies that do well across many scenarios assessed according to several values

– Not optimal strategies

• Employ strategies that evolve over time in response to changing conditions

– Not "fixed" strategies

• Use computer as “prosthesis for the imagination”

– Not a calculator

More InformationMore InformationDavid G. Groves, Robert J. Lempert, Debra Knopman, Sandra H. Berry: Preparing for an Uncertain Climate

Future: Identifying Robust Water Management Strategies, RAND DB-550-NSF, 2008.

Groves, David G, David Yates, Claudia Tebaldi, 2008: “Developing and Applying Uncertain Global Climate Change Projections for Regional Water Management Planning,” Water Resources Research, 44(12): W12413

David G. Groves and Robert J. Lempert, 2007: A new analytic method for finding policy-relevant scenarios, Global Environmental Change 17, 73-85.

Robert J. Lempert, Steven W. Popper, Steven C. Bankes, 2003: Shaping the Next One Hundred Years: New Methods for Quantitative, Long-Term Policy Analysis, RAND MR-1626-RPC, Aug.

www.rand.org/ise/projects/improvingdecisions/

David G. Groves, Robert J. Lempert, Debra Knopman, Sandra H. Berry: Preparing for an Uncertain Climate Future: Identifying Robust Water Management Strategies, RAND DB-550-NSF, 2008.

Groves, David G, David Yates, Claudia Tebaldi, 2008: “Developing and Applying Uncertain Global Climate Change Projections for Regional Water Management Planning,” Water Resources Research, 44(12): W12413

David G. Groves and Robert J. Lempert, 2007: A new analytic method for finding policy-relevant scenarios, Global Environmental Change 17, 73-85.

Robert J. Lempert, Steven W. Popper, Steven C. Bankes, 2003: Shaping the Next One Hundred Years: New Methods for Quantitative, Long-Term Policy Analysis, RAND MR-1626-RPC, Aug.

www.rand.org/ise/projects/improvingdecisions/

Thank you!

We Also Evaluated How This Analysis Affected Policy-Makers’ Views

• Four IEUA workshops presented modeling results to participants including:

– Agency professional managers and technical staff

– Local elected officials

– Community stakeholders

• “Real-time” surveys measured participants’

– Understanding of concepts

– Willingness to adjust policy choices based on information presented

– Views on RDM

• Four IEUA workshops presented modeling results to participants including:

– Agency professional managers and technical staff

– Local elected officials

– Community stakeholders

• “Real-time” surveys measured participants’

– Understanding of concepts

– Willingness to adjust policy choices based on information presented

– Views on RDM

First Three Workshops Compared Alternative Approaches to Uncertainty

Workshop design approximates on-going laboratory experiments

Compared three approaches

• Traditional qualitative scenarios

• Probabilistic forecasts

• RDM with Scenario discovery

RDM Scenarios More Useful, But More Difficult to Understand

Questionnaire item Traditional Scenarios

Scenario Discovery

Provides results that can be used in planning

Agree somewhat

Agree strongly

Provides information on how to improve plan

Agree somewhat

Is easy to explain to decisionmakers

Agree somewhat

Disagree strongly

• Traditional scenarios – Gave IEUA much of the information they needed

– Emphasized the importance of achieving goals in IEUA’s plan

• Scenario Discovery – Provided more useful information

– Sparked discussion of adaptive strategies

RDM Scenarios More Useful, But More Difficult to Understand

Questionnaire item Traditional Scenarios

Scenario Discovery

Provides results that can be used in planning

Agree somewhat

Agree strongly

Provides information on how to improve plan

Agree somewhat

Is easy to explain to decisionmakers

Agree somewhat

Disagree strongly

Fourth (Adaptive Strategy) Workshop Compared Different Presentations of RDM

Participants reported:– RDM helped support comparison of climate-related risks and choice among

– Preference for scatter plot over histogram scenario displays

After the workshop:– 35% said consequences of bad climate change now appeared “more

serious” than before

– 40% thought the likelihood of of bad climate change outcomes for the IEUA was “greater” than before

– 75% though the ability of IEUA planners to plan for and manage effects was “greater” than before

Overall, analysis increased:– Perceived likelihood of serious climate impacts

– Confidence that IEUA could take effective actions to reduce its vulnerability to climate change

– Support for near-term efficiency enhancements to current IEUA plan

Participants reported:– RDM helped support comparison of climate-related risks and choice among

– Preference for scatter plot over histogram scenario displays

After the workshop:– 35% said consequences of bad climate change now appeared “more

serious” than before

– 40% thought the likelihood of of bad climate change outcomes for the IEUA was “greater” than before

– 75% though the ability of IEUA planners to plan for and manage effects was “greater” than before

Overall, analysis increased:– Perceived likelihood of serious climate impacts

– Confidence that IEUA could take effective actions to reduce its vulnerability to climate change

– Support for near-term efficiency enhancements to current IEUA planGroves et. al. (2008)

ObservationsObservations• Analysis suggests IEUA’s current long-range plans:

– Vulnerable to climate change

– Can be made more resilient by near-term conservation, attention to storm intensity, and effective future monitoring and updating

• Measurements suggest

– RDM analysis effectively shifted views on seriousness of climate challenges and appropriate responses, but requires more work to be easily understood by policy-makers

– Importance of linking effective response options with presentation of climate uncertainty

• Currently using this approach to help several major water agencies include climate in their long-range plans

• Analysis suggests IEUA’s current long-range plans:

– Vulnerable to climate change

– Can be made more resilient by near-term conservation, attention to storm intensity, and effective future monitoring and updating

• Measurements suggest

– RDM analysis effectively shifted views on seriousness of climate challenges and appropriate responses, but requires more work to be easily understood by policy-makers

– Importance of linking effective response options with presentation of climate uncertainty

• Currently using this approach to help several major water agencies include climate in their long-range plans

decision making under deep uncertainty robert lempert director rand pardee center for longer range...

quantitative decision

blind decisionmakers

s energy use

alternative approaches

s forecasters

response option analysis

emitting pollution

projections of u

Documents

new methods to bridge long-term policy analysis and robust...

arie kapteyn, rand tatiana andreyeva, pardee rand graduate...

c. pardee -2-

antelope pardee transmission line...

for longer range global policy and the future human...

a: overview of project analysis and workshop contributions...

stakeholder-driven, multi-sectoral climate adaptation in...

d: initial uncertainty analysis for water and energy sectors...

arthur b. pardee - national academy of sciences · 2020....

museum newsletter - pardee home museum

aim higher: gaining a leadership...

the pardee papers / no. 15 / august 2011

pardee pram an

lempert . hukum jalam lmp a runent t l1uk'ullm 'a

fighting obesity in the united states with state legislation...

information needs for developing robust adaptive strategies...

educational effectiveness report pardee rand · pdf...

pardee resources company 3rd quarter 2015

antelope pardee transmission line project

water management and climate change in the us- mexico border...