New Methods to Bridge Long-Term Policy Analysis

and Robust Decision Making

New Methods to Bridge Long-Term Policy Analysis

and Robust Decision Making

Robert Lempert

DirectorRAND Frederick S. Pardee Center for Longer Range Global Policy and the Future Human Condition

BLOSSOM WorkshopEuropean Environmental Agency

April 30, 2008

2 4-30-08

Long-Term Decisions Present Difficult Challenges

Long-Term Decisions Present Difficult Challenges

• Long-term decisions occur

– When reflecting on potential events decades or more in the future

– Causes decision makers to choose near-term actions different than those they would otherwise pursue

• Long-term decisions occur

– When reflecting on potential events decades or more in the future

– Causes decision makers to choose near-term actions different than those they would otherwise pursue

• Long-term policy analysis often fails to persuade because

– Conditions of deep uncertainty prevail

– Short-term needs loom large

– Views of the future often anchored in the present

• Long-term policy analysis often fails to persuade because

– Conditions of deep uncertainty prevail

– Short-term needs loom large

– Views of the future often anchored in the present

3 4-30-08

Long-Term Decisions Present Difficult Challenges

Long-Term Decisions Present Difficult Challenges

• Long-term decisions occur

– When reflecting on potential events decades or more in the future

– Causes decision makers to choose near-term actions different than those they would otherwise pursue

• Long-term decisions occur

– When reflecting on potential events decades or more in the future

– Causes decision makers to choose near-term actions different than those they would otherwise pursue

“Missions to Mars” at

Disneyland’s Tomorrowland ca 1955

• Long-term policy analysis often fails to persuade because

– Conditions of deep uncertainty prevail

– Short-term needs loom large

– Views of the future often anchored in the present

• Long-term policy analysis often fails to persuade because

– Conditions of deep uncertainty prevail

– Short-term needs loom large

– Views of the future often anchored in the present

4 4-30-08

Scenarios Attractive for Long-Term Analysis, but Have Weakness in Contentious Public Debates

Scenarios Attractive for Long-Term Analysis, but Have Weakness in Contentious Public Debates

• At their best, scenarios can help decision makers

– Reduce overconfidence

– Challenge their mental models

– Overcome organizational and psychological barriers to considering threatening or inconvenient futures

• At their best, scenarios can help decision makers

– Reduce overconfidence

– Challenge their mental models

– Overcome organizational and psychological barriers to considering threatening or inconvenient futures

5 4-30-08

Scenarios Capture the Key Concept That a Multiplicity of Plausible Futures May Be as Close as We Get to the Truth

Scenarios Capture the Key Concept That a Multiplicity of Plausible Futures May Be as Close as We Get to the Truth

Rabbi Eliezer Ashkenazi (1580) chose to interpret the Tower of Babel story not as a challenge to divine power to which the Lord's response was to divide the human race but rather the opposite.

He saw the story as an attempt to establish a universal religious regime which God "was obliged to separate

…since the proliferation of doctrines aids and stimulates the investigator to attain the desired truths.”

6 4-30-08

Scenarios Attractive for Long-Term Analysis, but Have Weakness in Contentious Public Debates

Scenarios Attractive for Long-Term Analysis, but Have Weakness in Contentious Public Debates

• At their best, scenarios can help decision makers

– Reduce overconfidence

– Challenge their mental models

– Overcome organizational and psychological barriers to considering threatening or inconvenient futures

• But in contentious public debates, scenario methods can have difficulty

– Engaging mental models of diverse stakeholders

– Systematically informing decisions under uncertainty

– Addressing surprise and discontinuities

• At their best, scenarios can help decision makers

– Reduce overconfidence

– Challenge their mental models

– Overcome organizational and psychological barriers to considering threatening or inconvenient futures

• But in contentious public debates, scenario methods can have difficulty

– Engaging mental models of diverse stakeholders

– Systematically informing decisions under uncertainty

– Addressing surprise and discontinuities

7 4-30-08

OutlineOutline

• Robust decision making (RDM) provides framework for effective long-term analysis

• RDM’s “Scenario Discovery” approach offers useful scenario concept for public debates

• Recent work measures the impacts of these approaches with decision makers

• Robust decision making (RDM) provides framework for effective long-term analysis

• RDM’s “Scenario Discovery” approach offers useful scenario concept for public debates

• Recent work measures the impacts of these approaches with decision makers

8 4-30-08

RDM Views Scenarios As Part of Process Identifying and Building Consensus for Robust Strategies

RDM Views Scenarios As Part of Process Identifying and Building Consensus for Robust Strategies

Key Robust Decision Making Concepts:

• Construct ensemble of long-term scenarios that highlight key tradeoffs among near-term policy choices

• Consider near-term choices as one step in a sequence of decisions that evolve over time

• Use robustness criteria to compare alternative strategies

Key Robust Decision Making Concepts:

• Construct ensemble of long-term scenarios that highlight key tradeoffs among near-term policy choices

• Consider near-term choices as one step in a sequence of decisions that evolve over time

• Use robustness criteria to compare alternative strategies

– A robust strategy performs well compared to the alternatives over a wide range of plausible futures

– A robust strategy performs well compared to the alternatives over a wide range of plausible futures

9 4-30-08

New Technology Allows Computer to Serve As “Prosthesis for the Imagination”

New Technology Allows Computer to Serve As “Prosthesis for the Imagination”

• Robust Decision Making (RDM) is a quantitative decision analytic approach that

– Characterizes uncertainty with multiple, rather than single, views of the future

– Evaluates alternative decision options with a robustness, rather than optimality, criterion

• Robust Decision Making (RDM) is a quantitative decision analytic approach that

– Characterizes uncertainty with multiple, rather than single, views of the future

– Evaluates alternative decision options with a robustness, rather than optimality, criterion

10 4-30-08

New Technology Allows Computer to Serve As “Prosthesis for the Imagination”

New Technology Allows Computer to Serve As “Prosthesis for the Imagination”

• Robust Decision Making (RDM) is a quantitative decision analytic approach that

– Characterizes uncertainty with multiple, rather than single, views of the future

– Evaluates alternative decision options with a robustness, rather than optimality, criterion

– Iteratively identifies vulnerabilities of plans and evaluates potential responses

• Robust Decision Making (RDM) is a quantitative decision analytic approach that

– Characterizes uncertainty with multiple, rather than single, views of the future

– Evaluates alternative decision options with a robustness, rather than optimality, criterion

– Iteratively identifies vulnerabilities of plans and evaluates potential responses

Candidate strategy

Identify vulnerabilities

Assess alternatives for ameliorating vulnerabilities

• RDM combines key advantages of scenario planning and quantitative decision analysis in ways that

– Decision makers find credible

– Contribute usefully to contentious debates

• RDM combines key advantages of scenario planning and quantitative decision analysis in ways that

– Decision makers find credible

– Contribute usefully to contentious debates

11 4-30-08

Stylized Sustainability Example Summarizes RDM Approach

Stylized Sustainability Example Summarizes RDM Approach

What near-term actions can help ensure economic growth and environmental quality over the 21st century?

Economic growth rate (%)

1.0 2.0 3.0 4.0

–1.0

0

1.0

5.0

3.0

4.0

2.0

0

India since 1960

U.S. 1890-1930

U.S. since 1950

U.S. in 20th century

China since 1960

Brazil since 1980

Russia since 1993

Decoupling rate (%)

(Rate at which technology,

without regulation,

reduces emissions per

GDP)

Analysis suggests testing alternative strategies over this range of futures

12 4-30-08

Compare “Fixed” Near-Term Strategies Across Scenarios

Compare “Fixed” Near-Term Strategies Across Scenarios

Near term

Choose policies

Assume near-term policy continues until changed by future generations

Future decision-makers recognize

and correct our mistakes

Future

RAND MR-1626-RPC

13 4-30-08

1.0 2.0 3.0 4.0

–1.0

0

1.0

5.0

3.0

4.0

2.0

0

Conventionalworld

scenario

Strategy’sPerformanceStrategy’sPerformance

No regretMild

A lotOverwhelming

1.0 2.0 3.0 4.0

–1.0

0

1.0

5.0

3.0

4.0

2.0

0

Economic growth rate (%)

Decouplingrate (%)

Stay the CourseStay the Course

U.S. in 20th century

U.S. since 1950

Initial Scan Suggests No Fixed Emission Reduction Target Is Robust

Initial Scan Suggests No Fixed Emission Reduction Target Is Robust

StrategyVulner-abilities

Alternatives

1. Run simulation thousands of times

2. Display “scenario maps” showing deviation of proposed strategy from optimality over many futures

14 4-30-08

1.0 2.0 3.0 4.0

–1.0

0

1.0

5.0

3.0

4.0

2.0

0

Conventionalworld

scenario

Strategy’sPerformanceStrategy’sPerformance

No regretMild

A lotOverwhelming

1.0 2.0 3.0 4.0

–1.0

0

1.0

5.0

3.0

4.0

2.0

0

Economic growth rate (%)

Decouplingrate (%)

Stay the CourseStay the Course

U.S. in 20th century

U.S. since 1950

Initial Scan Suggests No Fixed Emission Reduction Target Is Robust

Initial Scan Suggests No Fixed Emission Reduction Target Is Robust

1.0 2.0 3.0 4.0

–1.0

0

1.0

5.0

3.0

4.0

2.0

0

Conventionalworld

scenario

1.0 2.0 3.0 4.0

–1.0

0

1.0

5.0

3.0

4.0

2.0

0

Crash EffortCrash Effort

U.S. in 20th century

U.S. since 1950

15 4-30-08

Craft Near-Term Adaptive Strategy That Aims to Balance Environmental and Economic Goals

Craft Near-Term Adaptive Strategy That Aims to Balance Environmental and Economic Goals

Present Future

NODoes the carrying capacity change?

Choose policies to maximize utility

Determine best policy to meet milestone

Select near-term milestone

YES

Is milestone achievable with

current approach?

Relax milestone

YES

NO

Implement policy

RAND MR-1626-RPC

16 4-30-08

Robust Strategy Reduces Uncertainty By Performing Well No Matter What Future Comes to Pass

Robust Strategy Reduces Uncertainty By Performing Well No Matter What Future Comes to Pass

Adaptive StrategyAdaptive Strategy

Economic growth rate (%)1.0 2.0 3.0 4.0

–1.0

0

1.0

5.0

3.0

4.0

2.0

0

No regretMild

A lotOverwhelming

U.S. in 19thcentury

U.S. since 1950

U.S. in 20thU.S. in 20thcenturycentury

Decoupling rate (%)

StrategyVulner-abilities Alternatives

17 4-30-08

OutlineOutline

• Robust decision making (RDM) provides framework for effective long-term analysis

• RDM’s “Scenario Discovery” approach offers useful scenario concept for public debates

• Recent work measures the impacts of these approaches with decision makers

• Robust decision making (RDM) provides framework for effective long-term analysis

• RDM’s “Scenario Discovery” approach offers useful scenario concept for public debates

• Recent work measures the impacts of these approaches with decision makers

18 4-30-08

Long-Term RDM Scenarios Highlight Trade-offs Among Near-Term DecisionsLong-Term RDM Scenarios Highlight

Trade-offs Among Near-Term Decisions1. Run simulation model for many different combinations of

uncertain input parameters

2. Identify those clusters of cases that highlight tradeoffs among near-term candidate strategies

1. Run simulation model for many different combinations of uncertain input parameters

2. Identify those clusters of cases that highlight tradeoffs among near-term candidate strategies

Candidate strategy

Identify vulnerabilities

Assess alternatives for ameliorating vulnerabilities

• Example future conditions highlighting near-term tradeoffs:

– 2007 Congressional Reauthorization of Terrorism Risk Insurance Act: In what situations would ending TRIA cost the taxpayer more than retaining the program?

– California water planning: Under what conditions would future climate change impacts suggest modifying current long-range water management plans?

• Example future conditions highlighting near-term tradeoffs:

– 2007 Congressional Reauthorization of Terrorism Risk Insurance Act: In what situations would ending TRIA cost the taxpayer more than retaining the program?

– California water planning: Under what conditions would future climate change impacts suggest modifying current long-range water management plans?

19 4-30-08

Scenario Discovery Implements This Concept for Computer-Assisted Scenario Development

Scenario Discovery Implements This Concept for Computer-Assisted Scenario Development

. . .. . ......

1. Indicate policy-relevant cases in database of simulation results

20 4-30-08

Scenario Discovery Implements This Concept for Computer-Assisted Scenario Development

Scenario Discovery Implements This Concept for Computer-Assisted Scenario Development

. . .. . ......Uncertain input

variable 2

1. Indicate policy-relevant cases in database of simulation results

2. Statistical analysis finds low-dimensional clusters with high density of these cases

Uncertain input variable 1

21 4-30-08

Scenario Discovery Implements This Concept for Computer-Assisted Scenario Development

Scenario Discovery Implements This Concept for Computer-Assisted Scenario Development

. . .. . ......Uncertain input

variable 2

1. Indicate policy-relevant cases in database of simulation results

2. Statistical analysis finds low-dimensional clusters with high density of these cases

3. Clusters represent scenarios and driving forces of interest to decision makers

Uncertain input variable 1

22 4-30-08

Scenario Discovery Implements This Concept for Computer-Assisted Scenario Development

Scenario Discovery Implements This Concept for Computer-Assisted Scenario Development

. . .. . ......Uncertain input

variable 2

1. Indicate policy-relevant cases in database of simulation results

2. Statistical analysis finds low-dimensional clusters with high density of these cases

3. Clusters represent scenarios and driving forces of interest to decision makers

Uncertain input variable 1

Density: • How many cases inside the

scenario are policy-relevant? (e.g. 75%)

Coverage: • How many of all the policy-

relevant cases do the scenarios include? (e.g. 82%)

Interpretability: • Is the number of scenarios

and driving forces sufficiently small to understand? (e.g. 1 scenario with two driving forces)

Density: • How many cases inside the

scenario are policy-relevant? (e.g. 75%)

Coverage: • How many of all the policy-

relevant cases do the scenarios include? (e.g. 82%)

Interpretability: • Is the number of scenarios

and driving forces sufficiently small to understand? (e.g. 1 scenario with two driving forces)

Approach provides measures of merit for scenario quality

23 4-30-08

Scenario Discovery May Improve Impact of Scenarios in Contentious Public Debates

Scenario Discovery May Improve Impact of Scenarios in Contentious Public Debates

• For instance, recent scenario discovery work on U.S. Federal terrorism insurance program was

– Based on a scenario not considered in the official budgetary analysis by government agencies

– Quoted on the floor of the United States Senate by a program supporter

– Criticized as “insidious” by program opponents

• But neither side in the debate could gain traction by quarrelling with our choice of scenario and its key driving forces

• For instance, recent scenario discovery work on U.S. Federal terrorism insurance program was

– Based on a scenario not considered in the official budgetary analysis by government agencies

– Quoted on the floor of the United States Senate by a program supporter

– Criticized as “insidious” by program opponents

• But neither side in the debate could gain traction by quarrelling with our choice of scenario and its key driving forces

24 4-30-08

OutlineOutline

• Robust decision making (RDM) provides framework for effective long-term analysis

• RDM’s “Scenario Discovery” approach offers useful scenario concept for public debates

• Recent work measures the impacts of these approaches with decision makers

• Robust decision making (RDM) provides framework for effective long-term analysis

• RDM’s “Scenario Discovery” approach offers useful scenario concept for public debates

• Recent work measures the impacts of these approaches with decision makers

25 4-30-08

How Does Climate Change Affect California’s Inland Empire Utilities Agency (IEUA)?

IEUA currently serves 800,000 people

– May add 300,000 by 2025

Water presents a significant challenge

26 4-30-08

How Does Climate Change Affect California’s Inland Empire Utilities Agency (IEUA)?

IEUA currently serves 800,000 people

– May add 300,000 by 2025

Water presents a significant challenge

IEUA’s 2005 long-range Urban Water Management Plan (UWMP)

– Aims to meet needs of growing population, but

– Did not address climate change

27 4-30-08

How Does Climate Change Affect California’s Inland Empire Utilities Agency (IEUA)?

IEUA currently serves 800,000 people

– May add 300,000 by 2025

Water presents a significant challenge

We conducted several analyses to help IEUA assess impact of climate change on their 2005 UWMP

– Traditional scenarios

– Probabilistic risk analysis

– Scenario Discovery

28 4-30-08

Conducted Workshops to Measure Impact of Alternative Analyses on IEUA

Conducted Workshops to Measure Impact of Alternative Analyses on IEUA

– Four IEUA workshops presented modeling results to participants including:

• Agency professional managers and technical staff

• Local elected officials

• Community stakeholders

– “Real-time” surveys measured participants’

• Understanding of concepts

• Willingness to adjust policy choices based on information presented

• Views on RDM, traditional scenarios, and probabilistic risk analysis

– Four IEUA workshops presented modeling results to participants including:

• Agency professional managers and technical staff

• Local elected officials

• Community stakeholders

– “Real-time” surveys measured participants’

• Understanding of concepts

• Willingness to adjust policy choices based on information presented

• Views on RDM, traditional scenarios, and probabilistic risk analysis

29 4-30-08

Disagree strongly

Agree somewhat

Agree somewhat

Agree somewhat

Agree strongly

Agree somewhat

Participants Ranked Scenario Discovery More Useful, But More Difficult to Understand

Participants Ranked Scenario Discovery More Useful, But More Difficult to Understand

Is easy to explain to decisionmakers

Provides information on how to improve plan

Provides results that can be used in planning

Scenario Discovery

Traditional Scenarios

Questionnaire item from first 3 workshops

– Traditional scenarios

• Gave IEUA much of the information they needed

• Emphasized the importance of achieving goals in IEUA’s plan

– Scenario discovery

• Provided more useful information for evaluating alternatives to plan

• Sparked discussion of adaptive strategies

30 4-30-08

Surveys Suggest RDM Analysis Changed Participants’ Views

Surveys Suggest RDM Analysis Changed Participants’ Views

Participants provided:

– Information on most effective RDM visualizations

After the workshop:

– 35% said consequences of bad climate change now appeared “more serious” than before

– 75% though the ability of IEUA planner to plan for and manage effects was “greater” than before

Overall, analysis:

– Increased support for near-term modifications to current IEUA plan

– Suggests that participants’ willingness to acknowledge a serious climate change threat increased after they felt more confident they could address the threat

Participants provided:

– Information on most effective RDM visualizations

After the workshop:

– 35% said consequences of bad climate change now appeared “more serious” than before

– 75% though the ability of IEUA planner to plan for and manage effects was “greater” than before

Overall, analysis:

– Increased support for near-term modifications to current IEUA plan

– Suggests that participants’ willingness to acknowledge a serious climate change threat increased after they felt more confident they could address the threat

31 4-30-08

Key ConceptsKey Concepts

Choose scenarios to highlight tradeoffs among near-term decisions

Otherwise number of potentially interesting scenarios remains unlimited

Use analytics to facilitate human creativity in designing policies robust across many futures

Measure scenarios’ impacts on decision makers to help improve process and methods

Designing measurements makes purpose clear

Can use framework for general thinking about long-term policy under deep uncertainty

Not just as basis for a modeling exercise

Choose scenarios to highlight tradeoffs among near-term decisions

Otherwise number of potentially interesting scenarios remains unlimited

Use analytics to facilitate human creativity in designing policies robust across many futures

Measure scenarios’ impacts on decision makers to help improve process and methods

Designing measurements makes purpose clear

Can use framework for general thinking about long-term policy under deep uncertainty

Not just as basis for a modeling exercise

32 4-30-08

For More InformationFor More Information

http://www.rand.org/international_programs/pardee/

Thank you!Thank you!