designing a risk model michael schilmoeller thursday, december 2, 2010 saac

Designing a Risk Model

Michael SchilmoellerThursday, December 2, 2010

SAAC

2

Overview

• Scope of uncertainty• Decision trees (briefly) and Monte Carlo

simulation• Implications of cost and risk accuracy to

the number of futures• The number of possible plans and finding

the “best” plan• Computational alternatives

3

Sources of Uncertainty

Scope of uncertainty

• Fifth Power Plan– Load requirements– Gas price– Hydrogeneration– Electricity price– Forced outage rates– Aluminum price– Carbon allowance cost– Production tax credits– Renewable Energy Credit

(Green tag value)

• Sixth Power Plan– aluminum price and

aluminum smelter loads were removed

– Power plant construction costs

– Technology availability– Conservation costs and

performance

4

Impact on NPV Costs and Risk

0

10

20

30

40

50

60

70

80

9030

40

50

60

70

80

90

100

110

120

130

140

150

160

170

180

190

200

210

220

230

Freq

uenc

y

Billions of 2006 Constant Dollars

NPV 20-Year Study Costs

Scope of uncertainty

C:\Documents and Settings\Michael Schilmoeller\Desktop\NWPCC - Council\SAAC\Presentation materials\L813 NPV Costs.xlsm

5

Decision Trees

• Estimating the number of branches– Assume possible 3 values (high, medium, low) for each of 9

variables, 80 periods, with two subperiods each; plus 70 possible hydro years, one for each of 20 years, on- and off-peak energy determined by hydro year

– Number of estimates cases, assuming independence: 6,048,000

• Studies, given equal number k of possible values for n uncertainties:

• Impact of adding an uncertainty:

Decision trees & Monte Carlo simulation

iesuncertaint values, , nkkN n

kN

N

1

6

Monte Carlo Simulation

• MC represents the more likely values• The number of samples is determined by the

accuracy requirement for the statistics of interest• The number of samples mk necessary to obtain

a given level of precision in estimates of averages grows much more slowly than the number of variables k:

Decision trees & Monte Carlo simulation

k

k

m

m

k

k 11

7

Overview





8

Monte Carlo Samples

• How many samples are necessary to achieve reasonable cost and risk estimates?

• How precise is the sample mean of the tail, that is, TailVaR90?

Implication to Number of Futures

9

Relationship Between the Size of the Sample and the Accuracy

• Depends on knowledge of the distribution• Given the distribution, requires knowledge

of how the accuracy depends on sample size


10

Central Limit Theorem

• Both our cost and our risk estimates are averages

• CLT says that as the number of samples used to estimate the mean increases, the distribution of the sample means tends to normal

• Unfortunately, it doesn’t say how fast it tends to normal or how the shape of the underlying distribution affects the rate of approach


11

TailVaR90

0123456789

10111213141516

109

115

121

127

133

139

145

151

157

163

169

175

181

187

193

199

205

211

217

223

Freq

uenc

y


Tail Risk


C:\Documents and Settings\Michael Schilmoeller\Desktop\NWPCC - Council\SAAC\Presentation materials\L813 NPV Costs 02.xlsm

12

Assumed Distribution

0123456789

10111213141516

109

115

121

127

133

139

145

151

157

163

169

175

181

187

193

199

205

211

217

223

Freq

uenc

y


Tail Risk



13

Set Up a Sampler

0123456789

10111213141516

109

115

121

127

133

139

145

151

157

163

169

175

181

187

193

199

205

211

217

223

Freq

uenc

y


Tail Risk



14

Dependence of Tail Average on Sample Size

0

10

20

30

40

50

60

70

80

109

111

113

115

117

119

121

123

125

127

129

131

133

135

137

139

141

143

145

147

5 samples per average0

10

20

30

40

50

60

70

80

90

30

40

50

60

70

80

90

100

110

120

130

140

150

160

170

180

190

200

210

220

230

Freq

uenc

y




C:\Documents and Settings\Michael Schilmoeller\Desktop\NWPCC - Council\SAAC\Presentation materials\L813 NPV Costs 02.xlsm, worksheet “Simulation”

15

0

20

40

60

80

100

120

109

111

113

115

117

119

121

123

125

127

129

131

133

135

137

139

141

143

145

147

10 samples per average




0

10

20

30

40

50

60

70

80

90

30

40

50

60

70

80

90

100

110

120

130

140

150

160

170

180

190

200

210

220

230

Freq

uenc

y



16

0

204060

80100120

140160180

109

111

113

115

117

119

121

123

125

127

129

131

133

135

137

139

141

143

145

147





0

10

20

30

40

50

60

70

80

90

30

40

50

60

70

80

90

100

110

120

130

140

150

160

170

180

190

200

210

220

230

Freq

uenc

y



17Implication to Number of Futures


0

10

20

30

40

50

60

70

11

6

11

6.7

5

11

7.5

11

8.2

5

11

9

11

9.7

5

12

0.5

12

1.2

5

12

2

12

2.7

5

12

3.5

12

4.2

5

12

5

12

5.7

5

12

6.5

12

7.2

5

12

8

12

8.7

5

12

9.5

13

0.2

5

13

1

13

1.7

5


σ=2.040

C:\Documents and Settings\Michael Schilmoeller\Desktop\NWPCC - Council\SAAC\Presentation materials\L813 NPV Costs 02.xlsm, worksheet “Samples_50”

0

10

20

30

40

50

60

70

80

90

30

40

50

60

70

80

90

100

110

120

130

140

150

160

170

180

190

200

210

220

230

Freq

uenc

y



18Implication to Number of Futures


0

10

20

30

40

50

60

70

11

6

11

6.7

5

11

7.5

11

8.2

5

11

9

11

9.7

5

12

0.5

12

1.2

5

12

2

12

2.7

5

12

3.5

12

4.2

5

12

5

12

5.7

5

12

6.5

12

7.2

5

12

8

12

8.7

5

12

9.5

13

0.2

5

13

1

13

1.7

5


C:\Documents and Settings\Michael Schilmoeller\Desktop\NWPCC - Council\SAAC\Presentation materials\L813 NPV Costs 02.xlsm, worksheet “Samples_75”

σ=1.677

0

10

20

30

40

50

60

70

80

90

30

40

50

60

70

80

90

100

110

120

130

140

150

160

170

180

190

200

210

220

230

Freq

uenc

y



19

Chi-Squared (X2) Tests

• Check the hypothesis that our sample has the variation from normal by chance (p)

• 50 samples per calculation: p=0.50• 75 samples per calculation: p=0.10


20

Accuracy and Sample Size• Estimated accuracy of TailVaR90 statistic is

still only ± $3.3 B (2σ)!*

0

10

20

30

40

50

60

70

80

90

30

40

50

60

70

80

90

100

110

120

130

140

150

160

170

180

190

200

210

220

230

Freq

uenc

y




0

10

20

30

40

50

60

70

116

116.

7511

7.5

118.

25 119

119.

7512

0.5

121.

25 122

122.

7512

3.5

124.

25 125

125.

7512

6.5

127.

25 128

128.

7512

9.5

130.

25 131

131.

75


*Stay tuned to see why the precision is actually 1000x better than this!

21

Accuracy Relative to the Efficient Frontier

123200

124200

125200

126200

127200

128200

129200

77000 78000 79000 80000 81000 82000 83000

Ris

k (N

PV

$2

00

6 M

)

Cost (NPV $2006 M)

L813

L813 L813 Frontier

C:\Backups\Plan 6\Studies\L813\Analysis of Optimization Run_L813vL811.xls


22

Conclusion

• At least 75 samples are needed for determining the value of our risk metric– Known distribution of statistic– The precision of the sample

• Our risk metric is 1/10 of the total number of futures

• We need to test our plan under 750 futures to obtain defensible results


23

Overview





24

Finding the Best Plan

• Each plan is exposed to exactly the same set of futures, except for electricity price

• Look for the plan that minimizes cost and risk

• Challenge: there may be many plans

Implication to Number of Plans

25

Avogadro’s Number

min start max states step size

DRAC 1 1 2 2 Discrete (1) 2DRSH 1 1 4 4 Discrete (1) x 4DRAG 1 1 4 4 Discrete (1) x 4DRIN 1 1 4 4 Discrete (1) x 4Lost Opp Cnsvsn 0 50 100 11 Discrete (10) x 11Discretionary Cnsvsn 0 10 100 11 Discrete (10) x 11

415 MW CC PNWE December 2009 0 0 1134 4 Discrete (378)December 2013 0 0 1134 4 Discrete (378)December 2015 0 0 1134 4 Discrete (378)December 2017 0 378 2268 7 Discrete (378)December 2019 0 378 2268 7 Discrete (378)December 2023 0 378 2268 7 Discrete (378)December 2025 0 378 2268 7 Discrete (378) x 1358

85 MW Frame GT December 2009 0 0 162 2 Discrete (162)December 2013 0 0 162 2 Discrete (162)December 2015 0 162 324 3 Discrete (162)December 2017 0 162 648 5 Discrete (162)December 2019 0 162 648 5 Discrete (162)December 2023 0 162 648 5 Discrete (162)December 2025 0 162 648 5 Discrete (162) x 220

Wind December 2009 0 0 1500 6 Discrete (300)December 2013 0 0 3000 11 Discrete (300)December 2015 0 900 3000 11 Discrete (300)December 2017 0 900 4800 17 Discrete (300)December 2019 0 2700 4800 17 Discrete (300)December 2023 0 2700 4800 17 Discrete (300)December 2025 0 2700 4800 17 Discrete (300) x 210085

IGCC wCSS December 2009 0 0 1036 3 Discrete (518)December 2013 0 0 1036 3 Discrete (518)December 2015 0 0 2072 5 Discrete (518)December 2017 0 0 2072 5 Discrete (518)December 2019 0 0 2590 6 Discrete (518)December 2023 0 0 2590 6 Discrete (518)December 2025 0 0 3108 7 Discrete (518) x 987

Geothermal December 2009 0 0 13 2 Discrete (13)December 2013 0 0 26 3 Discrete (13)December 2015 0 0 52 5 Discrete (13)December 2017 0 52 104 9 Discrete (13)December 2019 0 104 156 13 Discrete (13)December 2023 0 156 260 21 Discrete (13)December 2025 0 156 390 31 Discrete (13) x 209641

Woody Biomass December 2009 0 0 850 3 Discrete (425)December 2013 0 0 850 3 Discrete (425)December 2015 0 0 850 3 Discrete (425)December 2017 0 0 850 3 Discrete (425)December 2019 0 0 850 3 Discrete (425)December 2023 0 0 850 3 Discrete (425)December 2025 0 0 850 3 Discrete (425) x 36

Advanced Nuclear December 2009 0 0 5500 6 Discrete (1100)December 2013 0 0 5500 6 Discrete (1100)December 2015 0 1100 5500 6 Discrete (1100)December 2017 0 1100 5500 6 Discrete (1100)December 2019 0 1100 5500 6 Discrete (1100)December 2023 0 1100 5500 6 Discrete (1100)December 2025 0 1100 5500 6 Discrete (1100) x 792

MT WND Phase I December 2009 0 0 750 2 Discrete (750)December 2013 0 750 750 2 Discrete (750)December 2015 0 750 750 2 Discrete (750)December 2017 0 750 750 2 Discrete (750)December 2019 0 750 1500 3 Discrete (750)December 2023 0 750 1500 3 Discrete (750)December 2025 0 750 1500 3 Discrete (750) x 26

MT WND Phase II December 2009 0 0 250 2 Discrete (250)December 2013 0 0 250 2 Discrete (250)December 2015 0 0 500 3 Discrete (250)December 2017 0 0 500 3 Discrete (250)December 2019 0 250 750 4 Discrete (250)December 2023 0 250 750 4 Discrete (250)December 2025 0 250 750 4 Discrete (250) x 88

69 decision variables 1.3E+31

• In the draft Sixth Plan, there were at times nine capacity expansion candidates, not counting conservation and demand response

• Total number of possible plans:1.3 x 1031

• Number of molecules in a mole under standard conditions (Avogadro’s number):6.02 x 1023


26

Candidates for the Final PlanIn the case of the final study for the Sixth Power Plan, there were a mere 6.7 trillion


Source: C:\Backups\Olivia\SAAC 2010\101202 SAAC First Meeting\Presentation materials\States_L813.xls

min start max states step size total states

Cnsrvn_01 0 50 100 11 Discrete (10) 11Cnsrvn_02 0 10 100 11 Discrete (10) x 11DRAC 1 1 2 2 Discrete (1) x 2DRSH 1 1 4 4 Discrete (1) x 4DRAG 1 1 4 4 Discrete (1) x 4DRIN 1 1 4 4 Discrete (1) x 4

cumulative MW

CCCT Dec09 0 0 1134 4 Discrete (378)Dec13 0 0 1134 4 Discrete (378)Dec15 0 0 1134 4 Discrete (378)Dec17 0 378 2268 7 Discrete (378)Dec19 0 756 7560 21 Discrete (378)Dec21 0 756 7560 21 Discrete (378)Dec23 0 756 7560 21 Discrete (378) x 93,331

SCCT Dec09 0 0 162 2 Discrete (162)Dec13 0 0 162 2 Discrete (162)Dec15 0 162 648 5 Discrete (162)Dec17 0 162 648 5 Discrete (162)Dec19 0 162 1620 11 Discrete (162)Dec21 0 162 1620 11 Discrete (162)Dec23 0 162 1620 11 Discrete (162) x 4,613

20 decision variables 6.7E+12

27

Space of feasible solutions

The Set of Plans Precedes the Efficient Frontier

Relian

ce on th

e likeliest ou

tcome

Risk Aversion

Efficient Frontier


28

Finding the “Best” Plan

155600

155800

156000

156200

156400

156600

156800

157000

0 500

1000

1500

2000

2500

3000

3500

4000

4500

5000

5500

6000

6500

7000

7500

8000Ta

ilVar

90 ($

M N

PV)

simulation number

Reduction in TailVar90with increasing

simulations (plans)

C:\Documents and Settings\Michael Schilmoeller\Desktop\NWPCC - Council\SAAC\Presentation materials\Asymptotic reduction in risk with increasing plans.xlsm


29

OptQuest® Recommendations• The RPM used to produce the portfolio for the Council’s

draft Sixth Power Plan has 69 decision variables

• Our finding of 3500 simulations is consistent with OptQuest guidelines (page 156, OptQuest for Crystal Ball User Manual, © 2001, Decisioneering, Inc. )


30

Overview





31

How Many 20-Year Studies?

• How long would this take on the Council’s Aurora2 server?

studiesyear -20 10 2.625

750 3500

futures plans

6

n

Implication to Computational Burden

32

Time on Council’s Server

• Council’s server tech specs:– Xeon W3580 processor– 3.33 MHz, L3 Cache 8– Quad core, 8 Threads per core

• 20-year, hourly study requires 128 minutes• Total time requirement for one study: 2.33

x 105 days (639 years, 3 months, 7 days)


33

Time on a Supercomputer

• October 28, 2010: China acquires the fastest machine on earth: 2.5 petaflops (floating point operations per second)

The Tianhe-1A supercomputer is about 50% faster than its closest rival.


34

On the World’s Fastest Machine

• Assume a benchmark machine can process 20-year studies as fast:– Xeon 5365, 3.0 MHz, L2 Cache 2x4, 4 cores/4

threads per core– 38 GFLOPS on the LinPack standard– To the extent this machine underperforms the Council

server, the time estimate would be longer

• Total time requirement for one study on the Tianhe-1A: 3.54 days (3 days, 12 hours, 51 minutes) and estimated cost $37,318


35

How Do We AchieveOur Objectives?

• If it takes more that a workday to perform the simulation, the risk of making errors begins to dampen exploration

• In the next presentation, we consider alternatives and the RPM solution


36

End

designing a risk model michael schilmoeller thursday, december 2, 2010 saac

Documents

number of futuresc

number of variables

equal number

risk accuracy

risk estimates

monte carlo simulationmc

monte carlo sampleshow

sample size implication