designing a risk model michael schilmoeller thursday, december 2, 2010 saac
Post on 15-Jan-2016
213 Views
Preview:
TRANSCRIPT
Designing a Risk Model
Michael SchilmoellerThursday, December 2, 2010
SAAC
2
Overview
• Scope of uncertainty• Decision trees (briefly) and Monte Carlo
simulation• Implications of cost and risk accuracy to
the number of futures• The number of possible plans and finding
the “best” plan• Computational alternatives
3
Sources of Uncertainty
Scope of uncertainty
• Fifth Power Plan– Load requirements– Gas price– Hydrogeneration– Electricity price– Forced outage rates– Aluminum price– Carbon allowance cost– Production tax credits– Renewable Energy Credit
(Green tag value)
• Sixth Power Plan– aluminum price and
aluminum smelter loads were removed
– Power plant construction costs
– Technology availability– Conservation costs and
performance
4
Impact on NPV Costs and Risk
0
10
20
30
40
50
60
70
80
9030
40
50
60
70
80
90
100
110
120
130
140
150
160
170
180
190
200
210
220
230
Freq
uenc
y
Billions of 2006 Constant Dollars
NPV 20-Year Study Costs
Scope of uncertainty
C:\Documents and Settings\Michael Schilmoeller\Desktop\NWPCC - Council\SAAC\Presentation materials\L813 NPV Costs.xlsm
5
Decision Trees
• Estimating the number of branches– Assume possible 3 values (high, medium, low) for each of 9
variables, 80 periods, with two subperiods each; plus 70 possible hydro years, one for each of 20 years, on- and off-peak energy determined by hydro year
– Number of estimates cases, assuming independence: 6,048,000
• Studies, given equal number k of possible values for n uncertainties:
• Impact of adding an uncertainty:
Decision trees & Monte Carlo simulation
iesuncertaint values, , nkkN n
kN
N
1
6
Monte Carlo Simulation
• MC represents the more likely values• The number of samples is determined by the
accuracy requirement for the statistics of interest• The number of samples mk necessary to obtain
a given level of precision in estimates of averages grows much more slowly than the number of variables k:
Decision trees & Monte Carlo simulation
k
k
m
m
k
k 11
7
Overview
• Scope of uncertainty• Decision trees (briefly) and Monte Carlo
simulation• Implications of cost and risk accuracy to
the number of futures• The number of possible plans and finding
the “best” plan• Computational alternatives
8
Monte Carlo Samples
• How many samples are necessary to achieve reasonable cost and risk estimates?
• How precise is the sample mean of the tail, that is, TailVaR90?
Implication to Number of Futures
9
Relationship Between the Size of the Sample and the Accuracy
• Depends on knowledge of the distribution• Given the distribution, requires knowledge
of how the accuracy depends on sample size
Implication to Number of Futures
10
Central Limit Theorem
• Both our cost and our risk estimates are averages
• CLT says that as the number of samples used to estimate the mean increases, the distribution of the sample means tends to normal
• Unfortunately, it doesn’t say how fast it tends to normal or how the shape of the underlying distribution affects the rate of approach
Implication to Number of Futures
11
TailVaR90
0123456789
10111213141516
109
115
121
127
133
139
145
151
157
163
169
175
181
187
193
199
205
211
217
223
Freq
uenc
y
Billions of 2006 Constant Dollars
Tail Risk
Implication to Number of Futures
C:\Documents and Settings\Michael Schilmoeller\Desktop\NWPCC - Council\SAAC\Presentation materials\L813 NPV Costs 02.xlsm
12
Assumed Distribution
0123456789
10111213141516
109
115
121
127
133
139
145
151
157
163
169
175
181
187
193
199
205
211
217
223
Freq
uenc
y
Billions of 2006 Constant Dollars
Tail Risk
Implication to Number of Futures
C:\Documents and Settings\Michael Schilmoeller\Desktop\NWPCC - Council\SAAC\Presentation materials\L813 NPV Costs 02.xlsm
13
Set Up a Sampler
0123456789
10111213141516
109
115
121
127
133
139
145
151
157
163
169
175
181
187
193
199
205
211
217
223
Freq
uenc
y
Billions of 2006 Constant Dollars
Tail Risk
Implication to Number of Futures
C:\Documents and Settings\Michael Schilmoeller\Desktop\NWPCC - Council\SAAC\Presentation materials\L813 NPV Costs 02.xlsm
14
Dependence of Tail Average on Sample Size
0
10
20
30
40
50
60
70
80
109
111
113
115
117
119
121
123
125
127
129
131
133
135
137
139
141
143
145
147
5 samples per average0
10
20
30
40
50
60
70
80
90
30
40
50
60
70
80
90
100
110
120
130
140
150
160
170
180
190
200
210
220
230
Freq
uenc
y
Billions of 2006 Constant Dollars
NPV 20-Year Study Costs
Implication to Number of Futures
C:\Documents and Settings\Michael Schilmoeller\Desktop\NWPCC - Council\SAAC\Presentation materials\L813 NPV Costs 02.xlsm, worksheet “Simulation”
15
0
20
40
60
80
100
120
109
111
113
115
117
119
121
123
125
127
129
131
133
135
137
139
141
143
145
147
10 samples per average
Implication to Number of Futures
Dependence of Tail Average on Sample Size
C:\Documents and Settings\Michael Schilmoeller\Desktop\NWPCC - Council\SAAC\Presentation materials\L813 NPV Costs 02.xlsm, worksheet “Simulation”
0
10
20
30
40
50
60
70
80
90
30
40
50
60
70
80
90
100
110
120
130
140
150
160
170
180
190
200
210
220
230
Freq
uenc
y
Billions of 2006 Constant Dollars
NPV 20-Year Study Costs
16
0
204060
80100120
140160180
109
111
113
115
117
119
121
123
125
127
129
131
133
135
137
139
141
143
145
147
25 samples per average
Implication to Number of Futures
Dependence of Tail Average on Sample Size
C:\Documents and Settings\Michael Schilmoeller\Desktop\NWPCC - Council\SAAC\Presentation materials\L813 NPV Costs 02.xlsm, worksheet “Simulation”
0
10
20
30
40
50
60
70
80
90
30
40
50
60
70
80
90
100
110
120
130
140
150
160
170
180
190
200
210
220
230
Freq
uenc
y
Billions of 2006 Constant Dollars
NPV 20-Year Study Costs
17Implication to Number of Futures
Dependence of Tail Average on Sample Size
0
10
20
30
40
50
60
70
11
6
11
6.7
5
11
7.5
11
8.2
5
11
9
11
9.7
5
12
0.5
12
1.2
5
12
2
12
2.7
5
12
3.5
12
4.2
5
12
5
12
5.7
5
12
6.5
12
7.2
5
12
8
12
8.7
5
12
9.5
13
0.2
5
13
1
13
1.7
5
50 samples per average
σ=2.040
C:\Documents and Settings\Michael Schilmoeller\Desktop\NWPCC - Council\SAAC\Presentation materials\L813 NPV Costs 02.xlsm, worksheet “Samples_50”
0
10
20
30
40
50
60
70
80
90
30
40
50
60
70
80
90
100
110
120
130
140
150
160
170
180
190
200
210
220
230
Freq
uenc
y
Billions of 2006 Constant Dollars
NPV 20-Year Study Costs
18Implication to Number of Futures
Dependence of Tail Average on Sample Size
0
10
20
30
40
50
60
70
11
6
11
6.7
5
11
7.5
11
8.2
5
11
9
11
9.7
5
12
0.5
12
1.2
5
12
2
12
2.7
5
12
3.5
12
4.2
5
12
5
12
5.7
5
12
6.5
12
7.2
5
12
8
12
8.7
5
12
9.5
13
0.2
5
13
1
13
1.7
5
75 samples per average
C:\Documents and Settings\Michael Schilmoeller\Desktop\NWPCC - Council\SAAC\Presentation materials\L813 NPV Costs 02.xlsm, worksheet “Samples_75”
σ=1.677
0
10
20
30
40
50
60
70
80
90
30
40
50
60
70
80
90
100
110
120
130
140
150
160
170
180
190
200
210
220
230
Freq
uenc
y
Billions of 2006 Constant Dollars
NPV 20-Year Study Costs
19
Chi-Squared (X2) Tests
• Check the hypothesis that our sample has the variation from normal by chance (p)
• 50 samples per calculation: p=0.50• 75 samples per calculation: p=0.10
Implication to Number of Futures
20
Accuracy and Sample Size• Estimated accuracy of TailVaR90 statistic is
still only ± $3.3 B (2σ)!*
0
10
20
30
40
50
60
70
80
90
30
40
50
60
70
80
90
100
110
120
130
140
150
160
170
180
190
200
210
220
230
Freq
uenc
y
Billions of 2006 Constant Dollars
NPV 20-Year Study Costs
Implication to Number of Futures
0
10
20
30
40
50
60
70
116
116.
7511
7.5
118.
25 119
119.
7512
0.5
121.
25 122
122.
7512
3.5
124.
25 125
125.
7512
6.5
127.
25 128
128.
7512
9.5
130.
25 131
131.
75
75 samples per average
*Stay tuned to see why the precision is actually 1000x better than this!
21
Accuracy Relative to the Efficient Frontier
123200
124200
125200
126200
127200
128200
129200
77000 78000 79000 80000 81000 82000 83000
Ris
k (N
PV
$2
00
6 M
)
Cost (NPV $2006 M)
L813
L813 L813 Frontier
C:\Backups\Plan 6\Studies\L813\Analysis of Optimization Run_L813vL811.xls
Implication to Number of Futures
22
Conclusion
• At least 75 samples are needed for determining the value of our risk metric– Known distribution of statistic– The precision of the sample
• Our risk metric is 1/10 of the total number of futures
• We need to test our plan under 750 futures to obtain defensible results
Implication to Number of Futures
23
Overview
• Scope of uncertainty• Decision trees (briefly) and Monte Carlo
simulation• Implications of cost and risk accuracy to
the number of futures• The number of possible plans and finding
the “best” plan• Computational alternatives
24
Finding the Best Plan
• Each plan is exposed to exactly the same set of futures, except for electricity price
• Look for the plan that minimizes cost and risk
• Challenge: there may be many plans
Implication to Number of Plans
25
Avogadro’s Number
min start max states step size
DRAC 1 1 2 2 Discrete (1) 2DRSH 1 1 4 4 Discrete (1) x 4DRAG 1 1 4 4 Discrete (1) x 4DRIN 1 1 4 4 Discrete (1) x 4Lost Opp Cnsvsn 0 50 100 11 Discrete (10) x 11Discretionary Cnsvsn 0 10 100 11 Discrete (10) x 11
415 MW CC PNWE December 2009 0 0 1134 4 Discrete (378)December 2013 0 0 1134 4 Discrete (378)December 2015 0 0 1134 4 Discrete (378)December 2017 0 378 2268 7 Discrete (378)December 2019 0 378 2268 7 Discrete (378)December 2023 0 378 2268 7 Discrete (378)December 2025 0 378 2268 7 Discrete (378) x 1358
85 MW Frame GT December 2009 0 0 162 2 Discrete (162)December 2013 0 0 162 2 Discrete (162)December 2015 0 162 324 3 Discrete (162)December 2017 0 162 648 5 Discrete (162)December 2019 0 162 648 5 Discrete (162)December 2023 0 162 648 5 Discrete (162)December 2025 0 162 648 5 Discrete (162) x 220
Wind December 2009 0 0 1500 6 Discrete (300)December 2013 0 0 3000 11 Discrete (300)December 2015 0 900 3000 11 Discrete (300)December 2017 0 900 4800 17 Discrete (300)December 2019 0 2700 4800 17 Discrete (300)December 2023 0 2700 4800 17 Discrete (300)December 2025 0 2700 4800 17 Discrete (300) x 210085
IGCC wCSS December 2009 0 0 1036 3 Discrete (518)December 2013 0 0 1036 3 Discrete (518)December 2015 0 0 2072 5 Discrete (518)December 2017 0 0 2072 5 Discrete (518)December 2019 0 0 2590 6 Discrete (518)December 2023 0 0 2590 6 Discrete (518)December 2025 0 0 3108 7 Discrete (518) x 987
Geothermal December 2009 0 0 13 2 Discrete (13)December 2013 0 0 26 3 Discrete (13)December 2015 0 0 52 5 Discrete (13)December 2017 0 52 104 9 Discrete (13)December 2019 0 104 156 13 Discrete (13)December 2023 0 156 260 21 Discrete (13)December 2025 0 156 390 31 Discrete (13) x 209641
Woody Biomass December 2009 0 0 850 3 Discrete (425)December 2013 0 0 850 3 Discrete (425)December 2015 0 0 850 3 Discrete (425)December 2017 0 0 850 3 Discrete (425)December 2019 0 0 850 3 Discrete (425)December 2023 0 0 850 3 Discrete (425)December 2025 0 0 850 3 Discrete (425) x 36
Advanced Nuclear December 2009 0 0 5500 6 Discrete (1100)December 2013 0 0 5500 6 Discrete (1100)December 2015 0 1100 5500 6 Discrete (1100)December 2017 0 1100 5500 6 Discrete (1100)December 2019 0 1100 5500 6 Discrete (1100)December 2023 0 1100 5500 6 Discrete (1100)December 2025 0 1100 5500 6 Discrete (1100) x 792
MT WND Phase I December 2009 0 0 750 2 Discrete (750)December 2013 0 750 750 2 Discrete (750)December 2015 0 750 750 2 Discrete (750)December 2017 0 750 750 2 Discrete (750)December 2019 0 750 1500 3 Discrete (750)December 2023 0 750 1500 3 Discrete (750)December 2025 0 750 1500 3 Discrete (750) x 26
MT WND Phase II December 2009 0 0 250 2 Discrete (250)December 2013 0 0 250 2 Discrete (250)December 2015 0 0 500 3 Discrete (250)December 2017 0 0 500 3 Discrete (250)December 2019 0 250 750 4 Discrete (250)December 2023 0 250 750 4 Discrete (250)December 2025 0 250 750 4 Discrete (250) x 88
69 decision variables 1.3E+31
• In the draft Sixth Plan, there were at times nine capacity expansion candidates, not counting conservation and demand response
• Total number of possible plans:1.3 x 1031
• Number of molecules in a mole under standard conditions (Avogadro’s number):6.02 x 1023
Implication to Number of Plans
26
Candidates for the Final PlanIn the case of the final study for the Sixth Power Plan, there were a mere 6.7 trillion
Implication to Number of Plans
Source: C:\Backups\Olivia\SAAC 2010\101202 SAAC First Meeting\Presentation materials\States_L813.xls
min start max states step size total states
Cnsrvn_01 0 50 100 11 Discrete (10) 11Cnsrvn_02 0 10 100 11 Discrete (10) x 11DRAC 1 1 2 2 Discrete (1) x 2DRSH 1 1 4 4 Discrete (1) x 4DRAG 1 1 4 4 Discrete (1) x 4DRIN 1 1 4 4 Discrete (1) x 4
cumulative MW
CCCT Dec09 0 0 1134 4 Discrete (378)Dec13 0 0 1134 4 Discrete (378)Dec15 0 0 1134 4 Discrete (378)Dec17 0 378 2268 7 Discrete (378)Dec19 0 756 7560 21 Discrete (378)Dec21 0 756 7560 21 Discrete (378)Dec23 0 756 7560 21 Discrete (378) x 93,331
SCCT Dec09 0 0 162 2 Discrete (162)Dec13 0 0 162 2 Discrete (162)Dec15 0 162 648 5 Discrete (162)Dec17 0 162 648 5 Discrete (162)Dec19 0 162 1620 11 Discrete (162)Dec21 0 162 1620 11 Discrete (162)Dec23 0 162 1620 11 Discrete (162) x 4,613
20 decision variables 6.7E+12
27
Space of feasible solutions
The Set of Plans Precedes the Efficient Frontier
Relian
ce on th
e likeliest ou
tcome
Risk Aversion
Efficient Frontier
Implication to Number of Plans
28
Finding the “Best” Plan
155600
155800
156000
156200
156400
156600
156800
157000
0 500
1000
1500
2000
2500
3000
3500
4000
4500
5000
5500
6000
6500
7000
7500
8000Ta
ilVar
90 ($
M N
PV)
simulation number
Reduction in TailVar90with increasing
simulations (plans)
C:\Documents and Settings\Michael Schilmoeller\Desktop\NWPCC - Council\SAAC\Presentation materials\Asymptotic reduction in risk with increasing plans.xlsm
Implication to Number of Plans
29
OptQuest® Recommendations• The RPM used to produce the portfolio for the Council’s
draft Sixth Power Plan has 69 decision variables
• Our finding of 3500 simulations is consistent with OptQuest guidelines (page 156, OptQuest for Crystal Ball User Manual, © 2001, Decisioneering, Inc. )
Implication to Number of Plans
30
Overview
• Scope of uncertainty• Decision trees (briefly) and Monte Carlo
simulation• Implications of cost and risk accuracy to
the number of futures• The number of possible plans and finding
the “best” plan• Computational alternatives
31
How Many 20-Year Studies?
• How long would this take on the Council’s Aurora2 server?
studiesyear -20 10 2.625
750 3500
futures plans
6
n
Implication to Computational Burden
32
Time on Council’s Server
• Council’s server tech specs:– Xeon W3580 processor– 3.33 MHz, L3 Cache 8– Quad core, 8 Threads per core
• 20-year, hourly study requires 128 minutes• Total time requirement for one study: 2.33
x 105 days (639 years, 3 months, 7 days)
Implication to Computational Burden
33
Time on a Supercomputer
• October 28, 2010: China acquires the fastest machine on earth: 2.5 petaflops (floating point operations per second)
The Tianhe-1A supercomputer is about 50% faster than its closest rival.
Implication to Computational Burden
34
On the World’s Fastest Machine
• Assume a benchmark machine can process 20-year studies as fast:– Xeon 5365, 3.0 MHz, L2 Cache 2x4, 4 cores/4
threads per core– 38 GFLOPS on the LinPack standard– To the extent this machine underperforms the Council
server, the time estimate would be longer
• Total time requirement for one study on the Tianhe-1A: 3.54 days (3 days, 12 hours, 51 minutes) and estimated cost $37,318
Implication to Computational Burden
35
How Do We AchieveOur Objectives?
• If it takes more that a workday to perform the simulation, the risk of making errors begins to dampen exploration
• In the next presentation, we consider alternatives and the RPM solution
Implication to Computational Burden
36
End
top related