socio-economic analyses in perspective: uncertainties and ... · pdf filesocio-economic...
TRANSCRIPT
Socio-economic analyses in perspective: Uncertainties and bias in decision support
Associate Professor, PhD Kim Bang Salling DTU Transport Traffic days in Aalborg 2012 – Special session: “Uncertainties in Transport Project Evaluation (UNITE)”
DTU Transport, Technical University of Denmark 2
Project Plan of UNITE
Uncertainties in Transport Project Evaluation (UNITE): the five Work-Packages
(5) Evaluation methodologyWP5 project leader: Steen Leleur (DMG)
(4) Uncertainty calculation in transport modelsWP4 project leader: Otto Anker Nielsen (TMG)
(2) Organizational context of Modelling, an empirical study
WP2 project leader: Petter Næss (AAU)
(3) Uncertainty calculation of cost estimates
WP3 project leader: Bo Friis Nielsen (DTU Informatics)
(1) Systematic biases in transport models (recognized ignorance), an empirical studyWP1 project leader: Petter Næss (AAU)
DTU Transport, Technical University of Denmark 3
How do we evaluate transport projects?
• Various existing guideline report: –Denmark, Sweden, UK, European Union, ....
• Socio-economic analysis by the use of Cost-Benefit
Analysis (CBA)
• Produces single point estimates such as Net Present Values (NPV), Benefit Cost Ratios (BCR), etc
• However, no common rule have been set in order to acommodate the uncertainties in CBA!
–Recent conducted PhD dissertation proved this point
DTU Transport, Technical University of Denmark 4
Background & Motivation
• The Manual for socio-economic analysis in the transport sector (2003)
–Unique guidelines for evaluating transport infrastructure projects
–Lack of uncertainty handling –Expected revision 2012-2013
DTU Transport, Technical University of Denmark 5
How do we evaluate transport projects?
• However, no common rule have been set in order to acommodate the uncertainties in CBA!
–Recent conducted PhD dissertation proved this point
DTU Transport, Technical University of Denmark 6
The Case Study: HH-Connection • Connecting Denmark with Sweden: Scandinavian link
–Currently, close to the capacity limit on Oresund
HH-Connection (alternatives*)
Description (Alignment of connection)
Cost (million DKK)
Alternative 1 Tunnel for rail (2 tracks) person traffic only 7,700
Alternative 2 Tunnel for rail (1 track) goods traffic only 5,500
Alternative 3 Bridge for road and rail (2x2 lanes & 2 tracks) 11,500
Alternative 4 Bridge for road (2x2 lanes) 6,000
* Larsen, L.A. & Skougaard, B.Z. (2010). Vurdering af alternativer for en fast forbindelse Helsingør-Helsingborg, M.Sc. thesis, Department of Transport, Technical University of Denmark (in Danish)
DTU Transport, Technical University of Denmark 7
The UNITE-DSS Modelling Framework Todays Outline
DTU Transport, Technical University of Denmark 8
Results: Cost-Benefit Analysis
• Construction costs – by far the largest contributor of costs
• User Benefits – by far the largest contributor of benefits – Consists of Ticket revenue and time savings – Relies on the prognosis of future number of passengers i.e.
demand forecasts
HH-Connection (alternatives)
Cost (million DKK)
BCR NPV (million DKK)
Alternative 1 7,700 1.50 5,530
Alternative 2 5,500 0.16 -6,640
Alternative 3 11,500 2.71 28,240
Alternative 4 6,000 3.08 17,860
DTU Transport, Technical University of Denmark 9
Are we telling the truth?!?!
Construction cost overruns
0%
200%
400%
600%
800%
1000%
1200%
1400%
1600%
1800%
2000%
Suez
Can
al
Sydn
eyO
pera
Hou
se
Con
cord
eSu
pers
onic
Aero
plan
eBo
ston
'sAr
tery
/Tun
nel
Proj
ect,
USA
Hum
ber
Brid
ge, U
K
Bost
on-
Was
hing
ton-
New
Yor
kG
reat
Bel
tR
ail T
unne
l,D
KA6
Mot
orw
ayC
hape
l-en-
le-
Frith
/Wha
ley
Shin
kans
enJo
etsu
Rai
llin
e, J
apan
Was
hing
ton
met
ro, U
SA
Cha
nnel
Tunn
el, U
K &
Fran
ceKa
rlsru
he-
Bret
ten
light
rail,
Ger
man
yØ
resu
ndAc
cess
link
s,D
K &
Swed
enM
exic
o ci
tym
etro
line
,M
exic
oPa
ris-A
uber
-N
ante
rre
rail
line,
Fra
nce
Cos
t Ove
rrun
s (%
)
Q: Have we learned anything from history?
”Chunnel” in 1987 £2,600 million (’85 prices) Completion 1994 £4,650 million (’85 prices) Total cost overrun of approx. 80%
”Øresund access link” in 1991 3.2 billion DKK (’90 prices) Completion 1998 5.4 billion DKK (’90 prices) Total cost overrun of approx. 68%
DTU Transport, Technical University of Denmark 10
Theoretical anchoring The Transport Planning Phase: Adapted from the British Department for Transport (DfT) (2004)
Reference Class Forecasting: Optimism Bias
Inside View Outside View
”Uniqueness” of Project
”The Planning Fallacy”
Reference Class Forecasting
Forecasting of particular projects
Forecasting from a group of projects
(1) Identification of relevant reference
classes
(2) Establishing probability distribution
(3) Placing and comparing the
project
Optimism Bias UpliftsCurrent Situation
DTU Transport, Technical University of Denmark 11
Optimism Bias and uplifts
• Deriving uplifts is highly dependet on large data-sets –Flyvbjerg from (AAU) has since 2003 developed a large
database –Unfortunately, it looks upon mega-projects
• The basis is Reference Class Forecasting i.e. statistical measurements on various project pools
Source: Flyvbjerg and COWI (2004)
DTU Transport, Technical University of Denmark 12
Results : Optimism Bias Uplifts
• The BCR are lower, however, still point estimates towards DM –Moreover an advanced form of sensitivity analysis
• Imply to introduce risk analysis and Monte Carlo simulation
HH-Connection (alternatives)
Cost (uplifted) (million DKK)
BCR (orig.) (from slide 8)
BCR (uplifts): 80% uplift
Alternative 1 12,090 1.50 0.97
Alternative 2 8,640 0.16 0.10
Alternative 3 15,180 2.71 1.75
Alternative 4 7,920 3.08 1.98
DTU Transport, Technical University of Denmark 13
The UNITE Project Database (UPD)
• The convention used is as follows: ( )( )
forecasted
forecastedactual
XXX
U100×−
=
Over estimation of Demand
DTU Transport, Technical University of Denmark 14
• Demand forecasts (user benefits) are derived: – U is percent inaccuracy, – Xa is the actual traffic after the project is opened – Xf is the forecasted traffic on the decision to build
• Combination of two database samples
0
5
10
15
20
25
30
(-12
0;-1
00)
(-10
0;-8
0)
(-80
;-60)
(-60
;-40)
(-40
;-20)
(-20
;0)
(0;2
0)
(20;
40)
(40;
60)
(60;
80)
(80;
100)
(100
;120
)
(120
;140
)
(140
;160
)
(160
;180
)
(180
;200
)
(200
;220
)
(220
;240
)
Freq
uenc
y of
occ
uren
ce (%
)
Inaccuracies in demand forecasts (%)
Inaccuracies in demand forecasts (road projects)
Salling et al. (2012)
Flyvbjerg et al. (2003)
Nicolaisen et al. (2012)
DTU Transport, Technical University of Denmark 15
The UNITE Project Database (UPD)
• The convention used is as follows: ( )( )
forecasted
forecastedactual
XXX
U100×−
=
Under estimation of costs
DTU Transport, Technical University of Denmark 16
• Construction costs bias derived similarly: – U is percent inaccuracy, – Xa is the actual traffic after the project is opened – Xf is the forecasted traffic on the decision to build
• Combination of two database samples
0
10
20
30
40
50
(-10
0;-8
0)
(-80
;-60)
(-60
;-40)
(-40
;-20)
(-20
;0)
(0;2
0)
(20;
40)
(40;
60)
(60;
80)
(80;
100)
(100
;120
)
(120
;140
)
(140
;160
)
(160
;180
)
(180
;200
)
(200
;220
)
(220
;240
)
Freq
uenc
y of
occ
uren
ce (%
)
Inaccuracies in construction costs (%)
Inaccuracies in construction cost (road projects)
Salling et al (2012)
Flyvbjerg et al. (2003)
Nicolaisen et al. (2012)
DTU Transport, Technical University of Denmark 17
Results (RCF): Monte Carlo simulation
DTU Transport, Technical University of Denmark 18
Conclusions
• Feasibility risk assessment can be carried out by using historical experience stemming from RCF in order to obtain interval results
• An important aspect in RCF and UNITE is to set and validate input parameters. Hence, empirical data enter the assessment.
• The RCF approach has been illustrated on a case example concerning the construction of a new fixed link, the HH-Connection, between Denmark and Sweden.
• Clearly vital to include uncertainties within socio-economic analyses in order to validate results
DTU Transport, Technical University of Denmark 19
Perspectives
• Recovering of further data (UPD) with regard to both the demand forecast uncertainty as well as the construction costs through large-scale research study
• Producing so-called decision conferences in order to achieve better input parameters to the UNITE-DSS Model combined with overconfidence theory allows for expert opinions (SIMSIGHT)
• More info on UNITE can be found: (www.transport.dtu.dk/unite)
• An international conference on the topic is scheduled in September 2013 – a specific call will be posted in the upcoming month.
DTU Transport, Technical University of Denmark
SIMSIGHT: Decision Conferencing (DC)
20
• Producing so-called decision conferences in order to achieve better input parameters to the UNITE-DSS Model
• Enables to include Stakeholders and Decision-makers in an early stage, i.e. to include experts opinion on MIN and MAX values as entries to the Monte Carlo simulation
DTU Transport, Technical University of Denmark 21
Results from DC and RSF
DTU Transport, Technical University of Denmark
SIMSIGHT: Overconfidence
22
DTU Transport, Technical University of Denmark 23
Perspectives
• Recovering of further data (UPD) with regard to both the demand forecast uncertainty as well as the construction costs through large-scale research study
• Producing so-called decision conferences in order to achieve better input parameters to the UNITE-DSS Model combined with overconfidence theory allows for expert opinions (SIMSIGHT)
• More info on UNITE can be found: (www.transport.dtu.dk/unite)
• An international conference on the topic is scheduled in September 2013 – a specific call will be posted in the upcoming month.
DTU Transport, Technical University of Denmark 24
Thank you for your attention!
Affiliation:
Associate Professor, PhD Kim Bang Salling
Department of Transport Technical University of Denmark