lecture 3: project evaluation...4 4. project evaluation methodologies : an overview • hierarchy of...
TRANSCRIPT
1
Lecture3:ProjectEvaluation:
Dr Micheál Collins
1. Introduction & Purpose
2. Key Questions
3. Thinking about ‘Costs’ (& More)
4. Project Evaluation Methodologies: An Overview
5. When/What to use
6. National Guidelines
7. CBA Explained (in brief)
8. Reading/References for this lecture
1.Introduction&Purpose• ‘From programmes to projects’
• not a simple divide
• capital investments: transport projects, energy…
• current/capital interventions: healthcare, education…
• Goal:tools used to help us make the best possible use of available resources in a rational decision‐making context, when we want to accrue maximum benefit from our scarce resources
• Part of the decision making process, rather than the decision making process
• ‘appraisal’ and ‘evaluation’• before (ex ante) and after (ex post); interchangeable
2
2.KeyQuestions
• No project is adopted unless it has answered these questions:
• Are there better ways to achieve this objective?
• Are there better uses for these resources?
• This implies:
• exploring options/alternatives
• comparing options/alternatives
• justifying a project (even if no alternatives offered)
• is it welfare enhancing (costs < benefits)?
• Key challenge:
• in order to make comparisons between options
• find a common unit of value for everything
• in general: €
• but, return to this…
3.Thinkingabout‘Costs’(&More)
• A major focus
• Different words, phrases and classifications used
• Fixed, variable, semi‐variable, step costs
• Direct, indirect, intangible
• for an illness… to see
• for a transport project… to discuss
3
Anillness:
Direct Costs to the health care system…treatment costs
to the community and family…medication costs
Indirect Costs productivity losses to employers/society
lost income to the individual/family
Intangible Costs costs of pain, suffering, grief
loss of leisure time
loss of life (if appropriate)
Transportproject:
Direct Costs
Indirect Costs
Intangible Costs
• A few other issues at the outset:• Time horizons
• some brief, some many years
• depends on project ’lifetime’
• Placing values on things that have no value• priceless and without a price / without a market
• Difficult questions• especially in health area
• but, scarce resources
• Uncertainty and estimations
4
4.ProjectEvaluationMethodologies:AnOverview
• Hierarchy of evaluation techniques• Ideally, a complete economic evaluation aims to clarify, quantify and value all the relevant options, their inputs and consequences
• Ambitious; the gold‐standard
• Cost‐Benefit Analysis (CBA)
• Others, not as comprehensive
• CMA; CEA; CUA; MCA
• Overview here…see these again in applications and Lecture 11 (week 12)
• All have one principle in common:
• they examine one (or more) possible interventions/options and compare the inputs or resources necessary to carry out such interventions with their consequences and effects
The methods in brief:
Cost‐minimisation Analysis (CMA)
• used when the consequence of the intervention are the same
• therefore, focus only on the inputs
• what is the cheapest way to achieve the desired outcome?
5
Cost‐effectiveness Analysis (CEA)
• consequences of different interventions measured in an identical way
• inputs are costed
• compare the competing interventions in terms of cost per unit of consequence
• e.g. cost per unit increase in reading scores; cost per skilled job created
• e.g. cost per hospital admission avoided; cost per illness avoided
• Key measure is :
incremental cost‐effectiveness ratio (ICER)= change in costs / change in effects
Cost‐utility Analysis (CUA)
• mainly in the healthcare / development area
• used to compare interventions which produce different consequences in terms of quantity and quality of life
• measure the success of each using a common measure of utility (satisfaction / subjective gains)
• compare the outcomes
• e.g. most common utility measures are QALYs
• quality‐adjusted‐life‐years
• therefore, compare cost per QALY
• http://www.euroqol.org/
• similar concept: DALY
Cost‐benefit Analysis (CBA)
• the most comprehensive / difficult
• when both the inputs and the consequences of different interventions are compared in monetary terms
• choose the most desirable
• look at this in more detail shortly
6
Multi‐Criteria Analysis (MCA)
• It establishes preferences between options by reference to an explicit set of objectives and measurable criteria to assess the extent to which these have been met
• do this by:
• choosing a set of performance criteria
• assigning weights to these criteria
• then scoring options in terms of how well they perform
• the weighted scores are then summed
• options are ranked
• Sometimes called weighting & scoring
• It will reappear in Lecture 11 (week 12)
5.When/Whattouse
Evaluation requirements for projects:
• DPER / Public Spending Code guidelines
• projects between €5m and €20m – MCA
• projects >€20m – full CBA or CEA
• smaller projects (less than €5m) – a simple appraisal
• e.g. cashflow analysis; qualitative assessment of merit; assessment of needs, options and costs
• innovative projects / new technologies / involving significant future operational costs – full CBA or CEA
• post‐project reviews on all €20m+ and representative sample of completed €5m+ projects
6.NationalGuidelines
Overall• DPER The Public Spending Code (2011/2012+)
• DPER The Public Spending Code – technical references (2015)
• Department of Finance Guidelines for the Appraisal and Management of Capital Expenditure Proposals in the Public Sector (2005)
• CSF Working Rules for CBA (1999)
Sectoral• Department of Transport Guidelines on a Common Appraisal Framework for Transport Projects and Programmes (2009)
• National Roads Authority Project Appraisal Guidelines (2011) ++
• HIQUA Guidelines for the Economic Evaluation of Health Technologies in Ireland (2014)
• Others to be references in future classes
7
7.CBAExplained(inbrief)
Looking at the key project evaluation method
• Understanding this allows understanding of the others
• Brief overview
• Further notes on the website
• Definition
• Logic and Role
• The Approach
• Some Challenges
Definition:
Cost Benefit Analysis (CBA) provides a systematic set of procedures by which a firm or government can assess whether to undertake a project or programme and, when there is a choice among mutually exclusive projects or programmes, which one to undertake
Logic and Role
• value the benefits to society: economic and social
• value the costs to society: economic and social
• adjust these for time
• combine the costs and benefits and conclude
• if project worthwhile
• which alternative to choose …
8
• Goes beyond qualitative analysis to quantitative analysis
• reduce and control the number of variables faced by a decision maker and thereby enable more rational choices …common monetary scale (€)
• ensure that societies resources are put to their most highly valued uses
• achieve economic efficiency
• evaluate what the market omits: the perspective of society
Overall the general principal of CBA:
“to assess whether or not the social and economic benefits associated with a project are greater than its social and economic costs”
Department of Finance (2005:38) Guidelines for the Appraisal and Management of Capital Expenditure
Proposals in the Public Sector
TheApproach
• Many opinions on how to do a CBA
• Key references are:
• DPER (2011/12) D.03 and DPER (2015) Section E
• Department of Finance (2005)
• UK Treasury Green Book
• European Commission (2008)
• Boardman et al (2006)
• Overview of 8 major steps to follow when carrying out a CBA
9
The Major Steps:
Step 1: Defining objectives and boundaries
Step 2: Identifying alternatives
Step 3: Identifying constraints
Step 4: Estimating costs and benefits
Step 5: Adjusting the values of costs and benefits
Step 6: Calculating the decision criteria
Step 7: Sensitivity analysis
Step 8: Make a Recommendation
Step1:Definingobjectivesandboundaries
• At the outset it is crucial to specify the objectives of a project
• These should be as explicit, precise and amenable to measurement as possible (SMART)
• Why?• allows the possibility to identify alternatives
• facilitates assessment of the costs and benefits
• makes easier to identify for whom benefit is intended
• Examples / Multiple objectives: explicit and ordered
• The boundaries or scope of the project are also important• What to count
• ‘self‐sufficient unit of analysis’ (DPER p8)
• only the costs/benefits from decisions left to be made…no sunk costs
• For how long• ‘the economically useful life of the project’
• Overall: a clear indication of what the project is about and its scope
10
Step2:Identifyingalternatives
• Prepare a list of the range of actions which government could possibly take to achieve the identified objectives
• Normally include some counterfactuals:• “do‐nothing” option
• “do‐minimum” option**
• private sector option
• Big programme: wide range considered, then short‐list chosen for detailed appraisal…how?
• Care in short‐listing: reasons recorded for each exclusion
Step3:Identifyingconstraints
• These take several forms:
• Budgetary constraints
• Environmental constraints
• Production constraints
• Legal constraints
• Policy constraints
• Distributional constraints
• Those relevant must be borne in mind when appraising a programme
Step4:Estimatingcosts& benefits
A few points on step 4:
• Idea is to calc all the costs and benefits, add them together and see which project gives the greatest benefit
• important not to be “spuriously accurate”
• normally look at C&B over lifetime of project
• includes ‘residual costs’ (+ / ‐)
• normally based on market prices…opportunity cost
• wider social and environmental C&B must be included (no mkt price available)
11
Estimating costs and benefits
• Where mkt prices exist (accurately reflect…) then analysis is straightforward
• But often
(i) Market prices are misleading
(ii) Market prices are not available
• When and what to do…
(i) Market prices are misleading
• distorted …poor indicators of social c&b’s
• classic reason: imperfect competition
• higher P and lower Qty than in perfect competition
• others: price supports, subsidies, taxation…
• Solution is:
• shadow prices
• e.g. labour and high unemp = lower shadow price
• but care and requirements on estimates
• willingness to pay estimates
(ii) Market prices are not available
• In some cases there are no market prices
• for: public goods, externalities, intangibles
• Values obtained indirectly – guidelines provide these
• willingness to pay; stated preference; revealed preference
• Some examples:
• the value of time
• the value of life
• the value of environmental externalities
12
The value of time
Time savings = major benefits of transport initiatives
“time is money”
Assumption that: the wage rate provides a measure of an individual’s evaluation of their own time
Labour v leisure
a value is inferred or revealed given an individuals’ life
Basic ideal: if new motorway reduces commuting time by 20 mins and hourly wage rate for a person is €15, then value of saving to that person = €5
Some considerations:
labour market conditions and the value of marginal gains in time…shadow price of labour
small savings of time by many people may amount to a lot but be valued very little
value different depending on when you save the time
working time v leisure time
and where you save the time
leisure v walking v waiting
Department of Transport – 2002 values (updating)
see dropbox for document• Work time = €26.50 per hour
• Non‐work:
• They provide inflators to update values to today’s terms
• Critique of these?
Non-work Walk (x2) Waiting (x2.5)
Commuting €8.10 €16.20 €20.25
Non-Commuting €7.30 €14.60 €18.25
13
The value of life
‘Value of a statistical life’
Most controversial aspect of CBA
distasteful but highly relevant
healthcare projects, transport improvements, road safety campaigns…
3 approaches Foregone earnings
Wage premiums for risk
Contingent valuation
• Irish Values (2002 basis) from Department of Transport document (2004 and 2010)
• Statistical value of a life saved
• plus accidents avoided
• plus injuries avoided
• There are other values used
• health literature and broad ranges
• World Bank…
IrishValues(2002basis):
14
The value of environmental externalities
(i) Pricing Emissions new development in June 2009 (DOF circular)
Updated in 2015 – DPER (document E5)
require inclusion of C02e in CBA and other project appraisals (7 green house gases)
values provided – ‘shadow price of public carbon’ scope: only direct C02 emissions from within state jurisdiction and not already accounted for
• “the resources used to estimate C02e emissions should be proportionate to the scale of the project” (2009)
• prices 2014‐2020 and beyond:
September2015publishedvalues:
15
(ii) Pricing noise pollution
Dept of Finance June 2009:
“Departments/Agencies should consider sector specific
emissions, such as noise, air quality (NOx, SOx, particulatematter), and vibrations in addition to C02e emissions andwhere relevant, possible and significant include thesecosts/benefits in the CBA”
DPER September 2015:
“CBAs should also monetise the value of emissions of other specified non‐GHG emissions (NOX, SO2, PM and noise) where such emissions are considered relevant, significant and practicable for inclusion”
calculation of the impact of noise as a result of a project / development
e.g. new transport infrastructure (airport), industrial development, traffic plans…
measured as increase or decrease in noise levels in average decibels (dB(A))
DoT recommend €28 DB(A) per person per year (2010: 24‐25)
Step5:Adjustingthevaluesofcostsandbenefits
Six adjustments to consider
i. Present values
ii. The social discount rate
iii. Real Prices
iv. Shadow Price of Labour
v. Shadow Price of Public Funds
vi. Distributional issues
16
(i) Present values
costs and benefits occur in different time periods
“time preference”: people prefer to receive goods and services now rather than later
calculate the present value of costs and benefits use the interest rate (i) if the i rate is 5%, the present value of a benefit of €105 next year is €100
You discount the future value by a value between 0‐1, known as the discount rate or discount factor
Formula to calculate it…
Discount factor formula:
i = the interest rate
n = the number of years time the amount accrues
Discount factor =
See UK Green Book table ‐ attached
1
(1 + i ) n
(ii) The social discount rate
Just like people, society likes to receive goods and services now rather than later
tend to use a different discount rate to that used by the private sector
called the social discount rate Why? society takes a longer perspective, not as much rush for returns
market rates contain a risk premium, state is less risky
DPER: 5% in real terms
A case for different (lower) rates for projects of 30yrs+ ‐but these are few (hyperbolic discount rate)
17
(iii) Real prices
Future costs and benefits need to be adjusted for the effect of inflation
compared at ‘constant prices’ or in ‘real terms’
deflate all future values by the expected inflation value
Annual figure from Dept of Finance
“2% per annum over the medium to longer term”
See circular from July 2009 ‐ dropbox
(iv) Shadow Price of Labour
We have a market price for labour
In certain circumstances a case can be made for using a shadow price for labour costs where there is high unemployment there are some resources with zero or low opportunity costs
therefore full price is misleading…
DPRE (2015) suggest a shadow price of 80%‐100% Same as (1999) DOF approach
Care using this: need to justify
but even so, use the market price (100%) too
Not very common – sector applications only; costs only
(v) Shadow Price of Public Funds public and private funds should not be treated the same €1000 of additional private investment as a result of the imposition of €999 of additional tax could be counted as a net gain but in raising this tax – distorts the economy higher taxes might damage competitiveness… P… solution is to weight public costs vary depending on the tax system DPRE (2015) suggest a shadow price of 130% the above analysis: €1000 benefits v €1298.70 costs…a net loss
more or less as per international approach…deadweight loss of taxation
18
(vi) Distributional issues
How are the costs and benefits distributed? if person A gains €200 and person B looses €100: overall society gains but if person B was poor and is now poorer?
possible to take into account distributional issues by attaching weights to the benefits and costs
benefits to disadvantaged groups weighted more than those to the better‐off
but, define these groups; which weights to use? seldom used
alternative = compare inequality pre and post policy
• Overall:
Adjust the costs and benefits to present values by: shadow price use (if appropriate)
adjusting for inflation
and discounting using the social discount rate
Step6:Calculatingthedecisioncriteria
• Once our calculations are complete, how do we interpret the final result?
• 3 decision criteria
(i) net present value (NPV)
(ii) internal rate of return (IRR)
(iii) benefit‐cost ratio
19
(i) net present value (NPV)
Get net present costs and net present benefits costs are subtracted from benefits and the net benefits are expressed at their present value
if NPV is positive then project is accepted if NPV is zero then project is indifferent if NPV is negative then project is rejected projects with positive NPVs enable efficient allocation of resources and represent an improvement to the welfare of society
competing projects: choose one with highest NPV
(ii) internal rate of return (IRR)
the discount rate that will make the NPV of a project equal to zero
a project is worthwhile if the IRR is greater than the discount rate used
an IRR of 15% means that at a discount rate of 15% the project just breaks even; it could earn back all the capital and operating costs and pay 15% for the use of the money
problem: not good for comparing competing projects; does not account for the size of the project
(iii) benefit‐cost ratio
ratio of discounted benefits to discounted costs
a project is accepted if BC ratio > 1
more benefits than costs
problem: misleading if used to rank projects as it ignores the size of the project
20
• Of the three decision criteria, the NPV approach is considered:
• the most reliable method
• and the best method to use
Step7:Sensitivityanalysis
• CBA’s are performed ‘ex‐ante’
• always likely to be some difference between what is expected and what eventually happens
• need to take these uncertainties and risks into account
• key approach = sensitivity analysis
• involves recalculating the NPV at different values for parameters• if i rates were higher
• if wages rates grew by 10% and not 5%
• if revenues were lower than expected
• primary variables that are uncertain:• the discount rate• wage rate growth• forecast revenues• forecast demand• input prices• input quantities• project life span• C02e values
• performing this: pinpoint areas of risk to project• 1‐way; 2‐way; scenario analysis
• ‘best’ and ‘worst’ case
21
Step8:MakeaRecommendation
• Based on the analysis and the sensitivity testing…
Summary: The 8 Major Steps
Step 1: Defining objectives and boundaries
Step 2: Identifying alternatives
Step 3: Identifying constraints
Step 4: Estimating costs and benefits
Step 5: Adjusting the values of costs and benefits
Step 6: Calculating the decision criteria
Step 7: Sensitivity analysis
Step 8: Make a Recommendation
SomeChallenges
• Appropriate short‐listing
• Establishing prices
• shadow prices
• relying on willingness‐to‐pay techniques…subjective
• Over optimism & intentional pessimism
• Avoiding double counting
22
• Unquantifiable costs and benefits it may be the case that it is just not possible to quantify some costs and benefits too difficult; no data available; too much time involved, no suitable shadow prices… cannot ignore these…give misleading indications of best project to pursue solution (1): present results in a form which allows the decision maker to take these into account solution (2): provide a central estimate together with a max and min plausible valuation solution (3): adopt an alternative methodology that works without valuing everything
8.ReadingandReferences
• Documents in the dropbox
• Project Guidelines:• DPER (2011/12) Public Spending Code (Doc D.03)
• DPER (2015) Public Spending Code (Doc E – Technical References)
• Department of Finance Capital Appraisal (2005)
• CSF Working Rules for CBA (1999)
• Department of Transport Guidelines (2010)
• Department of Transport CBA values x2 (2005)
• NRA Guidelines (2011)
• HIQUA Guidelines (2014)
• European Commission (2008)
23
• Key Readings
• Boardman et al – CBA Steps (Handout)
• UK Green Book (2003 updated 2011)
• DPER (2011/12 & 2015) Public Spending Code
THE GREEN BOOK100
Annex 6: Discount Rate
01
23
45
67
89
1011
12
1314
1516
1718
1920
2122
2324
2526
2728
29
30
Year
1.00000.9901
0.98030.9706
0.96100.9515
0.94200.9327
0.92350.9143
0.90530.8963
0.8874
0.87870.8700
0.86130.8528
0.84440.8360
0.82770.8195
0.81140.8034
0.79540.7876
0.77980.7720
0.76440.7568
0.7493
0.7419
1.0%
1.00000.9804
0.96120.9423
0.92380.9057
0.88800.8706
0.85350.8368
0.82030.8043
0.7885
0.77300.7579
0.74300.7284
0.71420.7002
0.68640.6730
0.65980.6468
0.63420.6217
0.60950.5976
0.58590.5744
0.5631
0.5521
2.0%
1.00000.9709
0.94260.9151
0.88850.8626
0.83750.8131
0.78940.7664
0.74410.7224
0.7014
0.68100.6611
0.64190.6232
0.60500.5874
0.57030.5537
0.53750.5219
0.50670.4919
0.47760.4637
0.45020.4371
0.4243
0.4120
3.0%
1.00000.9662
0.93350.9019
0.87140.8420
0.81350.7860
0.75940.7337
0.70890.6849
0.6618
0.63940.6178
0.59690.5767
0.55720.5384
0.52020.5026
0.48560.4692
0.45330.4380
0.42310.4088
0.39500.3817
0.3687
0.3563
3.5%
1.00000.9615
0.92460.8890
0.85480.8219
0.79030.7599
0.73070.7026
0.67560.6496
0.6246
0.60060.5775
0.55530.5339
0.51340.4936
0.47460.4564
0.43880.4220
0.40570.3901
0.37510.3607
0.34680.3335
0.3207
0.3083
4.0% 5.0%
1.00000.9524
0.90700.8638
0.82270.7835
0.74620.7107
0.67680.6446
0.61390.5847
0.5568
0.53030.5051
0.48100.4581
0.43630.4155
0.39570.3769
0.35890.3418
0.32560.3101
0.29530.2812
0.26780.2551
0.2429
0.2314
6.0%
1.00000.9434
0.89000.8396
0.79210.7473
0.70500.6651
0.62740.5919
0.55840.5268
0.4970
0.46880.4423
0.41730.3936
0.37140.3503
0.33050.3118
0.29420.2775
0.26180.2470
0.23300.2198
0.20740.1956
0.1846
0.1741
7.0%
1.00000.9346
0.87340.8163
0.76290.7130
0.66630.6227
0.58200.5439
0.50830.4751
0.4440
0.41500.3878
0.36240.3387
0.31660.2959
0.27650.2584
0.24150.2257
0.21090.1971
0.18420.1722
0.16090.1504
0.1406
0.1314
8.0%
1.00000.9259
0.85730.7938
0.73500.6806
0.63020.5835
0.54030.5002
0.46320.4289
0.3971
0.36770.3405
0.31520.2919
0.27030.2502
0.23170.2145
0.19870.1839
0.17030.1577
0.14600.1352
0.12520.1159
0.1073
0.0994
9.0%
1.00000.9174
0.84170.7722
0.70840.6499
0.59630.5470
0.50190.4604
0.42240.3875
0.3555
0.32620.2992
0.27450.2519
0.23110.2120
0.19450.1784
0.16370.1502
0.13780.1264
0.11600.1064
0.09760.0895
0.0822
0.0754
10.0%
1.00000.9091
0.82640.7513
0.68300.6209
0.56450.5132
0.46650.4241
0.38550.3505
0.3186
0.28970.2633
0.23940.2176
0.19780.1799
0.16350.1486
0.13510.1228
0.11170.1015
0.09230.0839
0.07630.0693
0.0630
0.0573
Discount rates
0.4533
0.4380
0.4231
0.40880.3950
0.3817
0.36870.3563
0.26510.1973
0.14680.0942
0.0833
0.0651
0.05080.0274
0.01670.0062
0.00290.0014
0.0009
0.00050.0002
Long TermDiscount Factor
0
1
2
34
5
67
89
1011
12
13
1415
1617
1819
20
2122
Year
1.0000
0.9662
0.9335
0.90190.8714
0.8420
0.81350.7860
0.75940.7337
0.70890.6849
0.6618
0.6394
0.61780.5969
0.57670.5572
0.53840.5202
0.5026
0.48560.4692
Long TermDiscount Factor
23
24
25
2627
28
2930
4050
6075
80
90
100125
150200
250300
350
400500
Year
DISCOUNT FACTORS
LONG TERM DISCOUNT FACTORS