realising benefits from government ict investment – a fool's errand?
TRANSCRIPT
Realising Benefits from Government ICT investment –a fool’s errand?
Virtual Government Seminar, Tokyo 12th March 2010
Stephen Jenner [[email protected]]
Change programmes - the rationale
“The fundamental reason for beginning a programme is to realise benefits through change. The change may be to do things differently, to do different things, or to do things that will influence others to change.”
Office of Government Commerce (OGC)
“It is only possible to be sure that change has worked if we can measure the delivery of benefits it is supposed to bring.”
Cabinet Office
But the track record is not good in the UK
“Implementation of IT systems has resulted in delay, confusion and inconvenience to the citizen and, in many cases poor value for money to the taxpayer.”
Public Accounts Committee
“Deficiencies in benefits capture bedevils nearly 50% of government projects” and, “30-40% of systems to support business change deliver no benefits whatsoever.”
Office of Government Commerce
Or in Europe…
“After at least a decade of large investments (running into billions of Euro) at digitalizing the public sector, governments in Europe are still mostly unable to objectively quantify and show the benefits and returns of such investments.”
e GEP, 2006
“The Implementation of eGovernment solutions is expected to bring a number of advantages including efficiency, increased user satisfaction, and reduction of administrative burden. However, this cannot be taken for granted any longer.”
eGovMoNet, 2008
Or in Australia…
“Data from Australia on achieved benefit/cost ratios [in] central government indicates that rates of return on ICT investments often are low or negative.”
OECD
“benefits realisation and the measurement of benefits arising from investments in ICT are areas where there is substantial scope for improvement”
Sir Peter Gershon
Or indeed in the US…
“The United States federal government will spend over USD 60 billion on ICT technology and systems in 2006. This huge expenditure has led many to ask if ICT investment is providing more than USD 60 billion in value to government agencies and, ultimately, tax payers. The answer is increasingly that nobody knows.”
OECD, 2006
“the committee has no confidence that the amounts being assessed have any relationship to the benefits anticipated.”
US Senate Appropriations Committee, 2006
7
And it doesn’t appear that things are getting better
“Not all computing projects fail – only most of them. Now and again serendipity sees a company or government department buying and implementing a system that does as much as half of what was originally intended.”
Collins and Bicknell, 1998
“even when technically successful, ICT projects do not often deliver the financial and other benefits they promise. It is the remarkable ubiquity of the failure of ICT projects – particularly large ICT projects – and the large sums of money that can disappear as a result that should be of most concern”
Gauld and Goldfinch, 2006
And empirical research concludes…
“There is a demonstrated, systemic tendency for project appraisers to be overly optimistic. This is a worldwide phenomenon that affects both the private and public sectors…appraisers tend to overstate benefits, and underestimate timings and costs.” HM Treasury
Forecasts are “highly, systematically and significantly misleading (inflated). The result is large benefit shortfalls”.
Flyvbjerg
“Delusional optimism: we overemphasise projects’ potential benefits and underestimate likely costs, spinning success scenarios while ignoring the possibility of mistakes.” Lovallo and Kahneman
So we face 3 Benefit Challenges
1. Ensuring benefits claimed are robust and realisable.
2. Planning for all potential forms of value created.
3. Realising benefits and creating value – exploiting the capability created.
Capturing all forms of value
Moving beyond hurdle rates of return to ask – have all potential forms of value been identified?
Going beyond forecast to creating value
• Managing benefits from a portfolio perspective.
• Continuous Participative Engagement.
• Learning from experience.
From optimism in planning and pessimism in implementation to realism in planning and enthusiasm in
implementation
But – the problem starts with the business case so start with the end in mind
“It has often been observed when analysing IS/IT project failures that all the ingredients for failure existed on Day One of the project, or even before it started!”
Ward and Taylor
1. Understand the benefits you are buying
Beware ‘Strategic alignment’ as an investment justification,
“Our CEO defines ‘strategic projects’ as expensive projects without a Business case.”
Corporate Executive Board paper
Understand the benefits you are buying- Strategic Contribution Analysis
Understand the benefits you are buying- Strategic Contribution Analysis
Using a consistent benefits categorisation framework
User value Monetary Time-based non-monetary Value-based non-monetary
(such as less frustration from reduced unnecessary contact)
Departmental efficiency Contributions to departmental efficiency plans/targets (from website rationalisation, channel shift, shared infrastructure, removal of duplicate processes)
Departmental effectiveness Improved policy impact Increased regulatory
compliance, etc
Wider Public Value Trust Reputation Inclusion, etc
Understand the benefits you are buying- Efficiency benefits
Beware staff time savings – they are
vouchers
17
Understand the benefits you are buying- Efficiency benefits
“We were survivors, dwellers forever in the cracks of the vast organisational chart. Disperse us, downsize us, squash us, transfer us, and we will reassemble someday, somewhere, to once again build new layers of redundancy, waste, and glaring irrelevance.”
Lerner
2. Triangulate and Validate forecasts
• Use Reference class forecasting.
• Validate the benefits with the recipients – before investment.
• Establish effective accountability for benefits realisation.
3. Appraise Attractiveness/Return in the context of Achievability/Risk
4. ‘Gates with teeth’
• Challenge and Support.
• Based on joint planning for success.
•Staged release of funding
• Concludes with a formal re-commitment to benefits realisation.
5. A ‘Clear line of sight’ from strategic intentto benefits realisation
• Investment Appraisal report
• Benefits Scorecard
• Portfolio Analysis • Proving Model report
SR2004 CSR07 10 Year TotalQuality of
Benefit Forecast
Scale of Benefits Forecast
Quality ofRealisation
PlanningLikelihood ofRealisation
Confidence Victims and Witnesses OBTJ Enforcement Re-offending Q1 03/04 - Q4
05/06 2006/07 2007/08NSPIS Custody & Case Prep 79.08 199.82 348.06 AMBER AMBER AMBER AMBER Realised Plan ActualPentiP - 5.08 14.06 GREEN GREEN AMBER AMBER Efficiency Cashable - - 0.39 0.71 COMPASS CMS (50%) 10.95 20.79 52.74 GREEN AMBER GREEN AMBER HD=1 Efficiency Opportunity 4.95 2.69 26.136 50.59 NWNJ IT tool (WMS) 5.70 9.66 25.02 GREEN GREEN GREEN AMBER D=2 HD=1 NK=1 HD=1 Effectiveness 0.26 0.24 0.94 1.12 SOCA 0.61 1.21 3.43 GREEN GREEN GREEN GREEN M=1 Total 5.21 2.93 - 27.473 52.42 Libra application (includes benefits jointly delivered with 3a) 18.15 81.70 154.31 GREEN RED AMBER AMBER M=1 NK=1
OASys 79.52 93.18 239.12 GREEN GREEN GREEN GREEN MC=2Q1 03/04 - Q4
05/06 2006/07 2007/08NOMIS (70%) 5.06 50.00 162.33 GREEN GREEN GREEN GREEN MC=1 HD=1 Realised Plan ActualViSOR 0.93 2.21 6.08 GREEN GREEN GREEN GREEN Efficiency Cashable - - - - CJSE Release 1a (NSPIS-CMS) 1.63 9.76 17.88 GREEN GREEN GREEN AMBER Efficiency Opportunity 31.40 8.88 35.50 39.04 CJSE Release 1b (NSPIS - Libra) 0.87 19.08 37.29 AMBER GREEN AMBER AMBER Effectiveness 0.13 0.39 1.56 2.83
XHIBIT/CJSE Release 2a&b (XHIBIT Portal) 9.29 7.11 21.15 AMBER RED GREEN AMBER M=1 D=1 M=1 Total 31.53 9.27 - 37.06 41.87 PROGRESS 1.88 12.02 26.04 GREEN GREEN GREEN GREEN M=4 HD=1 D=2
Secure Email/Emailing Securely 1.49 1.90 4.78 AMBER RED AMBER AMBER D=2Q1 03/04 - Q4
05/06 2006/07 2007/08CJSE Release 3a 0.00 0.01 0.02 AMBER AMBER AMBER Realised Plan ActualCJSE Release 3b 1.10 2.13 6.07 GREEN AMBER AMBER Efficiency Cashable - 0.70 17.45 COMPASS infrastructure (50%) 85.95 95.79 252.74 see COMPASS CMS Efficiency Opportunity 2.59 0.53 2.67 1.70 Libra enabled 30.17 34.26 95.54 see LIBRA application Effectiveness - 0.34 2.54 4.42 NOMIS infrastructure (30%) 2.17 21.43 69.57 see NOMIS application Total 2.59 0.87 - 5.91 23.57 OMNI infrastructure 1.03 3.11 6.48 OMNI cost effectiveness - 9.36 36.37
LINK enabled 8.68 11.79 28.49 Q1 03/04 - Q4
05/06 2006/07 2007/08
Shared Access 5.85 7.80 24.05 D=2 Realised Plan ActualEquip direct 16.50 16.50 55.00 Efficiency Cashable - - - - Equip enabled 41.41 149.10 311.00 Efficiency Opportunity 26.77 5.68 22.70 24.36
CEI 9.78 21.43 45.54 D=9 M=1 HD=3 D=5 M=1 HD=11 D=3 M=1 D=5 M=1 NK=1 D=1 M=1 Effectiveness 12.40 5.26 21.06 21.21 Economic/Social Value Benefits 15.61 113.53 291.21 Total 39.17 10.94 - 43.76 45.57
433.41 999.74 2,334.36
1274.65 KEY
250.18 Q1 03/04 - Q4
05/06 2006/07 2007/08
12.90% 2003/04 2003-05 2003-06 2003-07 2003-08Benefits as %
of Cost Realised Plan Actual57.09 90.37 142.59 183.78 202.01 Efficiency Cashable - 0.53 1.63 5.21 32.69 85.10 Efficiency Opportunity 0.11 0.05 0.21 0.37
- - - - Effectiveness - 0.53 1.63 5.21 32.69 85.10 42% Total 0.11 0.05 - 0.21 0.37
111.21 194.71 269.22 408.09 519.61 5.50 15.38 39.29 83.26 129.20 - 8.55 10.12 19.22 49.96 Q3 05/06 Q4 05/06
5.50 23.93 49.40 102.47 179.16 34% No. % No. %25.05 58.82 97.87 132.70 170.63 6 6% 2 2%
- 2.05 31.53 68.59 110.46 26 27% 35 36%- 2.05 31.53 68.59 110.46 65% 13 13% 19 20%
154.55 283.61 426.40 512.40 585.28 52 54% 41 42%- 0.30 2.59 8.50 32.07
3.23 8.43 18.80 32.10 47.28 3.23 8.73 21.39 40.60 79.35 14%
3Ring Fence costs from 2003-06 and Delivery Plan RF budget from 2006-08. Full benefits by recipient used. Corrections includes YJB.
RAG
1
2
3
Summary of Key Mitigating Actions
1. Settlement Letter Conditions/Hurdle rates 2. Root Cause Model 3. Social Value Research 4. Analysis of benefits enabled by CJS IT funded infrastructure
1. Quarterly Benefits Integrity Check 2. Benefits Eligibility Framework
Risk Description
2006/07 Q1
Difficulty with tracking/measure.
Benefits Rating
Pro
g
RAG of Benefits in SR2004 Benefits Realisation Plans
2006/07 Q1
YJB Benefits Realisation Plan
Direct Benefits
2006/07 Q1
HMCS Benefits Realisation Plan
2006/07 Q1
Infr
astr
uctu
re
Scale of CJS IT benefits forecast
1. Process Modelling 2. CJO Benefit Realisation Plans approved by OB and BWG 3. Project Benefit Realisation Plans approved by BWG CJSIT benefits realisation
Total 10 year CJS IT Application benefits Ring Fence only
CJS IT Application NPV Ring Fence only
CJS IT Application IRR Ring Fence only Cost Benefit Analysis3
Quality of CJS IT benefits forecast
Risk Register - CJS IT Benefits Management
CJS IT BENEFITS SCORECARD June 2006This table shows the full CJS IT Programme benefits picture, including ring fence benefits and benefits enabled by the ring fence infrastructure (£m)
Ap
pli
cati
on
s
CJS IT Benefits Realisation Plans
CJS IT Projects
Forecast Benefit Values Self assessment of RAG status Police Benefits Realisation Plan
TOTAL BENEFITS
Total CPS RF Cost/BudgetDirect & Enabled BenefitsTotal CPS Benefits
Total Corrections RF Cost/BudgetDirect BenefitsEnabled BenefitsTotal Corrections Benefits
Total Police RF Cost/Budget
Enabled BenefitsTotal Police Benefits
Total HMCS Benefits
Total HMCS RF Cost/BudgetDirect BenefitsEnabled Benefits
Forecast
2. Independent Scores assessed by Proving Services
Benefit on track/ahead of scheduleBenefit not yet due for realisation
Benefit behind schedule
NOTE: Benefits shown only include quantified, validated benefits but other enabled benefits have been identified
and will be included as further work is undertaken
Forecast
Forecast
Forecast
Forecast
CPS Benefits Realisation Plan
2006/07 Q1
NOMS Benefits Realisation Plan
1. Based on BRP submissions & agreed with Strand Board leads, this shows the number of benefits to the Strands in terms of value (ie. mission critical, highly desirable etc)
MC = Mission Critical, HD = Highly Desirable, D = Desirable, M = Minimal, NK = Not Known
Strand Board Alignment1
Relative Contribution to
Strategic Drivers2
Benefits to the CJS and Society
Home CJO, 76%
x-CJS, 9% Social Value, 16%
Home CJO x-CJS Social Value
Benefits By Type
Efficiency Opportunity
61%
Efficiency Cashable
9%
Effectiveness30%
Efficiency Cashable Efficiency Opportunity Effectiveness
ATTRACTIVENESS ANALYSIS ECONOMIC ANALYSIS* NPV (£M) IRR (%) PAY BACK PERIOD
(YEARS) EXCLUDING OPTIMISM BIAS
INCLUDING OPTIMISM BIAS
CONFIRMATION THAT THE ABOVE FIGURES ARE COMPLIANT WITH BENEFITS ELIGIBILITY FRAMEWORK (BEF) Yes No
CONFIRMED BY: COMMENTS:
BENEFITS ANALYSIS OVER 10 YEAR PROJECT LIFE CYCLE (£M) STATE 10 YEAR PERIOD COVERED OR PROJECT LIFE SPAN IF LESS THEN 10 YEARS: RECIPIENT EFFICIENCY –
CASHABLE EFFICIENCY – OPPORTUNITY VALUE
EFFECTIVENESS - CASHABLE
EFFECTIVENESS – OPPORTUNITY VALUE
CJO: Police CJO: CPS CJO: DCA CJO: NOMS CJO: YJB CROSS CJS: BEYOND THE CJS(SPECIFY):
TOTALS:
INC
LU
DE
D IN
B
RP
A
GR
EE
D IN
P
RIN
CIP
LE
NO
T Y
ET
A
GR
EE
D
SOURCE OF CONFIRMATION OR PROPOSED ACTIONS & TIMESCALES TO ENSURE BENEFITS ARE INCLUDED IN BENEFITS REALISATION PLAN
BENEFITS INCLUDED IN THE POLICE BENEFITS REALISATION PLAN (BRP) OR AGREED IN PRINCIPLE BY THE POLICE BENEFITS REALISATION LEAD(BRL)
BENEFITS INCLUDED IN THE CPS BRP OR AGREED IN PRINCIPLE BY THE CPS BRL
BENEFITS INCLUDED IN THE DCA BRP OR AGREED IN PRINCIPLE BY THE DCA BRL
BENEFITS INCLUDED IN THE NOMS BRP OR AGREED IN PRINCIPLE BY THE NOMS BRL
BENEFITS INCLUDED IN THE YJB BRP OR AGREED IN PRINCIPLE BY THE YJB BRL
ACHIEVABILITY ANALYSIS
PROVING SERVICES ACHIEVABILITY SCORE BREAKDOWN CONCEPT CASE (COMPLETE AS APPROPRIATE)
BUSINESS CASE (COMPLETE AS APPROPRIATE)
ASSESSMENT OF DEGREE OF BUSINESS CHANGE REQUIRED TO REALISE BENEFITS RECIPIENT BRL ASSESSMENT COMMENTS
HIG
H
ME
DIU
M
LOW
N
/A
Police CPS DCA NOMS YJB Others [specify]
PROVING MODEL ASSESSMENT DATE OF CURRENT ASSESSMENT: ACHIEVABILITY SCORE: PROGRESS SINCE LAST ACHIEVABILITY ASSESSMENT:
(1) TOTAL PROJECT COSTS YEAR RESOURCE
(£M) CAPITAL
(£M) TOTAL (£M)
TOTAL
(2) RESOURCE/CAPITAL CURRENTLY ALLOCATED FROM THE RING FENCE
(3) OTHER FUNDING SOURCES
YEAR RESOURCE (£M)
CAPITAL (£M)
TOTAL (£M)
RESOURCE (£M)
CAPITAL (£M)
TOTAL (£M)
TOTAL
RESOURCE/CAPITAL GAP (1-(2+3)) RESOURCE/CAPITAL REQUIRED FROM THE RING FENCE
YEAR RESOURCE (£M)
CAPITAL (£M)
TOTAL (£M)
RESOURCE (£M)
CAPITAL (£M)
TOTAL (£M)
TOTAL
FUNDING – OTHER ISSUES AVAILABILITY CONFIRMED? FUNDING AVAILABLE TO REALISE BUSINESS CHANGE? Yes
No BY: DATE
CONFIRMATION THAT OVERSPENDS WILL BE MET BY DEPARTMENTAL BASELINES
Yes No
BY: DATE
RECYCLING OF COST SAVINGS YEAR LEGACY SYSTEM
SAVINGS SAVINGS
RECYCLED? AMOUNT BY WHICH THE RING FENCE
REQUEST CAN BE REDUCED TOTAL
• Recipient Organisation Benefits Realisation Report
• Contributor project benefits report
RecipientCost Avoidance
Self Assessment Cashable Opportunity Cashable Opportunity £ %Crown Courts 12.700 35.620 0.000 25.920 0.000 74.240 6.7%
Quality of Benefits Forecast Other CJOs 0.000 117.820 0.000 52.760 0.000 170.580 15.3%Scale of Benefits Forecast Cross CJS 0.000 0.000 0.000 173.710 0.000 173.710 15.6%Quality of Realisation Planning Outside CJS 0.000 0.000 0.000 694.860 0.000 694.860 62.4%Likelihood of Realisation Total £ 12.700 153.440 0.000 947.250 0.000 1113.390 100.0%
Benefits Realised this financial year £m Last report NowMonetary Value of Benefits Realised YTD 0.01 NPV £m* 154.301 -
4.10.0%
Variance -4.09
Performance £ M
From 1 month after Go-live at each court centre
Reduced waiting time claim valuesBenefit not yet due for realisation10 Reduction in solicitor waiting - 12.7 Efficiency cashable
PSA Targets 2, 3 and 5
Crown Court
Following rollout to all prisons in a CJA
Benefit not yet due for realisation9 NOMS Benefits - 36.9 Effectiveness opportunity
PSA Target 5
NOMS
Benefit ahead of or in line with plan10
Benefit ahead of or in line with plan
No of bail forms sent and time taken
9
3 months after go live in a CJ Area
No of enquiries made to CCBenefit ahead of or in line with plan
No of daily/warned lists sent and time taken
8 Police Benefits - 115.52 Efficiency opportunity
PSA Target 5
Police
3 months after go live in a CJ Area
Number of enquiries made to CCBenefit ahead of or in line with plan
No of daily lists faxed/time taken.
7 CPS Benefits - 2.3 Efficiency opportunity
PSA Target 5
CPS
From 1 month after Go-live at each court centre
Backlog countBenefit ahead of or in line with plan
Number of disposals
6 Crown Court Efficiency benefits - 35.61 Efficiency opportunity
PSA Targets 2, 3 and 5
Crown Court
From 3 months after Go-live at each court centre
Number of disposalsBenefit ahead of or in line with plan5 Court time saving (Same day delays) - 5.98 Effectiveness opportunity
PSA Targets 2, 3 and 5
Crown Court
From 6 months after Go-live at each court centre
Percentage of ineffective hearingsBenefit not yet due for realisation4 Court time saving ( Ineffective Hearings)
Assumption of improvement of 1-10% 31.78 Effectiveness opportunity
PSA Targets 1, 2, 3 and 5 Crown Court, Police, CPS, Probation
From Go-live at each court centre
No. of adjournments as a result of no PSRBenefit ahead of or in line with plan3 Court time saving (PSR adjournments)
Improvement from 15% to 3.4% 4.06 Effectiveness opportunity
PSA Targets 2, 3 and 5
Crown Court
To start 6 months after completion in a CJ Area
Time from CC result to PNC update
2 Reduction in the cost of crime
PNC update down to 3 days 694.86 Effectiveness opportunity
PSA Targets 1 and 3
UK economy
To start 6 months after completion in a CJ Area
Time from CC result to PNC updateBenefit realisation behind Plan
Realisation Ramp Up Key Measures/Indicators of Benefit
Benefits Realisation
Status
1 Reduction in the cost of crime
PNC update down to 3 days 173.71 Effectiveness opportunity
PSA Target 5
CJS
94.2*NPV discounted at 3.5%
Project Life Cycle Benefits Profile (Top 5-10 benefits over 10 years)
Benefit Description
Impact
Benefit Category Strategic AlignmentMain Recipient
Name(s) of recipient agreed with
Q4 Assessment
Q4 forecast for Q1
Forecast in latest published Delivery Plan YTD IRR %
Approved by SRO:
Benefit not yet due for realisation
CJS IT BENEFITS PROGRESS REPORT - QUARTER 4 2004-05 FOR CJS EXCHANGE XHIBIT PORTAL
10 Year Analysis
Efficiency £ M Effectiveness Total
Q3 Assessment
Q3 Forecast for Q4
Cumulative Cost and Benefit Forecast 2003-2013
-
200
400
600
800
1,000
1,200
2003-04
2004-05
2005-06
2006-07
2007-08
2008-09
2009-10
2010-11
2011-12
2012-13
Year
£'m
Cumulative Benefits Cumulative Costs
6. Independent review
“humans not only are prone to make biased predictions, we’re also damnably overconfident about our predictions and slow to change them in the face of new evidence. In fact, these problems of bias and overconfidence become more severe the more complicated the prediction.”
Ayers
6. Independent review
Ayers suggests an, “‘Advocatus Diaboli’… whose job it is to poke holes in pet projects. These professional “No” men could be an antidote to overconfidence bias.”
Davidson Frame proposes the use of “murder boards” to pull a proposal apart to, “make sure that arguments in support of project ideas do not have built into them the seeds of their own destruction.”
Steve Jenner - I suggest a ‘fool’ to ask the questions others don’t dare to ask and identify those, ‘assumptions that masquerade as facts’.
