british columbia ministry of forests post-implementation review framework june 2004 engagement:...
Post on 29-Dec-2015
219 Views
Preview:
TRANSCRIPT
British Columbia Ministry of ForestsPost-Implementation Review Framework
June 2004
Engagement: 220636851
Version 1.0
x
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 2
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Table of Contents
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 3
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Table of Contents
Introduction
PIR objectives
PIR model
PIR process
x
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 4
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Introduction
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 5
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Introduction
In light of the ever-increasing size and complexity of technology projects, organizations need to constantly review how projects are delivered and whether or not the project delivered the benefits that were anticipated.
As well, organisations need to continually build on the lessons they are learning though the course of a project to ensure that the same mistakes are not being repeated over and over again but also to capitalise on best practices identified in a project and propagate those to other projects within the organisation.
Gartner advises that public sector organizations implementing projects of the scope and magnitude of those of the ministry should perform a project assessment to assure project deliverables are aligned - not only with stated objectives of the project - but also with those of the government as a whole and the constituents it serves.
Experience demonstrates that the value of an independent review - that can be conducted either by a government audit organization or authority, or by an external trusted party - is to highlight new priorities and suggest which on going initiatives should be reconsidered, accelerated or initiated to fit those priorities. Performing this as a totally independent review is very important, given its implications and possible influences.
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 6
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Guiding principles
Some key guiding principles were followed though the development of this framework: Alignment with British Columbia Government standards
The framework was built taking into consideration the following documents:– 1. MoF Administrative Closure
– 2. XGOV PIR
Alignment with industry best practices Several standards were use in the definition of the framework:
– Project management institute: As the main standard-setting body in project management, PMI has defined the best practices for project management used in this framework.
– Gartner independent review and verification: Inspired form the IEEE Standard 1059–1993, Guide for Software Verification and Validation Plans, it describes best practices with respect to application development.
Simple and easy to understand Since PIR are often seen as “non-value added” by the project teams, the framework aimed at being simple to understand (requiring a minimum of investment in time to understand it) and
easy to administer.
Actionable The major benefit from a PIR is not in understanding what went right or wrong, but in ensuring that we can apply those lessons learned on other projects. The framework focus heavily on identifying those lessons learned and defining prevention / remediation plans to apply those lessons learned in other projects.
Bc Im/it coe policy
x
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 7
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
PIR’s objectives
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 8
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Objectives of a PIR
The purpose of a post-implementation review (PIR) is to evaluate an investment after it has completed development (e.g., after its transition from the implementation phase to the operation and maintenance phase) in order to validate actual investment results. This review is conducted to (1) examine differences between estimated versus actual investment costs and benefits and possible ramifications for unplanned funding needs in the future and (2) extract “lessons learned” about the investment selection and control processes that can be used as the basis for management improvements. Similarly, PIRs should be conducted for investment projects that were terminated before completion to readily identify potential management and process improvements.
The timing of a PIR can be problematic – a PIR conducted too soon after a investment has been implemented may fail to capture the full benefits of the new system. In contrast, the institutional knowledge about a investment can be lost if the PIR is conducted too late. As a general guideline, PIRs should be conducted within a range of 6 to 18 months after the investment begins its operational phase. However, this guideline should be adjusted depending upon the nature of the investment project and expectations for the timing of benefit realizations documented in the project plans.
x
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 9
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
PIR Model
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 10
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Seven evaluation domains
1. Project benefits
2. Project management
3. Requirements Definition
4. Project Organization
5. Quality Assurance
6. Risk Management
7. Project communication
The framework is based on seven evaluations domains. These domains aim at providing the
greatest coverage of the project critical success factors while keeping the framework simple.
By evaluating each of these seven domain, BC MoF will be able to gain a much better
understanding of what went right and what went wrong in its projects.
This framework is only to be used as a tool to guide the organisation in the definition of best
practices. BC MoF will have to determine how the results of this framework can best serve in the
definition of best practices.
Evaluation domains
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 11
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Evaluation domain:1. Project benefits
Objectives:
Elements evaluated:
Rational:
Validate that the benefits were properly identified and realized throughout the course of the project.
Benefits identificationBenefits realization scheduleOwnership of planned benefitsBenefit realization
The benefits of a project are the only reason the project is executed in the first place. If a project fails to meet its objectives (or to define them), then it has no reasons to be.
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 12
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Evaluation domain: 2. Project management
Objectives:
Elements evaluated:
Rational:
Validate that project management best practices were followed throughout all the phases of the project.
Initiation phase: Appropriate charterPlanning phase: Project plan, understanding of the project objectives by every member of the project teamExecution phase: Attention to quality, time and costsControlling phase: Control of the key project metrics, attention to changing circumstances, validation of hypothesis. Closing phase: Validating that objectives were met, sign-off on all deliverables.
Proper project management is essential for the execution of a project. Too often, projects are planned to a great level of details only to realize that those plans were never followed throughout the project.
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 13
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Evaluation domain: 3. Requirements Definition
Objectives:
Elements evaluated:
Rational:
Validate the process used to gather requirements and the validity of the requirements identified throughout the project. Validate the mechanism to identify trade-offs between the requirements and project constraints.
Process used to gather requirements.Relationship between requirements definition and Software Development Life Cycle. Visibility of requirements for gap analysis, release management, design review, testing and training.
Requirements definition is at the basis of the majority of IT projects. Without proper mechanisms and processes to gather requirements, the project might not be working towards the achievement of its business objectives, by working on too many, too few, or the wrong requirements.
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 14
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Evaluation domain: 4. Project Organization
Objectives:
Elements evaluated:
Rational:
Validate that the project was composed of the right mix of vendors, leadership, business and technical resources.
Leadership from the primary business unit(s) and support organization(s).Mid-management and staff involvement.Alignment with business.Skills and experience of the project management staff.Dealing with contractual issues. Alignment of the vendors’ own interest to that of the project.Roles and responsibilities of the entire implementation team
Projects composition tends to be very homogeneous (all IT people for example), promoting groupthink and a lack of understanding of the other realities surrounding the project and its stakeholders.
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 15
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Evaluation domain: 5. Quality Assurance
Objectives:
Elements evaluated:
Rational:
Validate that the project followed best practices with respect to its quality assurance program.
Linkage to requirements. Linkage to risks identified. With sufficient resources, tool (e.g., robots) and time for execution.Allowance for regression testing. Allowance for performance testing.Feedback to developers.
Oftentimes, in the sake of “getting it done”, quality is compromised leading to product defects. A sound, rational quality assurance program ensures that a consistent product is delivered.
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 16
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Evaluation domain: 6. Risk Management
Objectives:
Elements evaluated:
Rational:
Validate that the project risks were identified and managed throughout the course of the project.
On-going risk management plan – identification / assessment of project risks and risk mitigation steps.Visibility of project risks to the entire project team (internal and vendors).Ability to execute risk mitigation plans – e.g., additional resources, changes in schedule, communication with end users, etc.Striking balance between project risks versus time and budget. The importance of having appropriate and sufficient documentation,Linkage of risks with product support strategies (e.g., develop support strategies at the outset not after the fact).
Project teams are typically good at identifying risks but less so in managing them. Oftentimes, the risk management process is neglected during project execution, in the most critical phase of the project.
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 17
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Evaluation domain: 7. Project communication
Objectives:
Elements evaluated:
Rational:
Validate that the project team identified the project stakeholders, developed a communication plan and executed on the plan.
Identification of the project stakeholdersIdentification of their communication requirementsInvolvement of project stakeholders through various mechanisms
Project teams tend to isolate themselves from the project stakeholders, seldom communicating risks and issues leading to surprises for many of the people involved when an issue degenerate. Proper communication can resolve some of these issues before they become problematic.
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 18
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Assessment scale
The following slide describes the assessment scale to be used in the assessment of the different evaluation domains.
Each of the domains are rated based on the information gathered by the PIR team. The assessment is not meant to be a rigorously scientific exercise. Instead, it is meant to allow flexibility to the reviewers to look at each evaluation domains as a whole and determine if their level was appropriate for the size and realities of the project.
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 19
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Assessment scale
5
4
3
2
1
Best-in-class
Represent an example for others to follow.
Success
Best practices were followed systematically.
Passable
Best practices were followed most of the time.
Ad hoc
Mostly reactive approach to the project.
Failure
The project didn’t meet expectations
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 20
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Key questions?
Where does technology fits in? Projects rarely fails because of the technologies they use. Instead they fail because of a
mismatch of the technology for the need (selecting the solution before knowing the problem) or because of risks inherent to the selected technology (unproven technology, lack of skills and experience, bleeding-edge application). These are addressed within the risks, requirements definition and project management domains.
x
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 21
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
PIR Assessment
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 22
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Assessment criteria
The following pages describes some of the evaluation criteria that are suggested for each of the evaluation domains.
The evaluation criteria represent only a portion of all the topics than cold be evaluated under each domain. We believe that by using this subset, BC MoF will be able to obtain a very good understanding of the project status.
The evaluation domains are rated based on the evaluation criteria, but not in a direct relationship. The input from the evaluation criteria are used to assess the overall rating of the evaluation domain. This prevents the positive assessment of an evaluation domain where all the best practices were consistently followed, except for a key one that caused the failure of a project.
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 23
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Assessment criteria1. Project benefits
Are the benefits identified? Are the benefits quantifiable? Were the benefits met? Are the benefits assigned to an owner (sponsor for example)? Are the benefits realistic Is there a benefit realization schedule? Is the benefit realization schedule realistic?
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 24
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Assessment criteria2. Project management
Project initiation Is there a project charter? Is there a project sponsor Is the sponsor accountable for the project objectives? Is the project a priority for the parties involved?
Project planning Is there a project plan? Is the project plan complete? Did the entire team participate in the creation of the project plan? Did the project sponsor sign-off on the project plan? Did the client sign-off on the project plan? (not always relevant) Did the project team sign-off on the project plan? Are project dependencies and interdependencies identified? Does the project takes into account of administrative processes (procurement for example)?
Project execution Are the key metrics in place and reported regularly? Are the required resources assigned to the project? Did the team follow the project team? Was the maintenance team included in the development?
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 25
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Assessment criteria2. Project management
Project control Is there a change management mechanism? Are the deliverables validated by the “client”? Is the project delivering in frequent, small phases? Is the project steering committee engaged in project decisions? Is the project team engaging the different stakeholder groups regularly? Was the project plan updated to reflect changing conditions? Was there frequent project status reports to project sponsor?
Project close-out Were the administrative project closure activities completed? Were all deliverables signed-off by the “client”? Did the project meet its objectives?
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 26
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Assessment criteria3. Requirements definition
Is there a process to gather requirements? Was the process followed? Is there a sign-off process for users, sponsor? Is there a mechanism for users to change their requirements (change
management)? Is there a mechanism for trade-offs between requirements and project
constraints?
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 27
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Assessment criteria4. Project organization
Is there an assigned project manager? Is there a project manager form the “client” organisation? Is there an assigned project team? Is there sufficient resources for the project? Are the resources qualified for the project? Was the turnover-rate for the project high?
Is there a project steering committee? Is the composition of the project steering committee composed of representatives from the different stakeholder groups? Is the project steering committee engaged in the decision-making process?
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 28
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Assessment criteria5. Quality assurance
Is there a quality assurance plan?
Are there quality metrics (targets and actuals)?
Was the quality assurance process followed?
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 29
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Assessment criteria6. Risk management
Were the risks for the project identified?
Was there a mitigation plan built for the identified risks?
Was there a mechanism to monitor risks throughout the project?
Was there contingency plans identified for major risks?
Was there a budget assigned to contingency plans?
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 30
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Assessment criteria7. Project communication
Were the stakeholders identified for the project?
Is there a communication plan for each stakeholders / stakeholder groups?
Was the communication plan followed?
Did the communication plan account for communication-style of the stakeholder groups?
x
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 31
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
PIR Process
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 32
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Who does a PIR?
Project management office Typically responsible for most of the PIRs. Would have a PIR as part of the normal project close-out process. Appropriate for non-eventful projects, where the project is typically considered a success, in order to gain some lessons learned.
External 3rd party Typically done for projects considered challenging of for project having failed. Third parties have no stake in the project and can offer an impartial view which gives more credibility to the PIR results.
Audit team The organization’s audit team could perform the PIR as part of their internal audit duties. Does require some project management and technical competencies.
Senior management Unusual, senior management would typically rely on external 3rd party. Senior management would typically be the ones asking for a PIR for a project having failed.
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 33
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
PIR phases
Data collection
Analysis
Validation
Action plan
There are four phases in a post-implementation review (PIR): data collection, analysis, validation and action plan. Each phase builds from the previous one. The following pages describes each phases in more details.
One of the main issue with respect to PIR is the perceived lack of impartiality, the project team often feels the PIR team is trying to pin the blame on someone. It is therefore very important for the project team to have a chance to review and validate the result of the assessment before presenting them to anyone outside of the project.
In the event of a disagreement between the PIR team and the project team, the concerns of the project team and the reasons for dismissing them should be documented in the PIR report.
PIR Phases
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 34
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Phase 1. Data collection
Objectives:
Process:
Deliverables:
Identify the data sources and Collect the data necessary to perform the PIR.
Documentation list
Interview results
Data collection
Analysis
Validation
Action plan
Surveys: In some circumstances, survey can be used to gather end users impression of the project.
Interviews: Review the evaluation domains with the project sponsor, project manager, project team and key stakeholders of the project.
Documentation review: Review the project charter, project plan, project status report, risk reports, project control book.
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 35
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Phase 2. Analysis
Objectives:
Process:
Deliverables:
Analyse the data gathered in order to generate a preliminary assessment.
Preliminary assessment
Major themes
Data collection
Analysis
Validation
Action plan
Compare interview results to best practices. Point out discrepancies
Score the evaluation domains from one to five according to the assessment scale.
Document the major themes and hypothesis
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 36
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Phase 3. Validation
Objectives:
Process:
Deliverables:
Validate the preliminary assessment with the project team in order to 1) ensure that the assessment is accurate and 2) get buy-in from the project team.
Validated assessment
Data collection
Analysis
Validation
Action plan
Validate the preliminary assessment with the project manager.
Validate the preliminary assessment with the project team.
Update the assessment in light of the validation process.
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 37
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Phase 4. Action plan
Objectives:
Process:
Deliverables:
Develop an action plan to address issues or best practices identified in the assessment.
Action plan with ownerships
PIR Report
Data collection
Analysis
Validation
Action plan
Identify the root causes for the issues identified with the project team
Identify best practices to be repeated in the future
Develop a prevention / remediation plan for the issues and best practices identified
Assign ownership for the prevention / remediation plan
Document findings and action plan in a PIR report
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 38
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
PIR Report Table of Content
1. Executive summary Overall Assessment Lessons Learned Follow-up Actions
2. Background Project Background and Objectives Scope, terms of reference, direction and project team
3. Project assessment and measurement 1. Project benefits 2. Project management 3. Requirements Definition 4. Project Organization 5. Quality Assurance 6. Risk Management 7. Project communication
4. Observation an analysis Best practices identified Major issues identified Recommendations for improvement
5. Recommendation and conclusion Prevention / remediation plan Ownership of the action items Follow-up schedule
For internal use of British-Columbia Ministry of Forests only. © 2003 Gartner, Inc. and/or its affiliates.
All Rights Reserved.Page 39
British Columbia Ministry of ForestsPIR FrameworkEngagement: 220636851—June 2004
consulting
Example of a PIR assessment
0 1 2 3 4 5
Total
7. Project communication
6. Risk Management
5. Quality Assurance
4. Project Organization
3. Requirements Definition
2. Project management
1. Project benefits
In this high-level assessment, we can see that the project was well-managed but failed to do proper quality assurance.
top related