fairness monitor final report › se-fm › 2016 › documents › ... · fairness monitor final...
TRANSCRIPT
CANADIAN COAST GUARD
HELICOPTER RENEWAL PROJECT
FULL FLIGHT SIMULATOR
FAIRNESS MONITOR FINAL REPORT
Project: Canadian Coast Guard Helicopter Renewal Project – Full
Flight Simulator
Date of submission: January 7, 2016
Submitted to: Director, Fairness Monitoring
Submitted by:
PPI Consulting Limited
Centrepointe Chambers
86 Centrepointe Drive
Ottawa ON K2G 6B1
613-567-0000
www.ppiconsultinglimited.com
Ottawa | Toronto | Atlantic Canada
Table of Contents 1 INTRODUCTION ..................................................................................................................................... 3
2 ATTESTATION OF ASSURANCE .............................................................................................................. 4
3 PROJECT REQUIREMENT ....................................................................................................................... 5
4 FM ENGAGEMENT AND OBSERVATIONS .............................................................................................. 5
4.1 REQUEST FOR INFORMATION PROCESS ........................................................................................... 6
4.1.1 Activity Monitored ........................................................................................................................ 6
4.1.2 FM Specialist Observations - RFI Process ...................................................................................... 7
4.2 REQUEST FOR PROPOSALS PROCESS ................................................................................................ 7
4.2.1 Bid Open Period ............................................................................................................................ 7
4.2.2 Technical Bid Evaluation ............................................................................................................... 7
4.2.3 Financial Bids ................................................................................................................................. 9
4.2.4 FM Specialist Observations – RFP Process .................................................................................... 9
5 REFERENCE DOCUMENTS ................................................................................................................... 11
1 INTRODUCTION PPI Consulting Limited (PPI) was engaged as a Fairness Monitor (FM) on August 22, 2012, to observe the
procurement process related to the Canadian Coast Guard’s (CCG) acquisition of Light, Medium and
Polar Helicopters and a Full Flight Simulator. This report relates to the procurement process for a Full
Flight Simulator. The Request for Proposals (RFP) was issued by Public Works and Government Services
Canada (PWGSC), now Public Services and Procurement Canada (PSPC), on behalf of the Canadian Coast
Guard as a result of Solicitation # F7013-120014/I. PPI Consulting Limited is an independent third party
with respect to this activity.
We reviewed all of the information provided and observed all relevant activities.
We hereby submit our Final Report, covering the activities of the Fairness Monitor, commencing with
the Letter of Interest/Request for Information process, Solicitation No: F7013-120014/G issued on
October 16, 2014 continuing through the Request for Proposals process, Solicitation No: F7013-120014/I
issued on June 29, 2015, and subsequent evaluation.
This report includes our attestation of assurance, a summary of the scope and objectives of our
assignment, and relevant observations from the activities undertaken.
2 ATTESTATION OF ASSURANCE It is our professional opinion that the Request for Proposals process for the Canadian Coast Guard’s
procurement of a Full Flight Simulator, that we observed, was carried out in a fair, open and transparent
manner.
Original signed by
John Davis, CEO, PPI Consulting Limited
Original signed by
Ian Brennan, CSCMP, FM Team Leader
Original signed by
Mairi Curran, FM Specialist
3 PROJECT REQUIREMENT The CCG provides services for the safe, economical and efficient movement of ships in Canadian waters
through the provision of aids to navigation systems and services, marine communications and traffic
management services, icebreaking and ice management services, and channel maintenance. CCG is also
responsible for the marine component of the federal search and rescue program and responding to
marine pollution.
CCG developed a Fleet Renewal Plan, a long term investment strategy to modernize its fleet of aging
vessels and helicopters. The Fleet Renewal Plan included the acquisition of Light, Medium and Polar
Helicopters and a Full Flight Simulator. Two previous procurement processes have resulted in the
selection of the Bell 429 light lift helicopter and the Bell 412 EPI medium lift helicopter. The Simulator
will provide a platform for training for the CCG Configuration A Bell 429 light-lift helicopter and the CCG
Configuration A Bell 412 EPI medium-lift helicopter for the anticipated 30-year lifecycle of the new fleet.
The Full Flight Simulator is required to be sufficiently capable and configurable to conduct both initial
and recurrent helicopter type training and proficiency checking, as well as relevant scenario based
training in a realistic synthetic environment compatible with CCG helicopter operations.
4 FM ENGAGEMENT AND OBSERVATIONS PPI Consulting Limited was engaged as a Fairness Monitor (FM) to observe the Request for Information
(RFI) Process and Request for Proposals (RFP) process for the Canadian Coast Guard’s proposed
procurement of a Full Flight Simulator and to attest to the fairness, openness and transparency of this
monitored activity.
In accordance with the terms of our engagement, we familiarized ourselves with the relevant draft
documents, observed activities up to the end of the Request for Proposals phase, identifying fairness-
related matters to the Contracting Authority and to the CCG Technical Authority and ensuring that
responses and actions were reasonable and appropriate.
The following is a summary of the FM’s activities over the course of the engagement and related
observations.
4.1 REQUEST FOR INFORMATION PROCESS
4.1.1 Activity Monitored During the RFI Phase, the FM Specialist attended the following meetings:
Three rounds of Commercially Confidential Meetings as follows:
o Round 1 – held November 12, 13 and 14, 2014 with five (5) interested Full Flight
Simulator Manufacturers;
o Round 2 – held January 28, 29, 30 and February 2, 2015 with four (4) interested Full
Flight Simulator Manufacturers; and
o Round 3 – held April 29 and May 5, 2015 with two (2) interested Full Flight Simulator
Manufacturers. A third manufacturer cancelled its previously arranged meeting.
Site visits to three Full Flight Simulator Manufacturers, held February 19, 20 and 24, 2015; and
Project team meetings/discussions to review status, evaluation approach, issues resolution,
evaluation criteria and specifications.
During the RFI Phase, the FM Specialist was provided with the following documents for review:
Request for Letter of Interest;
Draft Request for Proposal (RFP) documents posted March 31, 2015. The FM Specialist was
provided with multiple iterations of the draft documents for review. The contracting authority
satisfactorily clarified all questions the FM raised with respect to the technical
specifications/process;
Correspondence with industry e.g. containing clarifications to questions raised during the
Commercially Confidential Meetings;
Draft Evaluation Plan; and
Minutes of Commercially Confidential Meetings.
The documents and activities completed in this phase complied with the fairness principles below:
All potential proponents were provided with the same opportunity and the same information at the
same time;
Potential proponents were afforded the same opportunity to provide feedback with respect to the
requirements and proposed procurement approach;
Potential proponents’ feedback received due consideration by the client based on an assessment
against the client’s operational and business requirements; and
Requests for further explanation or clarification were provided to all participants.
4.1.2 FM Specialist Observations - RFI Process
The FM Specialist reviewed the draft solicitation documents and attended three rounds of Commercially
Confidential Meetings with interested Full Flight Simulator Manufacturers as well as on site visits to
three of the interested manufacturers.
Any requests for clarification by the Full Flight Simulator Manufacturers during the Commercially
Confidential Meetings were addressed by the CCG team and by the Contracting Authority either during
the meetings or provided following the meetings. The same information was provided to all
participants.
Any observations or request for clarification made by the FM Specialist were addressed by the CCG team
and by the Contracting Authority.
The CCG team and the Contracting Authority were responsive in providing clarification when requested
by the FM Specialist.
4.2 REQUEST FOR PROPOSALS PROCESS
During the RFP Phase, the FM Specialist was provided with the following documents for review:
Draft and final RFP documents;
Draft and final Evaluation Plan;
Review of clarification questions and answers during bid open period; and
Correspondence with industry.
4.2.1 Bid Open Period The RFP for a Full Flight Simulator was issued as Solicitation No. F7013-120014/I, dated June 29, 2015
with an original closing date of August 31, 2015. During the Bid Open Period six Amendments were
posted on the open bidding system responding to 37 questions from industry. In Amendment 006 dated
September 25, 2015, the closing date was extended to October 7, 2015.
Two proposals were received by the submission closing date and time.
4.2.2 Technical Bid Evaluation
The FM Specialist attended the following meetings:
Evaluation meetings with three separate teams of evaluators held October 15, 21, 23, and
November 9 and 12, 2015; and
Meeting with the PSPC Contracting Authority to review and verify price proposals held
November 27, 2015.
The following process was undertaken to evaluate the bids:
Each evaluator conducted their own independent review and scoring of the proposal in their
evaluator workbook.
Once the independent reviews were completed, the evaluators participated in meetings to
agree the consensus assessment for both mandatory and rated requirements.
o Technical mandatory requirements were reviewed for compliance with the information
provided in both the compliance matrix and in the substantiating information in the
proposal. A record of compliance or non-compliance was recorded in the master record
along with the rationale.
o The scores for rated requirements and the rationale for the scores were recorded in the
master record. The evaluators used a majority rule to determine a score when there
was a variance in evaluator scores. The evaluator who was not in agreement recorded
their score in their workbook along with a comment that they had agreed to the
majority score. The majority score was the score recorded in the master record.
Evaluators recorded and initialled any changes agreed to during consensus in their individual
workbooks.
The FM confirmed with each member of the evaluation team that they were in agreement with
the final consensus scores and rationale for non-compliant technical requirements.
All evaluators confirmed their agreement to the recorded score and that they believed both
proponents had been treated equitably and fairly.
The evaluation team signed the consensus record.
The FM’s review of documents and monitoring of activities in this phase considered the principles of
fairness, openness and transparency. This included the following:
Evaluation Plan is consistent with the published procurement documents;
Evaluation team members were chosen and confirmed prior to the receipt of proposals;
Evaluation training should be provided to all evaluators and observers. This should include
informing evaluators of the following as a minimum:
o identity of the proposals received and requesting evaluators to declare any conflict of
interest;
o confidentiality protocols;
o document control;
o clarification process;
o overview of scoring workbooks and method for individual assessment;
o explanation of hidden criteria;
o explanation of fairness and the need for objectivity, consistency and equitable treatment of
all proposals; and
o guidance with respect to only assessing the information that is in the proposal – not
information from previous contracts, personal knowledge, etc.
The scoring criteria and assessment tools should be established prior to the receipt of proposals and
should be consistent with the RFP, i.e. contain no hidden scoring criteria;
The submissions should be logged and recorded upon receipt, clearly identifying that these were
submitted on time;
The pricing should be contained in a separate envelope (as applicable);
The proposals complied with the mandatory submission requirements;
The process for establishing one score from a team of evaluators should be established prior to the
receipt of proposals (consensus, majority, averaging, etc.);
The same team of evaluators should evaluate all proposals (or parts thereof);
The scoring assessment should be applied consistently and equitably by the evaluation team with no
evidence of bias;
A secure location for the evaluation exercise should be established for the period of the evaluation;
and
Proposal documents should be physically secured within a secure location.
At the conclusion of the technical evaluation one of the two proposals received was deemed non-
compliant with multiple mandatory technical requirements. The second proposal was deemed
compliant.
4.2.3 Financial Bids PSPC’s cost analyst reviewed and confirmed the compliant bidder’s financial capability in accordance
with the process indicated in the RFP.
The proposal also met the Canadian Content certification requirements.
The FM met with the PSPC Contracting Authority to review the financial bid of the compliant proposal.
The financial bid was complete and complied with the requirements of the RFP.
The FM had no fairness concerns related to the financial evaluation.
4.2.4 FM Specialist Observations – RFP Process The FM witnessed the technical bid evaluation sessions and confirms that the process was conducted
appropriately in accordance with the process established in the RFP and in a fair, open and transparent
manner.
The FM reviewed the financial proposal pricing and confirms that the process was conducted
appropriately in accordance with the process established in the RFP and in a fair, open and transparent
manner.
Any requests for clarification by the Full Flight Simulator Manufacturers during the bid clarification
process were addressed by the CCG team and by the Contracting Authority and issued as Amendments.
The same information was provided to all participants.
Any observations or request for clarification made by the FM Specialist were addressed by the CCG team
and by the Contracting Authority.
The CCG team and the Contracting Authority were responsive in providing clarification when requested
by the FM Specialist.
The evaluation teams were knowledgeable and diligent in their evaluation of proposals. Both proposals
were evaluated consistently and fairly.
At the conclusion of the evaluation process, the one compliant proposal was identified as the successful
proposal.
5 REFERENCE DOCUMENTS
# Document Document Date/Time
1. Helicopter Project (DFO) Letter of Interest/Request for Information F7013-120014/A
August 20, 2012
2. Helicopter Project (DFO) Notice of Proposed Procurement F7013-120014/B
December 13, 2012
3. Helicopter Project (DFO) Notice of Proposed Procurement F7013-120014/C
April 3, 2013
4. Helicopter Project (DFO) Letter of Interest/Request for Information F7013-120014/D
April 22, 2013
5. Helicopter Project (DFO) Notice of Proposed Procurement F7013-120014/E
June 6, 2013
6. Helicopter Project (DFO) Notice of Proposed Procurement F7013-120014/F
February 24, 2014
7. Helicopter Project (DFO) Letter of Interest F7013-120014/G Letter of Interest
October 16, 2014
8. Helicopter Project (DFO) Letter of Interest F7013-120014/G Letter of Interest Amendment 001
November 14, 2014
9. Notice of Proposed Procurement (F7013-120014/H) March 31, 2015
10. Notice of Proposed Procurement (F7013-120014/H) Amendment 001
April 10, 2015
11. Request for Proposals, Solicitation # F7013-120014/I, closing date 31 August 2015 ABES.PROD.PW_CAG.B003.E25220.EBSU000
June 29, 2015
12. Amendment 001 ABES.PROD.PW_CAG.B003.E25220.EBSU001
July 13, 2015
13. Amendment 002 ABES.PROD.PW_CAG.B003.E25220.EBSU001
August 14, 2015
14. Amendment 003 ABES.PROD.PW_CAG.B003.E25220.EBSU001
August 21, 2015
15. Amendment 004 ABES.PROD.PW_CAG.B003.E25220.EBSU001
September 4, 2015
16. Amendment 005 ABES.PROD.PW_CAG.B003.E25220.EBSU001
September 11, 2015
17. Amendment 006 ABES.PROD.PW_CAG.B003.E25220.EBSU001
September 28, 2015
Addendum to the Final Report
March 8, 2016
Addendum to the Fairness Monitor Final Report dated January 7, 2016 for the Canadian Coast Guard’s
Full Flight Simulator for its Helicopter Renewal Project.
This Addendum to the Fairness Monitor Final Report covers the period following the conclusion of the
evaluation phase.
Contract Award, Communications and Debriefing
The contract was awarded on February 2, 2016. A letter was sent to the unsuccessful bidder on
February 4, 2016. Both bidders requested a debriefing which was provided to the successful bidder on
February 18, 2016 and to the unsuccessful bidder on March 4, 2016.
The FM was provided with copies of correspondence and debriefing documentation and attended both
debriefings. The process conducted conformed to the requirements of the RFP and both bidders were
treated consistently and fairly.
The FM hereby provides the following unqualified assurance statement concerning the notification of
award, communications and debriefings of each proponent.
It is our professional opinion that the procurement process to select a Full Flight Simulator and Related
Services that we observed was carried out in a fair, open and transparent manner.
Original signed by
John Davis, CEO, PPI Consulting Limited
Original signed by
Ian Brennan, CSCMP, FM Team Leader
Original signed by
Mairi Curran, FM Specialist