Evaluation at NRCan:Information for Program Managers
Strategic Evaluation DivisionScience & Policy Integration
July 2012
2
.
Purpose
• The purpose of this document is to provide program managers with an overview of the evaluation function at NRCan.
• The TBS Policy on Evaluation (April 2009) requires that all direct spending, including all G&C Programs, be evaluated every five years
– most program managers will find themselves participating in an evaluation at some point.
3
.
What is Evaluation?
• Evaluations are the systematic collection and analysis of evidence on the outcomes of programs to make judgments about
– their relevance;
– performance; and
– alternative ways to deliver them or to achieve the same results.”
• Evaluations must be neutral, and evidence-based.
• An evaluation is not the same as an audit.
Evaluations AuditsFocus on whether we are doing the right things, and the extent to which a program is achieving its expected outcomes, in a cost-effective manner.
Internal audit is looking at financial management, processes, controls and risk.
Make assessments on the relevance & performance of programs
Identify strengths and weaknesses in the management control framework
4
.
Why do Evaluations?
• The objective of evaluation is to create a comprehensive and reliable base of evidence to support:
– policy and program improvement;
– expenditure management;
– Cabinet decision-making; and
– public accountability.
• Evaluations are often required to support TB submissions and Memoranda to Cabinet.
• They are also a critical source of evidence for Strategic Reviews, to support resource reallocation (next review will occur in 2014).
5
.
Background: Evaluation Stakeholders
NRCan DM
Canadian Public
Program Managers TBS
Evaluation Reports
Sector ADMs
Cabinet
Strategic Review
Policy & Program Improvement
Expenditure Management/ Cabinet Decision-making
Parliament
Accountability/ Public Reporting
The challenge is to meet the diverse information needs of many stakeholders for decision-making AND public accountability in a timely fashion.
6
.
The Evaluation Cycle
Evaluation Planning
Approvals/Posting
(2-3 months)
Evaluation Assessment
(1-3 months)
Report &Recommendations
(1-4 months)
Implementing Change
Contracting(1-2 months)
Field Work/Analysis(6-8 months)
Management Responses(1 month)
Including planning and approvals/posting, large
evaluations typically take 12-18 months to complete.
Some phases may overlap.
7
.
Evaluation Planning
• NRCan must evaluate all direct program spending, including all ongoing grant and contribution programs, every five years.
• NRCan has developed a five-year Evaluation Plan based on PAA units that is updated annually and approved by the Evaluation Committee.
• In most cases, the evaluation of an individual program will be conducted within the scope of a broader evaluation of a PAA unit.
• The current plan summary appears on the Strategic Evaluation Internet site at: http://nrcan.gc.ca/evaluation/plans-eng.php.
8
.
Phases of an Evaluation
1. Evaluation Assessment (1-3 months)
– research and planning to understand the programs
– develop the Terms of Reference
– obtain approval from the Evaluation Committee
2. Contracting (1-2 months)
– Consultants are often used to supplement in-house staff. Their roles will vary by project.
3. Fieldwork or Data Collection/Analysis (6-8 months)
– develop a detailed methodology report
– methodologies: key informant interviews; focus groups; file/document/literature reviews; surveys; case studies; and data and economic analysis
– analyse information collected from these multiple lines of evidence to develop conclusions
9
.
Phases of An Evaluation, cont’d
4. Reporting & Development of Recommendations (2-4 months)
– prepare preliminary findings and discuss with programs
– draft report
– address comments and revisions
– develop recommendations
5. Management Responses (1 month)
– obtain ADM-approved management responses and action
plans to the recommendations
6. Approvals/Posting of report (2-3 months)
– recommendation by the Evaluation Committee
– approval by the DM
– translation, ATIP review, media lines, release on Internet
10
.
Evaluation Questions and Issues
• Evaluations address relevance and performance.
• Relevance issues focus on:
– continued need for program;
– alignment with government priorities; and
– alignment with federal roles and responsibilities.
• Performance issues focus on effectiveness:
– achievement of expected outcomes; and
– demonstration of efficiency and economy.
• Evaluators work with program managers to develop more detailed evaluation questions relevant to their program.
11
.
Roles and Responsibilities
• Under the TBS Evaluation Policy, Deputy Ministers are responsible for the evaluation function.
• NRCan’s Departmental Evaluation Committee – an ADM-level Committee – is chaired by the DM
• NRCan’s Head of Evaluation – who is also the DG of Planning and Performance Management Reporting – reports to the Evaluation Committee.
12
.
The Role of the Evaluation Division
• The Strategic Evaluation Division (SED) is responsible for:
– Proposing a five year departmental evaluation plan to the Evaluation Committee, and updating it annually;
– Managing and conducting evaluation studies, including managing contracts and deliverables when consultants are used and issuing reports in a timely manner.
• Additionally, SED will help program managers develop their performance measurement strategies, with the goal of ensuring that good data is collected to support future evaluations.
– Evaluation will work with your team to develop objectives, a logic model, a performance measurement framework and evaluation requirements.
• NRCan's Strategic Evaluation Division is also responsible for reviewing and providing advice on the accountability and performance provisions in Cabinet documents (Memoranda to Cabinet (MCs) and TB Submissions).
13
.
The Role of Program Managers
• Program managers are key to conducting evaluations.
• They are responsible for developing, implementing and monitoring ongoing performance measurement – the foundation of evaluation.
• Additionally, during an evaluation, they must be actively involved in:
– explaining how their programs work;
– contributing to evaluation planning, including identifying more detailed evaluation questions;
– providing performance measurement information on resources used, activities undertaken and results achieved;
– providing detailed documentation (see next slide) and suggestions on potential interviewees, case studies etc.;
– participating in working groups to review questionnaires, preliminary findings, draft evaluation reports, etc.;
– developing management responses and action plans for their ADMs and implementing them after the evaluation.
14
.
Key Documents for An Evaluation
• In preparation for an evaluation, program managers will be asked to provide key documents as early as possible, including:
– Legislation, Regulations, MCs, TB Submissions
– RMAFs, RBAFs or Performance Measurement Frameworks
– references in budgets, SFTs, DPRs, RPPs
– briefing notes, reports --including annual and project reports, studies, databases
– websites and communications products
– five-years of financial expenditures for the PAA unit including
G&C expenditures, O&M, and salaries
15
.
Questions and Assistance
• If you have any questions on evaluation, or wish evaluation assistance in developing performance measurement information or Cabinet documents, please contact:
• The Director of Strategic Evaluation
(613) 996-9649
• Electronic copies of this document, completed evaluation reports and the Terms of Reference for the Evaluation Committee are available at:
http://nrcan.gc.ca/evaluation/index-eng.php