2015 INTERNATIONAL COOPERATION DAYS MAY 12 TH 14 TH 2015 DFATD RBM WHY COLLECT DATA ON OUTCOMES? Information Session 14 May 2015 DFATD - RBM Centre of Excellence The content and descriptions contained in this document have been developed by Result-Based Management Centre of Excellence (RBMCE) of the International Assistance Envelope Management Bureau, Strategic Policy Branch, DFATD and is the property of the Government of Canada. This document may not be edited or modified without prior authorization from RBMCE. 2 2 Objectives Provide a refresher and updated on DFATD RBM post- amalgamation. Assumptions, you: already have solid knowledge of RBM are familiar with former CIDA RBM Policy & RBM How to Guide Provide an overview of RBM links between Project, Country-Program and Corporate Level (key concepts and approach) Look at the synergy between results-based monitoring and development evaluation 3 3 RBM for International Development Programming Although the underlying Results-based Management (RBM) principles and concepts are the same between Trade, Foreign Affairs and Development, there is a difference in approach Former CIDA developed an RBM methodology that reflects the context of their development and humanitarian work, including international commitments (OECD DAC, Aid Effectiveness, MDGs, etc.) The former CIDA RBM Policy still applies 4 4 DFATD RBM and Core Concepts The core concepts that underpin RBM: Results orientation: designing and managing projects during implementation in such a way as to ensure a continuous focus on the achievement of outcomes Causality: making cause-and-effect links between expected outputs and outcomes that reflect a sound theory of change Continuous adjustment: applying results-based monitoring and evaluation by collecting data on outcome indicators to measure progress, comparing expected outcomes with actual outcomes, and adjusting operations throughout project implementation in order to maximize the achievement of results Participatory approach: involving key stakeholders, including intermediaries and beneficiaries Integration of cross-cutting themes: taking into consideration the cross-cutting themes of gender equality, environmental sustainability and governance in all aspects of results-based project planning, design and implementation. These concepts set the foundation for DFATDs approach to RBM of its international development programming. 5 5 What is Results-Based Management? RBM is an iterative modern management approach that requires managers to look beyond activities and outputs and to focus on results (outcomes). RBM is meant to be FLEXIBLE Its aim - to help us manage better for results throughout the project life cycle: from initiation (analysis, project planning and design), to implementation (results-based monitoring, adjustments and reporting), to closure (final evaluations and reports and integrating lessons learned into future programming). In other words, RBM is not a set of tools or instructions. It is a way of thinking about our projects or programs that helps us manage more effectively. By managing better, we can maximize the achievement of development results, (i.e., the positive changes we set out to achieve or contribute to with our programs or projects.) 6 6 Why RBM? Over the past few decades - pressure on governments around the world for greater transparency and accountability to taxpayers for the use of public resources. As noted by TBS (2000), [h]istorically, governments focused their attention on resource inputs (what they spend), activities (what they do) and outputs (what they produce). While information about inputs, activities and outputs is important, it did not tell them whether they were making progress toward addressing the issues they had identified. Losing sight of the results their projects or programs were aiming to achieve limited the effectiveness of their programming. A new management approach was needed that raised the standards of performance and defined success in terms of actual results achieved. RBM (also known as MfR or MfDR) was introduced to meet this need. 7 7 But Why RBM and not another Approach? The focus on activities at the expense of results is what management scholar P. Drucker, in 1954, referred to as the activity trap. Instead, RBM requires that you look beyond activities and outputs to focus on actual results (outcomes): the changes created by your programming or to which it contributed. By establishing clearly defined expected outcomes, assessing risk, collecting output and outcome data to assess progress on the expected outcomes on a regular basis during implementation, and making timely adjustments, practitioners can manage their projects and programs better in order to maximize the achievement of development results. Simple illustration education example 8 8 At the Heart of RBM is Results-based MONITORING Results-based monitoring: The continuous process of collecting and analyzing information on key indicators and comparing actual results with expected results in order to measure how well a project, program or policy is being implemented. [] It is a continuous process of measuring progress toward explicit short, intermediate-, and long-term results by tracking evidence of movement toward the achievement of specific, predetermined targets by the use of indicators. Results-based monitoring can provide feedback on progress (or the lack thereof) to staff and decision makers, who can use the information to improve performance. Performance Monitoring: The ongoing, systematic process of collecting, analyzing and using performance information to assess and report on an organization's progress in meeting expected results and, if necessary, make adjustments to ensure these results are achieved Source: Linda Morra Imas and Ray C. Rist, The Road to Results: Designing and Conducting Effective Development Evaluations (Washington, D.C.: The World Bank), p Source: TBS Lexicon, Retrieved from (add link) 9 9 Traditional vs. RB-Monitoring M&E have always been fundamental aspects of good project and program management. But before RBM, projects and programs used traditional M&E. Traditional M&E focuses on the monitoring and evaluation of inputs, activities, and outputs (that is, on project or program implementation). Results-based M&E combines the traditional approach of monitoring implementation with the assessment of outcomes []. It is this linking of progress on implementation with progress in achieving the desired [] results of government policies and programs that makes results- based M&E useful as a public management tool. Implementing this type of M&E system allows the organization to modify and make adjustments to both the theory of change and the implementation processes in order to more directly support the achievement of desired [] outcomes Source: Linda Morra Imas and Ray C. Rist, The Road to Results: Designing and Conducting Effective Development Evaluations (Washington, D.C.: The World Bank), p 10 Without outcome data, you cannot manage for results The information collected on the progress on outcomes allows you to make evidence-based decisions (our education example). Without evidence of this progress, decisions tend to be based on budgets or other inputs, activities and outputs. If you dont keep an eye on your progress on outcomes, you will never know whether you need to make adjustments to achieve them. When outcome based evidence is not used as a basis for decision making this can undermine the achievement of expected outcomes. This is why results-based monitoring is such a vital component of the RBM approach 10 11 Results-Based Monitoring enables Management Measuring outcomes, and not only outputs, via monitoring, is crucial during the implementation stage. Collecting data on selected indicators on a regular basis during the implementation empowers managers and stakeholders with real time information on progress on the achievement of outcomes. This helps identify strengths, weaknesses, and problems as they occur, and enables project managers to take timely corrective action during the projects life cycle. This in turn increases the chance of achieving the expected outcomes. 11 12 To Conclude This continuous cycle of measurement and adjustment is what makes RBM an iterative management methodology, as opposed to a reporting and data collection one for its own sake. After all, we first have to achieve the changes we set out to achieve before we can collect data and report on them. This focus on measuring progress on outcome during implementation was one of the fundamental changes introduced by RBM. Traditional approaches to management identified objectives or outcomes during planning, once implementation began monitoring focused on inputs, activities and outputs. With the advent of RBM the focus should remained on outcomes, not only during planning, but also during implementation. Data collected through RB monitoring, is essential for RB management 13 RBM is a management methodology that focuses on knowing where you want to be, knowing where you are as you go, and adjusting what you are doing accordingly to ensure you reach your destination. 14 Monitoring roles and responsibilities (R&R) Shared between Implementer, DFATD and Independent consultant or monitor (if applicable) Implementer: Has primary responsibility for collecting and analysing data on all the indicators of the PMF, according to the frequency and data collection method indicated. Monitors continually during implementation to: Help them manage for results (make adjustments) Report on progress on expected results Mainly uses the Performance Measurement Framework (PMF) and risk register. Supported by the monitoring system described in the PIP Also monitors project activities, stakeholders and development context. 15 R&Rs: DFATD & Independent Consultant DFATD Reviews project reports to assess progress against expected results. Ensures the implementer: is implementing the PMF and using both the output and outcome indicator data to management for results HQ or post staff may undertake site visits to validate reporting ensure local ownership. Participates in project governance bodies. May hire a monitor (independent consultant) Independent Consultant Useful for complex projects requiring the engagement of many actors. Useful in very technical projects that officers find difficult to assess. Two models: 1. Reports only to DFATD and 2. Reports to DFATD and the implementer: 16 The link with evaluation Monitoring is descriptive and continuous. It assesses whether change is happening, not why or why not. Evaluation is explanatory and punctual. It explains, for example, why targets and outcomes are, or are not, being achieved. Both require the same kind of critical thinking and analysis. A strong and well-applied monitoring system facilitates evaluation. Monitoring and evaluation should be planned together in order to complement each other. The cost of evaluation can be reduced substantially if monitoring information at outcomes is available 17 Theory of Change fundamental part RBM Every program is based on a theory of change a set of assumptions, risks and external factors that describes how and why the project is intended to work. This theory connects the projects activities with its expected ultimate outcome. In RBM the Theory of Change is the evidence-based story of cause-and-effect relationships and associated assumptions within a project. It is inherent in the project design and is often based on research, evaluations, best practices and lessons learned. Developing a theory of change combines a deep reflective process and analysis with the systematic mapping of the logical sequence of changes in a project from inputs to outcomes. It explains the expected outcomes, who will experience them, how they will be achieved, what assumptions are being made, and what external factors and risks may influence the achievement of those outcomes, all in reference to the projects complex and evolving context. 18 A Results Chain A results chain is a depiction of the causal or logical relationships between inputs, activities, outputs, and the outcomes of a given policy, program, or investment (project). (DFATD RBM Policy) The results chain addresses development practitioners need for a concept that allows them to break complex change down into manageable "building blocks" or steps related to each other by cause and effect, making change easier to engage with during both analysis and implementation. A results chain provides the conceptual framework for articulating that logical sequence of changes. Each organization will have its own results chain, which will depict and define the number and type of building blocks or levels it uses. These same steps also become the points at which practitioners will measure whether or not the expected change is actually occurring throughout project implementation. 19 DFATD Development Results Chain 20 The Logic Model and Output-Activity Matrix In sum, when development practitioners approach a specific problem, their respective results chain will provide a structure to their project design; telling them what types of building blocks they should be identifying as they work on their theory of change. Because change, particularly development change, is complex and multi-faceted, a full theory of change (and thus, logic model) rests on several complementary pathways combining to support one ultimate outcome. This is where the Logic Model comes in. At DFATD, the theory of change for a specific project is visually displayed in a set of tools - the logic model (LM), which shows the output and outcomes levels (NEW), and the outputs and activities matrix (NEW) which shows activities - and the ToC is fully explained in an accompanying narrative. DFATDs logic model and the outputs and activity matrix (OAM) templates align to DFATDs results-chain levels and display the logical sequence of key steps needed to bring about the changes the design team wish to achieve. 21 The LM combines a number of complementary approaches to a development issue into a pyramid under one ultimate outcome; different routes or pathways leading to the same destination. Each route addresses a different aspect of the problematic. The Logic Model (LM) An LM is a visual depiction of how we expect the project to work; the causal/logical connection between outputs and the outcomes to which they will contribute The LM, its supporting narrative (and the outputs and activities matrix [OAM]) capture the theory of change for a project The LM should not go over one page A LM is not static; it is an iterative tool. As the program changes, the LM should be revised to reflect the changes, but remain within the project scope. 22 A Logic Model Diagram & Flexibility 23 Activities-Output Matrix (OAM) Outputs and Activities Matrix Immediate Outcome 1110 Output 1111 Activity Activity Activity Activity Output 1112 Activity Activity Immediate Outcome 1120 Output 1121 Activity Activity Activity Output 1122 Activity Activity 24 Performance Measurement Framework (PMF) In order to monitor and evaluate your project, it is important to establish a structured plan for the collection and analysis of performance information during project design. Used to systematically plan the collection of relevant data to assess and demonstrate progress made in achieving expected results Ensures performance information is collected on a regular basis Allows for ongoing, evidence-based decision making Developed and monitored in consultation with partners, other donors, local stakeholders and sometimes beneficiaries The PMF is not a paper exercise 25 What is Reporting on Outcomes Expected results are clearly defined at the start of any intervention & form the focal point for any ongoing management decisions Implementation must continue to focus on achievement of expected results and taking corrective action as needed Reporting focuses on the comparison of the actual outcomes being achieved against expected outcomes established in the beginning of the project. How? Assess what is being achieved (actual outcomes) based on performance data (qualitative and quantitative) using indicators identified during planning & collected during monitoring and evaluation Not only how many schools were built, but also on the childrens improved knowledge Not only how many wells have been drilled, but also on citizens increased usage of safe drinking water Use data as evidence of progress Expected Results Actual Results Inputs, Activities, Outputs Corrective Actions 26 Why Report? Implementing Organizations: Reports inform DFATD of the progress of the project Reports demonstrate both what outcomes are being achieved & accountability to the DFATD contract or contribution agreement requirements Reports are an important management tool for the implementing agency used to: Communicate about project results/outcomes among other project stakeholders ; and Use performance information to make adjustments to project, through consultation and approval of the Project Steering Committee (PSC) and DFATD. 26 27 RBM is applied at every level DFATD manages for development results at micro level (project) meso level (program, bureau or branch) macro level (corporate) The underlying RBM concepts and principles are the same. However, the tools and methodologies are adapted to each level. 28 Results achieved mainly via projects, so project data is very important Corporate tools set the programming direction, butActual development results are only achieved through projects (or other work) in partner countries. DFATD collects data on these results mainly through project reports or project evaluations. No actual results data is produced at the meso or macro levels. Meso- and macro-level results data are simply roll-ups of micro data, i.e, project indicator data. This is why connecting the levels is so crucial (need a project-to-corporate link) We are currently strengthening our approach for this. Building an architecture that will link development results from micro to meso to macro, for both reporting and management at the different levels. Something that will help DFATD aggregate project-level results at a meso level (program, bureau or branch) and then aggregate those meso-level results at the corporate level better. 29 Horizontal Nesting Model Results: In 2009 former CIDA introduced a nesting model to structure the relationship between corporate, program and project level development results chains. In this model, the outcomes (ultimate, intermediate and immediate) are horizontally aligned across the corporate, program and project level. The types of expected changes at each level of the result chain remain the same, but their degree of specificity (what) and granularity (level of detail/who and where) increase as you move from corporate (broad) to project (specific). Indicators: A similar horizontal relationship exists between indicators at the corporate, program and project levels. However, with the exception of consolidated indicators, an indicator identified at corporate and program level could be the same as the one used at project level. 30 Nesting vs. Cascading: Snapshot The nesting model creates a parallel or horizontal relationship between macro, meso and micro level results chains. In other words, outcomes of the same type line up in parallel across levels, not cascading. UO INTO IMO not macro UO INTO IMO meso UO INTO IMO micro UO INTO IMO macro UO INTO IMO meso UO INTO IMO micro ParallelCascading Macro = corporate level; Meso = program, bureau, branch etc.; Micro = project Level UO = Ultimate Outcome; INTO = Intermediate Outcomes; IMO = Immediate Outcomes the red line represents the outcomes we expect to be achieved by the end of projects or programs 31 Cascading Result Example Intermediate Outcome: Improved health services for women, men, and boys and girls in target countries Ultimate Outcome: Improved health for women, men and boys and girls in Elbonia Intermediate Outcome: Improved management and delivery of health services by Elbonian MoH to men, women, boys and girls. Immediate Outcome: Increased capacity in health system management for Elbonian MoH. Ultimate Outcome: Increased well-being and empowerment of children and youth meso (Elbonia Program) Ultimate Outcome: Improved health for young mothers in selected rural communities of Region X Intermediate Outcome: Improved delivery of pre-, peri- and post-natal services to young mothers in Region X micro (Health Project in Elbonia) Immediate Outcome: Improved capacity of health service providers. macro (Corporate) Legend: Red line = The outcomes we expect to achieve by the end of project and program. However, progress on the ultimate outcomes should be monitored (when ever possible) during the life of a project or program, 32 Horizontal Nesting Results Example Intermediate Outcome: Improved health services for women, men, and boys and girls in target countries Ultimate Outcome: Improved health for women, men and boys and girls in Elbonia Intermediate Outcome: Improved management and delivery of health services by Elbonian MoH to men, women, boys and girls Immediate Outcome: Increased capacity in health system management for Elbonian MoH Ultimate Outcome: Increased well-being and empowerment of children and youth meso (Elbonia Program) Ultimate Outcome: Improved health for young mothers in selected rural communities of Region X Intermediate Outcome: Improved delivery of pre-, peri- and post-natal services to young mothers in Region X Immediate Outcome: Increased knowledge and skills in pre-, peri- and post natal care services among Midwives in Region X micro (Health Project in Elbonia) Immediate Outcome: Improved capacity of health service providers macro (DFATD) 33 Nesting within the results architecture Improved health services for mothers, newborns and children under 5 Improved health of women, men and children in Elbonia Improved management and delivery of national health services by Elbonian MoH to Elbonian men, women, boys and girls Increased capacity in health system management for Elbonian MoH. Country Program Results Chain Increased well-being and empowerment of children and youth Corporate Results macro meso Planning Flow Reporting Flow Improved health of young mothers in selected rural communities of Region X of Elbonia Improved delivery of pre-, peri- and post-natal services to young mothers in Region X. Increased knowledge and skills in pre-, peri- and post natal care services among Midwives in selected communities of Region X. One Projects' Results Chain micro Ultimate Outcomes Intermediate Outcomes Immediate Outcomes More granular statements of the meso ones and always at same level More granular statements of the macro ones and always at same level Broader statements at corporate Level 34 Indicators by Level of Result Chain Indicators can be classified according to what they measures in the result chain: Output indicators: Measure the products and services stemming from project activities and delivered by the implementer to the intermediaries or beneficiaries. Immediate Outcome Indicators: Measure the changes in skill, knowledge, abilities, or awareness of the intermediaries or beneficiaries as a consequence of the outputs. Intermediate Outcome Indicators: Measure the changes in behaviour, practice or performance of intermediaries or beneficiaries as a consequence of the immediate outcomes. Ultimate Outcomes Indicators: Measure the sustainable changes in the lives of beneficiaries as a consequence of the intermediate outcomes. 35 Nesting Advantages: Data Rollup and Reporting The nesting model helps with the roll up of outcome (and output) indicators from project to corporate level. Why? Each outcome level captures a different kind of change in a result chain and thus uses indicators that measure that specific change. When rolling up data it is easier to compare and aggregate apples with apples than apples with carrots. UO INTO IMO macro UO INTO IMO meso UO INTO IMO micro Ghana Project Portfolio Ghana Represents indicators and their data Increasing Reduction & Simplification of Indicators & their Data Total Quantity of Indicators and their Data Master Slide EDRMS No Adapted from diagram found in:Indicators & their Data for Project Managers in DFATD & in Executing Agencies & Specialists Indicators & their Data for the Program Manager and Specialists Indicators & their Data for Corporate Needs (Reporting to Public & Parliament) & for Multi donor work Diagram illustrating different requirement for indicators & their data at different levels (Simplified) 37 macro micro meso consolidated indicator (only meso and macro) Fewer indicators required Pyramid of Indicators: Intermediate Outcome Indicator Example indicator UO INTO IMO macro UO INTO IMO meso UO INTO IMO micro Source: Master Slide EDRMS No Adapted from diagram found in:Meso = bridge between macro and micro 38 What are Consolidated Indicators? Consolidated indicators are created by DFATD from project level indicators that are clustered together with other related indicators (a family of indicators) to form one indicator that will allow aggregation (summarise) related data at the program, branch or corporate level. Example No. 1 project 1: #/total of nurses (F/M) passing certification exams project 2: #/total of doctors (F/M) granted membership in college of physicians project 3: #/total of employees (F/M) in rural health clinics who passed training courses on health information systems Consolidated program-level indicator: #/total of qualified health professionals (F/M) in Elbonia 39 The program-level PMF What does the program PMF measure? The performance of the DFATD program in the country Outcomes (in country / region) to which your program contributes The PMF is the backbone of programs monitoring framework or system? It facilitates data collection to allow analyst to measure and assess performance of the programs projects in the country The program PMF does not measure The total sum of portfolio of projects The performance of the country or region Outcomes in the country or region to which we are not making a contribution 40 Selecting indicators for program PMF Most indicators & data will be drawn from project-level monitoring and reports. Program Analysts (PA) conduct project portfolio analysis and strategically select indicators from portfolio suitable for program level (ongoing) If needed, program analysts identify other indicators not tracked by projects that help them manage & tell performance story Criteria for selection, for example, are any of the project indicators: Indicators from a country national development plan Relevant Program Alignment Architecture/Corporate Performance Measurement Framework outcome indicators Key performance indicators (KPIs) or family members for consolidated KPIs, if relevant. Indicators tracked by multiple projects in the portfolio (that reflect depth of program projects) Indicators that could be grouped into a family of indicators & made into a DFATD consolidated indicators Every indicator selected must measure some aspect of program outcomes (has to be direct relationship) If any of the above indicators appear in the projects, they should be included in the PMF When using indicators tracked by systems in host country, ensure that: a)indicators measure outcomes to which DFATD programming contributes (PBAs, SWAPs, multi-donor projects, etc) and b)indicators are valid measure of outcome in program LM 41 To Conclude For DFATD to report at program level, we need actual data from the projects This means that every time DFATD partners report on their projects outcomes they should: provide actual data for each indicator in PMF according to frequency specified for each outcome (and output), provide evidence-based narrative describing progress made on or toward achieving outcome (and output) using data collected on indicators in the PMF explain any variance Performance data, collected through performance monitoring, is essential for both reporting and for results-based management at project & program level THANK YOU ANY QUESTIONS? INTERMEDIATE OUTCOMES 1000 Improved health of women, men, girls and boys living in community X 1100 Increased proper usage of safe drinking water by women, men, girls and boys in community X Improved management of water, waste and sanitation infrastructure in community X 1110 Increased equitable access to safe drinking water for women, men, girls and boys living in community X 1120 Increased knowledge and awareness of the importance and proper usage of safe drinking water among women, men, girls and boys in community X 1210 Increased ability of women in community X to maintain wells 1220 Increased knowledge and skills in waste management and sanitation among male and female local community development committee members 1111 Wells built in community X, in consultation with local stakeholders, especially women as primary water managers in the community 1112 Existing wells rehabilitated in community X 1121 Awareness material, including material appropriate for a non-literate audience, developed in consultation with male and female community members 1122 Awareness campaigns conducted on the importance and proper usage of safe drinking water for women, men, girls and boys in community X 1211 Training provided to women in community X on maintenance of wells 1212 xxxxxxxxx 1221 TA in waste management and sanitation provided to local community development committees members, both women and men xxxxxxxxxxxxx IMMEDIATE OUTCOMES ULTIMATE OUTCOME OUTPUTS LM Example 44 Elements of a PMF Result Statements from Logic Model IndicatorsBaselineTargetData Sources Collection Methods FrequencyResponsibility Ultimate Outcome Intermediate Outcomes Immediate Outcomes Outputs 45 Data Collection Methods Source: Ray Rist, 10 Steps to an RB M&E System World Bank, 2004, page 85