challenges to institutionalizing impact evaluation

Challenges to Institutionalizing Impact

Evaluation

Discussion Points* presented at the International Conference on Impact

EvaluationCairo, Egypt

March 28- April2, 2009

*By Getahun Tafesse, CIDA –ECCO M&E Advisor

Opportunities

• Enormous development challenges in Ethiopia - widespread poverty, chronic food insecurity, HIV/AIDs, illiteracy, environmental degradation etc.

• This provides fertile ground for evaluation as intervention resources are limited and different stakeholders are pressurized to demonstrate results

• The Government of Ethiopia (GoE) policy statements express commitment to poverty reduction and sustainable development.

Strengths and Challenges to Institutionalizing Impact Evaluation

Opportunities

• Ethiopia has also formally adapted the MDGs as an overarching development guiding framework

• The existence of well established sectoral annual review mechanisms

• The pool of M&E professionals in the country is gradually increasing as more and more organizations are currently recruiting experts specifically assigned as such


Opportunities

• Recent trends show growing recognition and institutionalization of monitoring and evaluation practices:

• A growing appreciation of evaluation across different stakeholders and particularly encouragement of participatory evaluation at different levels

• Institutionalization of M&E in non-government circles• Increasing conduct of M&E training by organizations• Incorporation of M&E courses in some training programs


Actors involved in M &E

Actors involved in Evaluation

• The Government of Ethiopia (GoE)– National – MOFED, PMO, NBE– Sector ministries– Regional and Woreda administrations

• Donors (through projects/ programs)– Bilateral– Multilateral

• INGOs/ NGOs (through projects/ programs)– Relief and Development– Advocacy– Professional associations

• Academic and Research Institutions• The Private Sector/ business community• Communities

Ministry of Finance & Economic Development

Sector Ministries

Donors INGOs/NGOs

Welfare Monitoring Unit

Central Statistical Authority

Regional Bureaus

Woreda DesksFacilities

Grass Root Communities

Programs

Projects

Branch Offices

Projects

Branch Offices

Surveys/Censuses

Participatory Poverty Assessments

National Welfare Monitoring Steering

Committee

Central Statistical Authority

Welfare Monitoring Unit

Same as Above

Chair

Data Users

Ministry of Finance and Economic Development

INGOs/ NGOs

National Welfare Monitoring Technical Advisory Committee

Welfare Monitoring System in Ethiopia

Ministry of Agriculture and Rural Dev’t

Ministry of Health

Ministry of Labour and Social Affairs

Ministry of Women Affairs

Ministry of Education

Member

Member

Member

Member

Member

HHICES

Welfare Monitoring Survey

Sector Reports

PPAs

RCBP interest could be reflected here

Ethiopian Road Authority Member

The Status Quo in Impact Evaluation• Impact Evaluation is not a common practice

• Some sorts of Evaluation are practiced mostly to fulfil donor requirements

Can Impact Evaluation meet the sense of urgency that characterize the need for development intervention in low income countries?

• Frequently development needs require urgent/ immediate assistance

• Programming in developing countries is geared towards fulfilment of basic needs

• Choice of area of Intervention is hardly questionable

* Difficult to make the main thrust of development programming as Experimentation

The Status Quo in Impact Evaluation

Is there national ownership in Impact Evaluation?

• Impact Evaluation implies there is a preconceived desired State

• Whose definition of ‘ development ‘ matters

• If desired state is imposed, ownership to IE is lost

The Status Quo in Impact EvaluationIs the Cure for development problems known?• Good governance – participation,

accountability, transparency…• Investment on education, health,

agriculture, road…• Equity, peace, security…• Ownership, partnership, harmonization…

*But the degree of effectiveness of a specific programming in these areas is not easily known

The Status Quo in Impact EvaluationThe Impact evaluation Dilemma• There is sense of urgency for development

programming• Appropriate cure depends on Impact

evaluation/ experimentation

So, the focus of Impact Evaluation in LDCs should be– Not on identification of Areas of

investment– Rather on methods of deliveryThe Cure is known (the vaccine is identified).

The main question is how to deliver best.


Methodological ChallengesHow to make the Case of IE strong?• Demonstrating practical benefits of Impact

Evaluation• Resources are scarce – how much to spend

on something that is intuitively known is good• Flexible and easy IE techniques/ methods

that are less costly in terms of time and resources

• IE aimed at guiding/ improving decision making– Show possible alternative use of resources

The Status Quo in Impact EvaluationRecommendations• Make Impact Evaluation a parallel endeavour/ not a

major thrust of development programming – experimentation on a small scale

• Impact evaluation on key programs without disturbing regular programming, i.e., without creating discontinuity in program

• Impact Evaluation as a Second Stage Experimentation with the focus being on Approach to delivery, exploring practical alternatives

• Impact Evaluation in developing countries should have a strong component of comparison element among known strategies

• From the supply side, effort should be made to expand the availability of easy and flexible tools of IE that are less costly in terms of both time and resources


Recommendations• Impact Evaluation in developing countries

should have a strong component of comparison element among known strategies – not simple and full focus on the specific program that is the subject of evaluation

• IE should be designed to aide decision making on ‘so what’

• From the supply side, effort should be made to expand the availability of easy and flexible tools of IE that are less costly in terms of both time and resources


Strengths• GoE national Reports have greatly been improved in quality

(depth of analyses) and coverage (sectors, sub-sectors, themes) due to

Growing demand for and use of such reports DAG financial and technical support extended to MoFED & use of

professionals (consultants) for data analyses and report production Inclusion of Governance, foreign Aid, MDGs, Environment, Gender,

etc. themes in the reports although not to sufficient degree Implementation of regular household surveys (HHICE, WMS, DHS) Improved capacity and performance on the part of CSA Improved capacity and performance on the part of selected

sectoral line ministries

• Great improvements in sectoral reports, especially Health and Education aided by

Sector Management Information Systems Annual Review Mechanisms High political commitment (Health, for example) Improved standardization, rationalization an harmonization of

indicators, data collection and reporting procedures

Progresses• Strong and growing capacity in statistical data collection

CSA’s impressive data collection program Annual and periodical regular surveys Long experience and institutional capacity

• Civil service reform including business process reengineering (BPR)

Streamlined tasks and responsibilities Result oriented work planning

• Good practice of contracting out data collection and analyses

Supported by growing # and capacity of private consultants


Opportunities

• Strong culture of collaboration by beneficiaries in responding to studies and gradual development in their level of active participation and articulate responses


Opportunities

• Availability of administrative data by sector ministries (eg. education, health etc), and survey data by Central Statistical Agency

• Cooperation among stakeholders (government and non-government alike) to share data available

• Structured societal organizations established down to small community level.

• Some level of established practice in using evaluation for planning purposes…


Challenges• Lack of informed debate on local development

perspectives and relevant evaluation conceptual frameworks and approaches; which leads to:

• Lack of consensus on development concept and measurement criteria

• Lack of consensus on evaluation concept and criteria• Uniform application of evaluation techniques and lack of

adoption to specific cultural and behavioral contexts


Limitations/ Challenges

• Widespread traditional management practice that gives focus to counting activities and outputs rather than focus to assessing higher level results, i.e., poor result based management practice. This is particularly reflected:

• In absence of baseline data• Poor feedback mechanism• Lack of informed decision making or poor

linkage between assessment and decision making

Challenges• Data collection, analyses and

reporting aimed at demonstrating achievements and less focus given to analyses of constraints and challenges

• Significant discrepancies between administrative and survey data


Limitations• Lack of linkage/ integration across

different sectoral M&E systems• Different timing• Different level of reporting• duplication

• sectors at varying stages of capacity and performance in evaluation practice

• poor practice of verification methods/ triangulation of data from different sources


Limitations

• Generating Compelling and evidence based results attributable to programming

• Large number and varying quality of indicators used in GoE reports and challenges to discern overall progress

• Missing reference comparisons or lack of standard reference points

• Missing indicators w.r.t. gender, disadvantaged groups (disabled, destitute, etc.)

Limitations/ Challenges Routine data collection at lower level

cumbersome and unsystematic Simplifying data collection & ensuring timely

use of data The progressive data aggregation at

each higher level not necessarily conducive to data analyses

No systematic integration of national data collection activities The need to support national strategy for the

development of statistics Integration of data collection systems (within

& outside sectors)

Limitations/ Challenges Poor feed back and linkage with

planning and decision making

• weak data producer and users common forums and linkages

Limitations/ Challenges• Lack of standardization of survey

methods, definition of indicators and measurement tools.

• Poor recognition given to evaluation importance as reflected by:

• Poor integration of evaluation approach in programs/ projects design

• Lack of earmarked budget for M&E• Lack of earmarked human resource for

M&E in established structures• M&E task usually undertaken as an add-on

task

Limitations Generation of data disaggregated

at woreda level– Different levels of reporting across different

sectors

No regular complementary qualitative information (PPA, citizen card, etc.)/ public opinion

•


Poor Maximizing of benefits from data analyses

– Gender disaggregated data– Thorough/ in depth analyses of data sets– Timely analyses of data

• Inadequate practice and capacity to review and enforce good ethical standards in the undertaking of evaluation

• Teaching in evaluation not well recognized and integrated in curricula of different disciplines

Limitations/ Challenges Poor institutional capacity

especially at lower levels Fragile and over loaded No earmarked human and financial

resource – M&E is usually an add-on task

High staff turn-over


• Lack of agency/ home, networks for

• developing and disseminating knowledge on evaluation approaches, tools, and best practices

• adoption and popularization of internationally set goals and commitments (ex., MDGs, Conventions, Declarations )

• adoption and popularization of established methodologies

• sharing of experiences and exchanging ideas

Food SecurityMonitoring & Evaluation

GoE Food Security M&E System• The FSCB has an overall

responsibility for programme M&E • Food Security Programs M&E Plan

• different stakeholders were involved in the process & many acquired opportunity to comment

• Simple and Practical• 4 Principles were applied:

– Simplicity –vs- Utility– Process –vs- Product/ outcome– Decentralization –vs- Accountability– Participation –vs- Rigour

GoE Food Security M&E System• Food Security Programs M&E Plan

– Result Frameworks (recently revised based on two 2-day workshops involving GoE & Donors)

– Monitoring Formats• Activity Reports (from community up to federal)• Quarterly financial and procurement reports

developed by FSCB in accordance with GoE accounting procedures and PSNP Procurement Guidelines.

– Focal persons responsible at different levels Training given to focal persons on Monitoring

Formats

GoE Food Security M&E System

• Food Security Programs M&E Plan– Programme Description– FSP Logical Framework– M&E System: Objectives and Approaches– Institutional Roles and Information Flow– Monitoring Guidelines and Methods– Evaluation Guidelines and Methods– Human Resource Needs and Training Plan– Reporting Formats

GoE Food Security M&E System• Monitoring and Evaluation Technical

Task Force– Comprises members from GoE and Donors

financing food security programs– Meets every two-weeks– Oversees the implementation of the M&E

plan– Reviews study designs and mobilizes the

necessary technical assistance and capacity building resources

GoE Food Security M&E System• JCC - meetings every fortnight to discuss

and decide on various issues related to the implementation of the program (resource flows to the beneficiary, targeting issues, capacity building and other pertinent issues as they arise).

• Rapid Response Mechanism field monitoring (usually with both FSCB and donor representatives) undertaken on monthly basis to examine ad hoc issues and constraints as they arise. Team reports identifying issues and recommendations are presented to the JCC for consideration.

GoE Food Security M&E System• An Information Centre has been set up and staffed

in the FSCB to regularly follow up on problems at woreda and regional levels (particularly with respect to the flow of funds to woredas and payments to beneficiaries).

• Joint Implementation Support Missions are conducted twice each year (May and October) to review progress with program implementation per se and with related capacity building actions (e.g., financial management, public works, woreda planning and quality control, etc.). The FSCB presents financial and technical progress reports during the missions, as well as reports prepared by the RRM Teams. Additional reports may be produced as required, including reviews of procurement.


• Surveys and Studies– Baseline Survey– Annual Survey– Public Work Reviews– Food Aid Assessments– Other specific studies

• Targeting• Institutional Assessment • Impact


Other data sources:• Annual Agricultural Statistics

– Crop forecast, Actual crop assessment– Woreda level Ethiopian Agricultural

Sample Enumeration (EASE) • Agricultural Census• DPPA Early Warning System• Vulnerability Profiles

challenges to institutionalizing impact evaluation

Documents