challenges to institutionalizing impact evaluation
Post on 01-Feb-2016
31 Views
Preview:
DESCRIPTION
TRANSCRIPT
Challenges to Institutionalizing Impact
Evaluation
Discussion Points* presented at the International Conference on Impact
EvaluationCairo, Egypt
March 28- April2, 2009
*By Getahun Tafesse, CIDA –ECCO M&E Advisor
Opportunities
• Enormous development challenges in Ethiopia - widespread poverty, chronic food insecurity, HIV/AIDs, illiteracy, environmental degradation etc.
• This provides fertile ground for evaluation as intervention resources are limited and different stakeholders are pressurized to demonstrate results
• The Government of Ethiopia (GoE) policy statements express commitment to poverty reduction and sustainable development.
Strengths and Challenges to Institutionalizing Impact Evaluation
Opportunities
• Ethiopia has also formally adapted the MDGs as an overarching development guiding framework
• The existence of well established sectoral annual review mechanisms
• The pool of M&E professionals in the country is gradually increasing as more and more organizations are currently recruiting experts specifically assigned as such
Strengths and Challenges to Institutionalizing Impact Evaluation
Opportunities
• Recent trends show growing recognition and institutionalization of monitoring and evaluation practices:
• A growing appreciation of evaluation across different stakeholders and particularly encouragement of participatory evaluation at different levels
• Institutionalization of M&E in non-government circles• Increasing conduct of M&E training by organizations• Incorporation of M&E courses in some training programs
Strengths and Challenges to Institutionalizing Impact Evaluation
Actors involved in M &E
Actors involved in Evaluation
• The Government of Ethiopia (GoE)– National – MOFED, PMO, NBE– Sector ministries– Regional and Woreda administrations
• Donors (through projects/ programs)– Bilateral– Multilateral
• INGOs/ NGOs (through projects/ programs)– Relief and Development– Advocacy– Professional associations
• Academic and Research Institutions• The Private Sector/ business community• Communities
Ministry of Finance & Economic Development
Sector Ministries
Donors INGOs/NGOs
Welfare Monitoring Unit
Central Statistical Authority
Regional Bureaus
Woreda DesksFacilities
Grass Root Communities
Programs
Projects
Branch Offices
Projects
Branch Offices
Surveys/Censuses
Participatory Poverty Assessments
National Welfare Monitoring Steering
Committee
Central Statistical Authority
Welfare Monitoring Unit
Same as Above
Chair
Data Users
Ministry of Finance and Economic Development
INGOs/ NGOs
National Welfare Monitoring Technical Advisory Committee
Welfare Monitoring System in Ethiopia
Ministry of Agriculture and Rural Dev’t
Ministry of Health
Ministry of Labour and Social Affairs
Ministry of Women Affairs
Ministry of Education
Member
Member
Member
Member
Member
HHICES
Welfare Monitoring Survey
Sector Reports
PPAs
RCBP interest could be reflected here
Ethiopian Road Authority Member
The Status Quo in Impact Evaluation• Impact Evaluation is not a common practice
• Some sorts of Evaluation are practiced mostly to fulfil donor requirements
Can Impact Evaluation meet the sense of urgency that characterize the need for development intervention in low income countries?
• Frequently development needs require urgent/ immediate assistance
• Programming in developing countries is geared towards fulfilment of basic needs
• Choice of area of Intervention is hardly questionable
* Difficult to make the main thrust of development programming as Experimentation
The Status Quo in Impact Evaluation
Is there national ownership in Impact Evaluation?
• Impact Evaluation implies there is a preconceived desired State
• Whose definition of ‘ development ‘ matters
• If desired state is imposed, ownership to IE is lost
The Status Quo in Impact EvaluationIs the Cure for development problems known?• Good governance – participation,
accountability, transparency…• Investment on education, health,
agriculture, road…• Equity, peace, security…• Ownership, partnership, harmonization…
*But the degree of effectiveness of a specific programming in these areas is not easily known
The Status Quo in Impact EvaluationThe Impact evaluation Dilemma• There is sense of urgency for development
programming• Appropriate cure depends on Impact
evaluation/ experimentation
So, the focus of Impact Evaluation in LDCs should be– Not on identification of Areas of
investment– Rather on methods of deliveryThe Cure is known (the vaccine is identified).
The main question is how to deliver best.
The Status Quo in Impact Evaluation
Methodological ChallengesHow to make the Case of IE strong?• Demonstrating practical benefits of Impact
Evaluation• Resources are scarce – how much to spend
on something that is intuitively known is good• Flexible and easy IE techniques/ methods
that are less costly in terms of time and resources
• IE aimed at guiding/ improving decision making– Show possible alternative use of resources
The Status Quo in Impact EvaluationRecommendations• Make Impact Evaluation a parallel endeavour/ not a
major thrust of development programming – experimentation on a small scale
• Impact evaluation on key programs without disturbing regular programming, i.e., without creating discontinuity in program
• Impact Evaluation as a Second Stage Experimentation with the focus being on Approach to delivery, exploring practical alternatives
• Impact Evaluation in developing countries should have a strong component of comparison element among known strategies
• From the supply side, effort should be made to expand the availability of easy and flexible tools of IE that are less costly in terms of both time and resources
The Status Quo in Impact Evaluation
Recommendations• Impact Evaluation in developing countries
should have a strong component of comparison element among known strategies – not simple and full focus on the specific program that is the subject of evaluation
• IE should be designed to aide decision making on ‘so what’
• From the supply side, effort should be made to expand the availability of easy and flexible tools of IE that are less costly in terms of both time and resources
Strengths and Challenges to Institutionalizing Impact Evaluation
Strengths• GoE national Reports have greatly been improved in quality
(depth of analyses) and coverage (sectors, sub-sectors, themes) due to
Growing demand for and use of such reports DAG financial and technical support extended to MoFED & use of
professionals (consultants) for data analyses and report production Inclusion of Governance, foreign Aid, MDGs, Environment, Gender,
etc. themes in the reports although not to sufficient degree Implementation of regular household surveys (HHICE, WMS, DHS) Improved capacity and performance on the part of CSA Improved capacity and performance on the part of selected
sectoral line ministries
• Great improvements in sectoral reports, especially Health and Education aided by
Sector Management Information Systems Annual Review Mechanisms High political commitment (Health, for example) Improved standardization, rationalization an harmonization of
indicators, data collection and reporting procedures
Progresses• Strong and growing capacity in statistical data collection
CSA’s impressive data collection program Annual and periodical regular surveys Long experience and institutional capacity
• Civil service reform including business process reengineering (BPR)
Streamlined tasks and responsibilities Result oriented work planning
• Good practice of contracting out data collection and analyses
Supported by growing # and capacity of private consultants
Strengths and Challenges to Institutionalizing Impact Evaluation
Opportunities
• Strong culture of collaboration by beneficiaries in responding to studies and gradual development in their level of active participation and articulate responses
Strengths and Challenges to Institutionalizing Impact Evaluation
Opportunities
• Availability of administrative data by sector ministries (eg. education, health etc), and survey data by Central Statistical Agency
• Cooperation among stakeholders (government and non-government alike) to share data available
• Structured societal organizations established down to small community level.
• Some level of established practice in using evaluation for planning purposes…
Strengths and Challenges to Institutionalizing Impact Evaluation
Challenges• Lack of informed debate on local development
perspectives and relevant evaluation conceptual frameworks and approaches; which leads to:
• Lack of consensus on development concept and measurement criteria
• Lack of consensus on evaluation concept and criteria• Uniform application of evaluation techniques and lack of
adoption to specific cultural and behavioral contexts
Strengths and Challenges to Institutionalizing Impact Evaluation
Limitations/ Challenges
• Widespread traditional management practice that gives focus to counting activities and outputs rather than focus to assessing higher level results, i.e., poor result based management practice. This is particularly reflected:
• In absence of baseline data• Poor feedback mechanism• Lack of informed decision making or poor
linkage between assessment and decision making
Challenges• Data collection, analyses and
reporting aimed at demonstrating achievements and less focus given to analyses of constraints and challenges
• Significant discrepancies between administrative and survey data
Strengths and Challenges to Institutionalizing Impact Evaluation
Limitations• Lack of linkage/ integration across
different sectoral M&E systems• Different timing• Different level of reporting• duplication
• sectors at varying stages of capacity and performance in evaluation practice
• poor practice of verification methods/ triangulation of data from different sources
Strengths and Challenges to Institutionalizing Impact Evaluation
Limitations
• Generating Compelling and evidence based results attributable to programming
• Large number and varying quality of indicators used in GoE reports and challenges to discern overall progress
• Missing reference comparisons or lack of standard reference points
• Missing indicators w.r.t. gender, disadvantaged groups (disabled, destitute, etc.)
Limitations/ Challenges Routine data collection at lower level
cumbersome and unsystematic Simplifying data collection & ensuring timely
use of data The progressive data aggregation at
each higher level not necessarily conducive to data analyses
No systematic integration of national data collection activities The need to support national strategy for the
development of statistics Integration of data collection systems (within
& outside sectors)
Limitations/ Challenges Poor feed back and linkage with
planning and decision making
• weak data producer and users common forums and linkages
Limitations/ Challenges• Lack of standardization of survey
methods, definition of indicators and measurement tools.
• Poor recognition given to evaluation importance as reflected by:
• Poor integration of evaluation approach in programs/ projects design
• Lack of earmarked budget for M&E• Lack of earmarked human resource for
M&E in established structures• M&E task usually undertaken as an add-on
task
Limitations Generation of data disaggregated
at woreda level– Different levels of reporting across different
sectors
No regular complementary qualitative information (PPA, citizen card, etc.)/ public opinion
•
Limitations/ Challenges
Poor Maximizing of benefits from data analyses
– Gender disaggregated data– Thorough/ in depth analyses of data sets– Timely analyses of data
• Inadequate practice and capacity to review and enforce good ethical standards in the undertaking of evaluation
• Teaching in evaluation not well recognized and integrated in curricula of different disciplines
Limitations/ Challenges Poor institutional capacity
especially at lower levels Fragile and over loaded No earmarked human and financial
resource – M&E is usually an add-on task
High staff turn-over
Limitations/ Challenges
• Lack of agency/ home, networks for
• developing and disseminating knowledge on evaluation approaches, tools, and best practices
• adoption and popularization of internationally set goals and commitments (ex., MDGs, Conventions, Declarations )
• adoption and popularization of established methodologies
• sharing of experiences and exchanging ideas
Food SecurityMonitoring & Evaluation
GoE Food Security M&E System• The FSCB has an overall
responsibility for programme M&E • Food Security Programs M&E Plan
• different stakeholders were involved in the process & many acquired opportunity to comment
• Simple and Practical• 4 Principles were applied:
– Simplicity –vs- Utility– Process –vs- Product/ outcome– Decentralization –vs- Accountability– Participation –vs- Rigour
GoE Food Security M&E System• Food Security Programs M&E Plan
– Result Frameworks (recently revised based on two 2-day workshops involving GoE & Donors)
– Monitoring Formats• Activity Reports (from community up to federal)• Quarterly financial and procurement reports
developed by FSCB in accordance with GoE accounting procedures and PSNP Procurement Guidelines.
– Focal persons responsible at different levels Training given to focal persons on Monitoring
Formats
GoE Food Security M&E System
• Food Security Programs M&E Plan– Programme Description– FSP Logical Framework– M&E System: Objectives and Approaches– Institutional Roles and Information Flow– Monitoring Guidelines and Methods– Evaluation Guidelines and Methods– Human Resource Needs and Training Plan– Reporting Formats
GoE Food Security M&E System• Monitoring and Evaluation Technical
Task Force– Comprises members from GoE and Donors
financing food security programs– Meets every two-weeks– Oversees the implementation of the M&E
plan– Reviews study designs and mobilizes the
necessary technical assistance and capacity building resources
GoE Food Security M&E System• JCC - meetings every fortnight to discuss
and decide on various issues related to the implementation of the program (resource flows to the beneficiary, targeting issues, capacity building and other pertinent issues as they arise).
• Rapid Response Mechanism field monitoring (usually with both FSCB and donor representatives) undertaken on monthly basis to examine ad hoc issues and constraints as they arise. Team reports identifying issues and recommendations are presented to the JCC for consideration.
GoE Food Security M&E System• An Information Centre has been set up and staffed
in the FSCB to regularly follow up on problems at woreda and regional levels (particularly with respect to the flow of funds to woredas and payments to beneficiaries).
• Joint Implementation Support Missions are conducted twice each year (May and October) to review progress with program implementation per se and with related capacity building actions (e.g., financial management, public works, woreda planning and quality control, etc.). The FSCB presents financial and technical progress reports during the missions, as well as reports prepared by the RRM Teams. Additional reports may be produced as required, including reviews of procurement.
GoE Food Security M&E System
• Surveys and Studies– Baseline Survey– Annual Survey– Public Work Reviews– Food Aid Assessments– Other specific studies
• Targeting• Institutional Assessment • Impact
GoE Food Security M&E System
Other data sources:• Annual Agricultural Statistics
– Crop forecast, Actual crop assessment– Woreda level Ethiopian Agricultural
Sample Enumeration (EASE) • Agricultural Census• DPPA Early Warning System• Vulnerability Profiles
top related