inception report - international fund for agricultural

Independent External Evaluation of IFAD

Inception Report

Final Version

Submitted by

ITAD Ltd

1 March 2004

Independent External Evaluation of IFAD Inception Report

Page i

Acronyms and Abbreviations

AfDB African Development Bank AFESD Arab Fund for Economic and Social Development AsDB Asian Development Bank BCIE Central American Bank for Economic Integration BOAD West African Development Bank CAF Andean Development Corporation CDB Caribbean Development Bank CI Co-operating Institution COSOP Country Strategic Opportunities Paper CPM Country Portfolio Manager CSO Civil society organizations CWIQ Core Welfare Indicators Questionnaire DS Desk Study ERT External Review Team (External Review of IFAD in 2002) FS Field Study GFATM Global Fund Against AIDS, TB and Malaria HIPC Highly Indebted Poor Countries IEE Independent External Evaluation of IFAD IFI International Financial Institution MDB Multilateral Development Bank MDG Millennium Development Goal MFI Microfinance Institution MIS Management Information System MTR Mid Term Review o&m Operation and maintenance OE Office of Evaluation OSC Operational Strategy and Policy Guidance Committee PCR Project Completion Report PM Project Manager (in-country) PPP Purchasing Power Parity PRSP Poverty Reduction Strategy Paper PSR Project Status Report QA Quality Assurance REA Rapid External Assessment (of IFAD in 1994) RRP Report and Recommendation of the President SF IFAD Strategic Framework 2002-2006 TAG Technical Assistance Grant UNOPS UN Office for Project Services WDR World Development Report (World Bank annual publication) WFP World Food Programme


Page ii

Table of Contents

ACRONYMS AND ABBREVIATIONS......................................................................................................................I TABLE OF CONTENTS.......................................................................................................................................... II ACKNOWLEDGEMENTS......................................................................................................................................III

SUMMARY.............................................................................................................................................................V

1 INTRODUCTION ........................................................................................................................................1

THE INDEPENDENT EXTERNAL EVALUATION...................................................................................................1 OBJECTIVES OF THE EVALUATION......................................................................................................................1 PREVIOUS EXTERNAL ASSESSMENT AND REVIEW ............................................................................................2 STRUCTURE OF THE REPORT................................................................................................................................2

2 IFAD – POLICY AND STRATEGY IN A CHANGING ENVIRONMENT ...............................3

THE CHANGING AGRICULTURAL AND POVERTY CONTEXT .............................................................................3 IFAD’S MANDATE...............................................................................................................................................5 EVOLUTION OF IFAD’S STRATEGY ...................................................................................................................6 EVOLUTION OF STRATEGIES IN OTHER DEVELOPMENT AGENCIES.................................................................7 THE INTERNATIONAL CONSENSUS....................................................................................................................11 IMPLICATIONS FOR IFAD OF THE CHANGING CONTEXT AND STRATEGIC ORIENTATION.........................13

3 EVALUATION FRAMEWORK AND QUESTIONS ..................................................................... 14

EVALUATION ISSUES ..........................................................................................................................................14 FUNCTIONAL MODEL OF IFAD PROCESSES.....................................................................................................18 EVALUATION FRAMEWORK...............................................................................................................................20 EVALUATION OF DEVELOPMENT IMPACT ........................................................................................................23 EVALUATION OF CORPORATE PERFORMANCE ................................................................................................25

4 METHODOLOGY .................................................................................................................................... 33

SAMPLING OF COUNTRIES AND PROJECTS.......................................................................................................33 DESK STUDY........................................................................................................................................................33 COUNTRY VISITS.................................................................................................................................................36 ANALYSIS OF DATA............................................................................................................................................40 REPORTS: DESK, COUNTRY, REGIONAL AND FINAL.......................................................................................40 QUALITY ASSURANCE ........................................................................................................................................44

5 WORKPLAN .............................................................................................................................................. 44

TIMETABLE WITH ROLES AND PLANNED INPUTS............................................................................................44 ESTIMATED LEVEL OF EFFORT ..........................................................................................................................44

ANNEX 1 BIBLIOGRAPHY .......................................................................................................................... 50

ANNEX 2 SAMPLING OF COUNTRIES, PROJECTS AND GRANTS........................................... 51

POPULATION TO BE STUDIED.............................................................................................................................51 REGION.................................................................................................................................................................51 COUNTRY.............................................................................................................................................................52 SELECTION FOR THE COUNTRY VISITS.............................................................................................................53 PROJECTS.............................................................................................................................................................53 TECHNICAL ASSISTANCE GRANTS...................................................................................................................53

ANNEX 3 COSOP REVIEW PROFORMA ................................................................................................ 57

ANNEX 4 METHODOLOGY, DATA SOURCES AND DRAFT FORMATS FOR PROJECT-RELATED INFORMATION........................................................................................................................... 59

DESK STUDY SOURCES OF INFORMATION........................................................................................................59 FIELD STUDY SOURCES OF INFORMATION .......................................................................................................60 DRAFT SUMMARY FORM 1 DESK STUDY: PROJECT AND GRANT REVIEW.............................................63


Page iii

DRAFT SUMMARY FORM 2 FIELD STUDY: PROJECT MANAGEMENT INTERVIEWS.................................69 FIELD STUDY DRAFT METHODOLOGY FOR ESTIMATING BENEFICIARY NUMBERS..................................75 DRAFT SUMMARY FORM 3 FIELD STUDY: PARTNER INTERVIEWS...........................................................77 DRAFT SUMMARY FORM 4 FIELD STUDY: PROJECT BENEFICIARIES........................................................80 DRAFT SUMMARY FORM 5 OVERALL 12-POINT PROJECT ASSESSMENT..................................................84

Acknowledgements

This report was written during the Inception Phase of the Independent External Evaluation of IFAD, between 26 November 2003 and 19 January 2004. It was revised following a review by the IEE Steering Committee, Senior Independent Advisers and IFAD Senior Management. The principal authors were Derek Poate, Ian Hill, Andy Batkin, John Rowley and Mary Underwood. Additional contributions on policy analysis were made by Alison Evans, and on sampling and data processing by Neil Chalmers.


Page iv


Page v

Summary

Objectives The objectives of the evaluation highlight three key elements: to evaluate development results and impact in regard to rural poverty reduction; to evaluate the corporate learning process and performance of policy and operations; and make forward looking recommendations to improve future performance.

Evaluation of impact concerns the economic, institutional and social change arising from IFAD’s policies, projects and technical assistance grants. Evaluation of corporate performance concerns the relevance, efficiency and effectiveness of the processes and mechanisms by which IFAD develops policies and strategies, and identifies, designs, and implements project loans and grants. A functional model of the role of corporate processes is shown in Figure 1, page 19.

Changing agricultural and poverty context Comparison of the evolution of strategies adopted by IFAD and the other development agencies shows that in the 1980s IFAD was unique in its focus on the poor. Now, however, poverty reduction is a priority, if not the main focus of all the main development agencies. Measures to reduce poverty, such as equitable growth, improved access to assets and markets, empowerment, improved governance and the removal of social barriers are common to the strategies of IFAD and the other agencies. IFAD’s 1980s niche role in poverty reduction has been filled by changes in the way the strategies of other development agencies have evolved. All are now concerned with poverty reduction. The evaluation will, therefore, focus on IFAD’s comparative advantage in providing development assistance in an environment in which IFAD’s original mandate has been almost universally adopted.

Evaluation framework The evaluation framework for the study is developed in Chapter 3. It is structured by the two evaluation objectives of development impact and corporate performance. The issues are defined by IFAD’s stated objectives in its mandate and strategic frameworks for the period under review. Questions and sources of data are given in Table 7.

Evaluation framework IFAD’s objectives

1. Development results and impact on rural poverty reduction 1.1 Policy influence To strengthen policies and institutions 1.2 Project impact To introduce improved or expanded food production in the poorest

food deficit countries; to improve the nutritional level of the poorest populations; to strengthen capacity of the rural poor and their organizations; to improve equitable access to productive natural resources and technology; and to increase access of the poor to financial assets and markets

2. Corporate performance of policy and operations 2.1 Resource allocation To allocate funding on the basis of objective economic and social

criteria with regard to the needs of low income countries and fair geographic distribution; and to enhance catalytic impact

2.2 Policy & strategy development

To provide leadership in rural poverty eradication

2.3 Partnerships To develop national partnerships, regional and international coalitions

2.4 Project cycle management The design and implementation of innovative, cost-effective and replicable programmes

2.5 Knowledge management To manage innovation, learning processes, influencing and scaling up 2.6 Human resource management

To become a catalytic institution with highly motivated, well-trained and efficient staff


Page vi

This evaluation will use well-established evaluation criteria of relevance, effectiveness, efficiency, impact and sustainability.

Sampling of loans and grants Details of the sample design are given in Annex 2. The number of countries to be sampled in each region will be in proportion to the share of that region in IFAD’s portfolio of loan projects. Four countries will be sampled in all regions except Asia and the Pacific, from which five will be sampled, making a total of twenty one. Countries will be sampled at random with probability proportional to number of loan projects. A sub sample of two countries will be drawn randomly in each region for the country visits. Two projects will be sampled by simple random sample for detailed study in each country. Country specific technical assistance grants will be reviewed in the sample countries. An additional sample of 4 TAGs will also be sampled randomly.

Desk review Details of the desk review are given in Chapter 4. The review will combine analysis of documents with structured and semi-structured group and individual interviews. In order to ensure consistent interpretation across the team members and enable quantification of findings, document review proforma have been designed for documents with common formats such as COSOPs and projects. Draft examples are in Annex 3. The desk review will take place during February and March 2004.

Country visit Country visits will take place from April to September 2004. Work will be planned during the desk review. One or more team members will visit each country up to three times. The first, to make a reconnaissance visit to identify stakeholder groups and contract a national team of consultants. The second visit will be to initiate the process of fieldwork and hold preliminary meetings. The third will be to analyse data and draft a report jointly with the national evaluation team.

Three distinct classes of stakeholders will be contacted. Project Beneficiaries – people the project aimed to reach and who have been involved in project activities to date. IFAD also uses the term primary stakeholders to describe this group. Discussions will be held with groups, households and individuals. Partners – those who have knowledge of IFAD and/or its projects and beneficiaries but who are not directly involved in project implementation. This will be a wide ranging group and includes senior government personnel, co-financiers, donors, NGOs and key informants relating to project beneficiaries and their livelihoods. Project Management – those who are directly involved in implementing the project including managers, staff, TA and sub-contracted implementers. Interview topics will include both development impact and management issues.

Quantification of impact The terms of reference for the evaluation stress the need for the team to quantify impact as far as possible (para. 11(b)). A single -visit interview will be used, which means that most of the information to be collected will be based on memory recall by respondents. Draft Summary Form 4 in Annex 4 contains a first draft of questions. This will be developed during the desk study and formatted for computer processing.

The design concept is to maximise the value of information to be gathered by working with groups using participatory techniques, and with individuals or households following structured questions. The topics to be covered are complementary, as shown in the draft form. Classification of the data will enable analysis to be undertaken according to location, project, age of the project, component cluster, and socio-economic characteristic of the respondents. Furthermore, the approach will enable aggregation of results within regions and for IFAD as a whole. Quantitative secondary data appropriate to particular countries/projects will also be identified and sourced.

Reports and workplan The complex nature of the study will result in a substantial written output. Proposed structures for the reports are given in Table 11, page 41. A workplan and summary estimate of consultants’ inputs are given in Tables 12 to 14 starting at page 46.


Page 1

1 Introduction

The Independent External Evaluation

1.1 During its deliberations, the Consultation on the Sixth Replenishment of IFAD’s Resources endorsed a proposal to have an independent external evaluation (IEE) of IFAD. Accordingly, the Governing Council decided that the evaluation should be planned and begun in 2003 and completed in 2004, in time to allow for full consideration of the IEE Report by the Executive Board, prior to its recommendation to the Governing Council on the Consultation on the Seventh Replenishment of IFAD’s Resources. Following a competitive international tender, ITAD Ltd was contracted to undertake the assignment. This document is the Inception Report for the evaluation and responds specifically to Task 1 of the terms of reference. It builds on ITAD’s technical proposal by:

• Taking into account the perspectives of IFAD expressed through interviews with senior management and review of documents

• Operationalizing the scope, focus, main questions, methodology and tasks within the TOR

• Presenting a methodology for sampling

• Identifying documents and other sources of information for use in the IEE

• Developing a detailed timetable for the desk review (task 2) and consultant deployment and indicative timing of later tasks

• Making proposals for content and length of subsequent reports

1.2 The report reflects several rounds of support and advice from IFAD’s Office of Evaluation (OE), consultant advisers to the Director, OE and Senior Independent Advisers to the Director, OE. This final version incorporates comments received on an earlier draft.

Objectives of the evaluation

1.3 The objectives of the evaluation are set out in Section II of the Terms of Reference, and state:

‘The main objective of the IEE is to determine IFAD’s contribution to rural poverty reduction, the results and impact it has achieved in this area, and the relevance of the organization’s mission and objectives in relation to international development goals and the national development strategies of IFAD borrowing countries. The evaluation is further expected to assess whether and what IFAD has learned from past experience and how the Fund’s policies and operations have evolved in response to lessons learned from that experience, and finally, to offer recommendations on the policy directions IFAD should pursue and other steps it should take to improve its future performance.’1

1.4 This objective encapsulates three strands:

• development results and impact in regard to rural poverty reduction;

• the corporate learning process and performance of policy and operations; and,

• forward looking recommendations to improve future performance.

1.5 The first two concern the work of the evaluation, the third guides analysis and presentation of findings and recommendations. Evaluation of development results and impact 1 Annex 1. IFAD (2003) Independent External Evaluation of IFAD, Terms of Reference, Para. 3 Rome


Page 2

concerns the economic, institutional and social change arising from IFAD’s policies, projects and technical assistance grants. Evaluation of corporate performance concerns the relevance, efficiency and effectiveness of the processes and mechanisms by which IFAD develops policies and strategies, and identifies, designs, and implements project loans and grants.

1.6 The results of the evaluation will be used to inform discussions of IFAD’s next replenishment. In order to do this, the IEE will identify what is and is not working for the rural poor within the framework of IFAD’s activities and how IFAD has learned from experience.

Previous external assessment and review

1.7 This evaluation follows two comparable, though smaller studies: a Rapid External Assessment (Head et al, 1994) and an External Review of the Results and Impact of IFAD Operations (Dabelstein et al, 2002). The 1994 Assessment (REA) found evidence of the relative success of the Fund’s projects in reaching the poor and helping them to increase incomes. It called for IFAD to become a knowledge organisation: more transparent, analytical, accountable; promoting high quality and extensive evaluation, and sharing its experience widely. The REA saw IFAD’s potential as a leader, engaging in innovative and pioneering activities under a systematic framework of strategic policies.

1.8 The 2002 Review (ERT) found clear indications that IFAD has made direct and indirect contributions to achieving the Millennium Development Goals, and indications of impact on poverty reduction. It concluded that IFAD has shown the most promising impact in its innovative approach to social capital and empowerment, including attention given to the rights of women. It reaffirmed the importance of innovation in order for IFAD to sustain its leadership in two areas: to shape strategy and influence partners; and knowledge management. It also recognized a need for IFAD to strengthen its closeness to the field.

1.9 Both evaluations highlighted the importance of ensuring sustainability and replicability of development gains. Both provide a foundation from which to construct hypotheses about change that can be tested by the IEE. The ERT paid specific attention to the limitations on evaluation methodology. It reported constraints on the availability of evaluative documentation (page 2), noted that the number of OE evaluation reports was small (page 3), and found impact data not comparable or amenable to aggregation (page 4). Although evaluation reports were commended for being explicit about beneficiary performance (page 18), information about production, income and consumption was held to be ‘less than one might expect’ (page 9).

1.10 In the period since the ERT report, a significant number of new and independent studies have been undertaken or commissioned by OE.2 In comparison with the earlier studies the current IEE has larger resources, a longer period of duration and is independent from and external to IFAD’s management. The primary added value is expected to be the evidence of development impact from original fieldwork in ten countries chosen at random from the five regions and supported by the OE studies.

Structure of the report

1.11 The primary objective of the evaluation is to assess IFAD’s development impact. This Inception Report starts in Chapter 2 by reviewing the changing context for agricultural development and overcoming poverty. Next IFAD objectives are identified, how they were described in the Articles of Association, and how they have evolved through policies and 2 For example, Country Programme Evaluations; Evaluation of IFAD’s capacity as a Promoter of Replicable Innovation; Evaluation of IFAD’s Technical Assistance Grants Programmes for Agricultural Research; Evaluation of Supervision; and the start of a new series of annual reports on the Results and Impact of IFAD Operations.


Page 3

strategies in response to the changing environment. The Chapter concludes by examining the link between the MDGs and the current strategic framework.

1.12 Chapter 3 develops the evaluation framework for the study. This builds on methodology developed by OE but tailored to the IEE terms of reference. Issues of evaluability and the use of benchmarking are discussed and the evaluation criteria to be applied are also defined. The main part of the chapter sets out the evaluation questions and instruments or data collection methods to be used, following the structure of the framework.

1.13 Chapter 4 describes the methods the team will use during the desk review and country visits. Fully developed and draft instruments are attached as appendices. Chapter 5 contains a workplan. The levels of effort of the team are shown together with links to OE’s timetable, the ‘Supermaster’.

2 IFAD – policy and strategy in a changing environment

2.1 IFAD started in 1978 to help small and landless farmers escape poverty through increased food production and rural employment.3 Since then much has changed in the operational and policy environment, presenting new challenges and offering new ways of doing business. In evaluating IFAD’s overall performance, the IEE will address three central questions:

• How relevant are IFAD’s overarching objectives to the needs of Member States, and the evolution of thinking and practice on poverty reduction?

• How well do IFAD strategic objectives fit alongside those of other major development partners in working towards achievement of the MDGs?

• How well aligned are IFAD’s lending policies and operating principles to new ways of doing business at country level, in particular the adoption of country-led PRSPs?

2.2 This chapter reviews key highlights of the changing context and examines how IFAD’s approach has evolved accordingly.

The changing agricultural and poverty context

2.3 First, the need which IFAD was created to tackle continues. World Bank poverty researchers estimate headcount poverty (incomes below $1 at 1993 PPPs) at 1.1bn people in 2000. This was down from 1.24bn in 1990, but is still over one in five of the world’s population.4 IFAD’s 2001 Rural Poverty Report5 estimates that three-quarters of these acute poor live in rural areas. Over the last decade, the net number of the very poor has fallen sharply in East Asia, slightly in South Asia and risen everywhere else.6 The green revolution productivity increases, which were key to rural poverty reduction from the 1960’s have tailed-off. In Africa, not only has cereal production grown more slowly than population, but most of that growth has come from increased areas rather than productivity.7 For much of Africa, the potential benefits of demographic transition are now being eaten away by the HIV-AIDs epidemic and disease. For tropical commodity farmers, rapidly rising production in the face of only steady increases in demand have led to the lowest prices since records began.8

3 IFAD Lending Policies and Criteria, Objectives; 1978 4 World Bank Group PovertyNet website 5 IFAD (2001) Rural Poverty Report 2001. The challenge of ending rural poverty. OUP, Oxford 6 World Bank PovertyNet data 7 See, inter alia, Dorward et al (2004) A Policy Agenda for pro-poor agricultural growth, World Development vol. 32 issue 1, January 2004. 8 See, inter alia, Peter Robbins (2003) Stolen Fruit: The Tropical Commodities Disaster, Zed Books


Page 4

2.4 Second, political economy has fluctuated dramatically from the days of IFAD’s foundation. From the early 1980’s, economic intervention came under attack and tackling state-failure, real or alleged, was central. The collapse of the former Soviet Union was the iconic event. By the millennium, problems of market failure had re-emerged. Why was it that so many countries in Central Asia, in Latin America and Africa, which had gone through painful, apparently market-friendly, changes were not thriving? Market fundamentalism and openness was further challenged by new understandings of how heterodox policies – in ownership, land, investment, credit and trade - had contributed to the varied histories of success in Asia.9 The rise of civil society organisations (CSO) has brought in new actors and new possibilities for engagement. Working with CSOs is clearly an important dimension of IFAD’s regional and country strategies.

2.5 The failure of policy reform to kick-start growth and poverty reduction in so many places led to a new focus on the deep structure of incentives and how policy change is transmitted within an economy. Country-studies and the growing volume of governance panel-data suggested that the quality of national institutions is the key mediator between the accidents of history and geography, and development.10 By the end of the millennium, this focus on national institutional functioning was complemented by new attention to inter-country relations, particularly between developed and developing nations, driven by major changes in trade patterns; recognition of the complex benefits and costs of lower trade barriers and subsidies; increased ease of transnational travel, communication and investment; and global environmental, migration and security concerns.

2.6 Third, the donor world has changed substantially over the last 25 years. From the early 1980’s, constrained public expenditure in much of the developed world hit aid budgets alongside domestic spending. Net aid disbursements fell from a third to a quarter of OECD GDP by the end of the century. Agriculture sector aid fell dramatically.11 The lack of apparent impact from traditional capital-gap financing together with disastrous economic imbalances in many developing countries led to a new emphasis on policy and structural adjustment, driven by imposed conditionalities. From 1990, the Human Development approach sought to protect and enhance social spending in the face of necessarily drastic adjustment in many countries.12 The series of UN-led conferences throughout the 1990’s added targets to the poverty reduction and human development agenda, and broadened it to embrace commitments on gender, population and environmental sustainability. Civil society voice increased as cold war barriers disappeared and privatised aid flows strengthened the NGO sector. All major donors came to adopt poverty reduction as their goal, and the new global consensus culminated in the MDG’s, supplemented at country level by the PRSP’s, arising from pressure to ensure that debt-forgiveness funds were effectively used.

2.7 What difference have these changes meant for the rural poor? The majority of the poor live in rural areas. Direct support to agricultural development has been replaced by the rural development model, founded on the enabling role of rural infrastructure and social provision. But for the residents of remote, arid, mountainous, marginal or sparsely-populated lands, it is not clear that enabling actions alone are sufficient to overcome the high risks and transaction 9 See, inter alia, the extensive literature on Developmental states in East Asia from IDS, Sussex, and by Dani Rodrik and collaborators. 10 In the footsteps of Douglass North’s seminal Institutions, Institutional Change and Economic Performance, Douglass C North, CUP 1990, see, inter alia: Determinants of Economic Growth, Robert J Barro, MIT 1997; Tropics, Germs, and Crops: How Endowments Influence Economic Development, William Easterly and Ross Levine, NBER Working Paper No. 9106, August 2002; Governance Matters, World Bank Working Paper No.2196, Daniel Kaufmann, Aart Kraay, Zoido-Lobatón, October 1999. 11 See, for example Lipton, op cit Annex Table 2.5, page 66. 12 See the UNDP Human Development Report series. Amartya Sen's Development as Freedom (2001) is arguably the definitive statement.


Page 5

costs of getting markets to work properly, or of ensuring that well-adapted, higher productivity small-holder technologies emerge. And if enabling rural development is not sufficient to overcome development traps and kindle growth in difficult areas, the swings in development fashion over the past quarter-century have left the question of whose job it is to step in and help even more opaque. For many agencies, poverty reduction involves the provision of social services and the development of rural infrastructure. IFAD’s focus on productive elements and economic empowerment of poor people complements their work.

2.8 IFAD has published major reviews of trends in rural poverty and the donor response.13 It has continued to review and develop its own strategy and resource allocation in the light of these trends as discussed in the next section.14 In view of the continuing challenge this evaluation will assess relevance in the light of changing conditions, the evolution of thinking and practice about poverty reduction, and shifting views about the role of donors and lenders. The following sections describe how IFAD’s strategy has evolved in response to global changes and internal reflections.

IFAD’s Mandate

2.9 The basic mandate of IFAD is set out in the Agreement Establishing IFAD. Article 2 states ‘the objective of the Fund shall be to mobilize additional resources to be made available on concessional terms for agricultural development in developing Member States. In fulfilling this objective the Fund shall provide financing primarily for projects and programmes specifically designed to introduce, expand or improve food production systems and to strengthen related policies and institutions within the framework of national priorities and strategies, taking into consideration: the need to increase food production in the poorest food deficit countries; the potential for increasing food production in other developing countries; and the importance of improving the nutritional level of the poorest populations in developing countries and the conditions of their lives.’ 15

2.10 Article 7 provides further guidance, that ‘In allocating it’s resources the Fund shall be guided by the following priorities:

i. The need to increase food production and to improve the nutritional level of the poorest populations to the poorest food deficit countries;

ii. The potential for increasing food production in other developing countries. Likewise, emphasis shall be placed on improving the nutritional level of the poorest populations in these countries and the conditions of their lives.

Within the framework of the above-mentioned priorities, eligibility for assistance shall be on the basis of objective economic and social criteria with special emphasis on the needs of the low income countries and their potential for increasing food production, as well as due regard to a fair geographic distribution in the use of such resources.’16

2.11 The 1977 Lending Policies and Criteria set out the means by which IFAD was to achieve its objectives.

Increase food production through: a. overcoming constraints due to input supply and institutional barriers to dissemination of new technology

13 Notably: The State of World Rural Poverty, Jazairy et al, IFAD 1992; Lipton, op.cit.; the Regional Assessments of Rual Poverty (2001 to 2003). 14 Notably: The Challenge of Rural Poverty: The Role of IFAD, 1994; the Strategic Frameworks 1998-2000 and 2002-06; the six Regional Strategies, March 2002; Lending Policies and Criteria Annex, 1995. 15 IFAD (1976) Agreement Establishing the International Fund for Agricultural Development, Article 2 16 Ibid. Article 7


Page 6

b. increasing productivity on cultivated lands by provision of supporting services to farmers and funds to expand or improve irrigation and flood control systems and develop groundwater resources c. Co-operating with other agencies to support long-term investments in land and water development d. improving policies related to land tenure, pricing, credit, marketing, subsidies and cost recovery

Reduce poverty and improve nutrition through: a. including very small farmers in overall production strategies b. formulating programmes and projects targeted at landless farmers c. increasing purchasing power of poor people through wide range of agriculture related activities whose benefits accrue to landless workers and small farmers

Evolution of IFAD’s Strategy

2.12 The mandated objectives and priorities provide a basis on which the Fund’s lending policies and criteria have been developed by the Executive Board and Governing Council.17 This evaluation is concerned with projects effective during the period since 1994 and must reflect the structure of objectives that influenced those and have prevailed since that time. The Fund’s operational objectives during the mid-1990s were influenced by the findings and recommendations of the 1994 Rapid External Assessment.18 Following that report a new goal was adopted ‘to lead global efforts in helping the world’s poorest’, together with a vision statement with four elements:19

IFAD should be the leader in showing the way and galvanizing energies to eradicate rural poverty and hunger, and to unleash, through their own participation, the capacities of our clients: poor rural people.

Together with our partners, we shall ensure the design and implementation of innovative, cost effective and replicable programmes with sustainable impact, responding to the expectations of poor rural people and our Member States.

We want IFAD to be the catalytic institution of a mutual enrichment process which mobilizes resources and knowledge in a strategic, complementary and dynamic coalition of clients, governments, financial and development institutions, NGOs and the private sector.

Our leadership role depends on the teamwork of highly motivated, well-trained and efficient staff, committed to the cause of the rural poor. We want to become a Knowledge Organization about rural poverty, by sharing the lessons learned from our own experience, capturing and using the insights and knowledge of others, and constantly reinventing ourselves in the light of evolving circumstances.

2.13 This vision highlights a diverse range of characteristics which define IFAD’s operating niche: to provide leadership; work through partnerships; be a catalyst to mobilize resources; have committed staff; and develop as a knowledge organization. The statement highlights the assumed connection between corporate processes and impact on the ground. Leadership, innovation, catalysis and knowledge management are promoted as means to enhance impact, by enabling IFAD to function more efficiently and effectively. The logic is developed later in 17 IFAD (1998) Lending Policies and Criteria – adopted 14 December 1978 and amended at various times up to 12 February 1998 18 Head I et al (1994) The Challenge of Rural Poverty: The Role of IFAD. Report of the Rapid External Assessment. Rome 19 Developed from an institution-wide participatory process on 17 May 1995. Subsequently quoted in, inter alia, IFAD (1998) Meeting Challenges in a Changing World. IFAD’s Strategic Framework 1998-2000 Rome [Emphasis added]


Page 7

Figure 1 and summarized here. Policy is developed in response to changing political priorities and evidence about poverty. From those policies, strategies bring together an assessment of need and availability of funding and other resources. Global strategy provides a framework within which regional and country strategies can be designed. Within a country strategy IFAD follows a project cycle that starts with objective analysis of poverty, builds on its own and partners’ experience and is geared towards result-based, pro-poor interventions. Human resource management is a key element in developing the necessary staff knowledge and skills to fulfil these tasks. The extent to which IFAD’s current instruments (loans, grants and policy dialogue) are relevant to this changing strategic framework will be examined, as will be the issue of in-country capacity and field presence. The terms of reference for the IEE recognise that it is necessary to evaluate corporate performance in order to understand IFAD’s resulting impact.

2.14 The vision first articulated in the mid-1990s was further developed during the IFAD V and VI replenishment processes and has evolved into the 2002-2006 Strategic Framework. A summary is given in Box 1.

Box 1 Objectives in the 2002-2006 Strategic Framework

Mission: To enable the poor to overcome their poverty by becoming the driving force in their own development Three strategic objectives: 1. Strengthen the capacity of the rural poor and their organizations 2. Improve equitable access to productive natural resources and technology 3. Increase access of the poor to financial assets and markets Objective to enhance catalytic impact: • harnessing knowledge and disseminating to a broad spectrum of national and international

partners • supporting development of national partnerships among the poor, governments, private

sector and civil society • building regional and international coalitions • establishing institutional and policy frameworks in support of the poor

2.15 Given the ways in which IFAD has evolved, both as an institution and in its approach and focus, it will be important for the evaluation to judge the success of IFAD operations according to the institutional goals and operational strategy prevailing at the time. New strategies, such as the Strategic Framework 2002-2006 were informed by project operations and thus in part reflect experience over the years prior to its preparation. Thus the evaluation must take into account the time lag between policy changes and their reflection in project design, but in the context of learning from experience.

Evolution of strategies in other development agencies

2.16 There has been an evolution in the strategic objectives of other development agencies as shown in Tables 1 and 2 which compare the main objectives of the three major multilateral development banks, two bi-lateral agencies and international initiatives. It can be seen that during the 1980s, development strategies were either based on project lending, some of which was focused on agriculture to increase food production or multi-dimensional rural development, or on structural adjustment lending. There was no specific focus on poverty reduction as the assumption was that improved economic growth would “trickle down” to the poor.


Page 8

2.17 From about 1990 development strategies started to take more account of poverty alleviation with the need for specific poverty reduction measures, associated with actions to empower communities, address gender issues and remove social barriers that exclude the disadvantaged. By 2000 virtually all the main development agencies had made poverty alleviation a priority and this has been reflected in international agreement on the Millennium Development Goals. The strategies of most of the development agencies now have a number of themes in common. These include equitable growth, improved governance, the empowerment of communities through capacity building and access to information, and increased access to productive assets and markets. The development agencies also lay stress on the importance of building partnerships with clients, NGOs and other development agencies. The evolving strategies have led some agencies to re-organize management structures and emphasize the importance of knowledge management and the need for innovation. In parallel new lending instruments, such as the World Bank’s LILs and JICAs Community Empowerment Program have been developed.

2.18 The emphasis on poverty reduction and associated initiatives amongst the main agencies is also reflected in the number of publications devoted to poverty issues, partnerships and knowledge management. These include conference proceedings, monographs and technical bulletins as well as academic papers. For example, the World Bank has over 500 publications related to poverty issues, including a Source Book for Poverty Reduction Strategies. The Asian Development Bank lists over 200 such papers and each agency has contributed significantly to the literature.


Page 9

Table 1 Comparison of strategy evolution among World Bank, Asian Development Bank and African Development Bank

Date The World Bank Date Asian Development Bank Date African Development Bank

1980 – 93 1990 1997 1994 – 98 1997 – 03 1999 2000 -2001 2003

Structural adjustment. 144 programmes in 53 countries. Poverty reduction not an explicit objective, but OED review Structural Adjustment and the Poor, showed that poverty was reduced in countries that successfully implemented structural adjustment policies WDR. Poverty. A strategy of economic policy reforms and productive investments to improve incomes among the poor through labour intensive growth and to expand poor people’s access to social services and social safety nets. WDR. Rural Development: From Vision to Action. IDA 10, 11, 12 Replenishments. IDA required to put poverty at the centre of its Country Assistance Strategies and go beyond the 1990 poverty strategy by emphasizing gender, environment and governance as part of a broad-based framework for poverty reduction Country Assistance Strategies required to have a poverty focus Comprehensive Development Framework – poverty reduction central; development efforts should be long-term, holistic, results focused and partnership based With IMF – PRSP replace PFPs CAS to be aligned with country PRSP WDR. Attacking Poverty. Promote economic opportunities for poor through equitable growth, access to markets and assets, empowerment , governance and removing social barriers that exclude women and disadvantaged Reaching the Rural Poor: A Renewed Strategy for Rural Development.

Pre 1980 1980s 1992-95 2001 2002

Project based lending with about 50% of lending for basic needs., the remainder mainly for public sector works. Agriculture and rural development a priority. Project based lending dominant with agriculture and rural development still a priority, but structural adjustment, sector and program loans introduced. Medium Term Strategic Framework focused on economic growth, reduction of poverty, improvement of status of women and natural resource management and the env ironment.

Moving the Poverty Reduction Agenda Forward. Long-Term Strategic framework of the ADB. 2001 –2015. Poverty reduction the overarching mission through sustainable economic growth, inclusive social development empowering disadvantaged to gain access to assets, and improved governance.

Annual Report 2002 emphasizes bilateral and multilateral partnerships and knowledge management.

Early 1980s 1987 -91 1992 2000 2001

Strategy based on project loans and Technical Assistance. Agriculture the main beneficiary to ensure self sufficiency in food production in low income countries. Five Year Operational Program. Project financing placed emphasis on priority productive sectors, in particular agriculture and multi-dimensional rural development projects but no specific poverty focus. About 20% of lending for structural adjustment and sectoral adjustment loans. Poverty Alleviation Strategy and Action Program ADF VI lending policy was poverty alleviation as the main aim of Fund development activities based on beneficiary participation, targeting, poverty assessment and women in development. Introduction of participatory approaches supported by Sector Investment Loans. Vision Statement. Primordial challenge is that of poverty reduction and mission is to break the vicious cycle of poverty in member countries. Development assistance strategy to promote sustainable economic development with equity and poverty reduction as central goal, achieved through good governance and programs to provide poor with access to productive assets, information and social services. Partnerships and co-operation with clients and development agencies.


Page 10

Table 2 Comparison of international initiatives and strategy evolution by JICA and DFID

Date International Initiatives Date JICA Date DFID

1996

1997

2000

Highly Indebted Poor Countries (HIPC) Initiative. Objective was to reduce the debt of eligible poor countries. Enhanced HIPC. Direct link between debt relief and poverty reduction through targeting debt service savings to spending on social sectors. EHIPC objectives were broadened to not only provide a permanent exit from debt rescheduling, but also to promote growth and release resources for increased social spending. Millennium Development Goals. 8 goals, 18 targets and 48 performance indicators relating to poverty reduction, including income and non-income measures of well being

1992

1990s

1998

2000

2002

Charter stipulated no use of ODA for military purposes, promotion of democratization, introduction of market oriented economies and securing basic human rights Financial and technical support for development projects implemented by Japanese firms, many of which were infrastructural development. However, in other projects mainstreaming gender was emphasized and there was a significant technical assistance program. Introduction of Community Empowerment Program to build partnerships with NGOs, local government and academic institutions Goal is people centred development and poverty reduction through expansion of direct anti-poverty measures such as the CEP and an emphasis on poverty reduction in all facets of JICA operations. Further emphasis to building partnerships through and linkages to and collaboration with bilateral and multilateral agencies.

1980s

1997

1998

1999

2000

2001

2002

Projects and technical assistance based on political, industrial and commercial considerations, alongside basic humanitarian considerations. Aid and Trade Provision to assist UK firms secure orders in developing countries White Paper. Eliminating World Poverty: A challenge for the 21st Century. Policy statement. Eliminating World Poverty: Why the Environment Matters ILO’s Declaration of Fundamental Principles and Rights at Work acknowledged as central to poverty elimination Policy on Conflict Reduction and Humanitarian Assistance summarizes key strategies in relation to overall goal of poverty elimination White Paper. Eliminating World Poverty: Making Globalization Work for the Poor. Sets out an agenda to ensure that globalization brings sustainable benefits to the poor. Public Service Agreement emphasis on increase in development assistance to low income countries CAPs link DFID activities closely to poverty reduction outcomes and thus MDGs. Start from partner government’s Poverty Reduction Strategy.

Source: Policy documents, Annual Reports and Official Web Sites


Page 11

The international consensus

2.19 The most visible influence on development policy in recent years has been the global commitment to tackle poverty, expressed through the Millennium Development Goals. IFAD was not involved in development of the OECD’s forerunner International Development Targets,20 hence consideration of the approach came only with the adoption of the MDGs by the UN family of agencies.21 The MDGs were a direct stimulus to the IFAD Strategic Framework 2002-200622 (SF) that brought together experience from the 1990s and before to restate the Fund’s mission and objectives.

2.20 In the text of the SF, the role of leadership is given less explicit prominence than in the previous strategy but overall the SF remains consistent with IFAD’s mandate and the subsequent elements of partnership, catalyst and knowledge management. The strategic framework also embraced the MDGs, identifying specific roles for IFAD. The link between IFAD’s objectives and the overarching poverty reduction goal is self-evident. But linkages to other goals have also been stressed. Table 3 is a compilation by the evaluation team of IFAD’s stated roles.

Table 3 Links between MDGs and IFAD’s 2002-2006 Strategic Framework

MDG IFAD strategic objectives

Comment

Goal 1: Eradicate extreme poverty and hunger

All IFAD strategic objectives contribute to this goal

Key rationale for IFAD – rural poverty reduction Central to all projects, country/regional strategies, policy papers and international discussions

Goal 2: Achieve universal primary education

Not directly linked Related to IFAD’s mandate through provision of rural infrastructure, empowerment and gender equality.

Goal 3: Promote gender equality and empower women


IFAD places high emphasis on promoting gender equality and women’s empowerment;23 it also regards addressing gender inequalities and increasing women’s capabilities as preconditions for achieving the MDG. 24

Goal 4: Reduce child mortality

Not directly linked Related to IFAD’s mandate through improvements in nutrition25

Goal 5: Improve maternal health

Not directly linked Related to IFAD’s mandate through improvements in nutrition

Goal 6: Combat Especially relevant to In the strategic framework HIV is mentioned with regard to the vulnerability of the poor (page 3)

20 OECD (1996) Shaping the 21st Century: The contribution of development cooperation. Development Assistance Committee, Paris 21 United Nations (2001) Road Map Towards Implementation of the Millennium Declaration 56th Session of the General Assembly. New York 22 IFAD (2001) Enabling the rural poor to overcome their poverty. Strategic Framework 2002-2006 23 IFAD (2003) Women as Agents of Change. Discussion Paper page 3 24 IFAD (2001) Enabling the rural poor to overcome their poverty. Strategic Framework 2002-2006 page 2 25 The MDGs are not sector-specific. Each sector is linked to more than one MDG and many sectors are linked to many MDGs. For example, reduction of child mortality has a strong link to agriculture and rural development through aspects of food and nutrition. See, for example World Bank (2002) Annual Review of Development Effectiveness - Achieving Development Outcomes: The Millennium Challenge. Chapter 3. Washington


Page 12

MDG IFAD strategic objectives

Comment

HIV/AIDS

strategic objective 1 regard to the vulnerability of the poor (page 3) however it is als o recognized that HIV/AIDS is creating a new poverty dynamic and is also driven by poverty26

Goal 7: Ensure environmental sustainability

Especially relevant to strategic objectives 1 and 2

No specific link made to this goal, although environmental s ustainability is given as one of the areas that IFAD will “foster” in its work to enable the poor to overcome poverty; approximately 70 per cent of IFAD rural poverty projects are located in fragile, marginal environments.27

Goal 8: Develop a global partnership for development


IFAD has always worked in partnership with others and in the strategic framework is committed to supporting the development of national partnerships and building regional and international coalitions.

2.21 IFAD identifies clearly defined linkages between objectives set out in the Strategic Framework or subsequent working papers, and Goals 1, 3, 6, 7 and 8. Links are also evident for the health- and education-related Goals 2, 4 and 5. Work by OE has already mapped IFAD’s impact objectives against both the 2002-2006 Strategic Framework objectives and the MDGs.28

2.22 There are a number of reasons for the almost universal priority now given to poverty alleviation by development agencies.29 In the 1970s social development specialists were largely academics and if they were involved at all in project formulation it was as short-term consultants. The development agencies had little in-house expertise, if any, and the intellectual agenda was dominated by economists, who made judgements of social impact mainly from a distributional point of view. Even though aid at that time focused on meeting basic needs, the approach was essentially technical and/or economic. In the late 1970s and 1980s development agencies, led by the World Bank and ODA (now DFID) began recruiting social development staff. Cernea (1991), Chambers (1989) and others raised awareness of the need for social analysis to determine the social impact of aid projects. This was reinforced by the results of evaluation studies that showed some projects failed, not because of technical faults, but because they were not suited to the social structures of the beneficiaries. The 1984 famine in Ethiopia created enormous public awareness of the need to address the causes of poverty.

2.23 At the same time, NGOs and others were campaigning strongly against the negative social impact of structural adjustment policies and in 1990 the World Bank responded with the Poverty WDR, which had a considerable influence on thinking in many development agencies. The use of logical frameworks focused attention on impact and outcomes, rather than simply on delivery of outputs and cost/benefit analysis, and made a multi-disciplinary approach to project formulation essential. The growing influence of social development staff also resulted in adoption of participatory approaches and brought gender issues to the forefront. All these factors were building blocks in what has now become a new consensus in the development community about supporting poverty reduction. Encapsulated in the late 26 IFAD (2003) Achieving the MDG by Enabling the Rural Poor to Overcome their Poverty. Discussion paper page 12 27 IFAD (2002) Environment and Natural Resource Management: IFAD’s Growing Commitment 28 IFAD (2003) Annual report on the results and impact of operations evaluated in 2002. Office of Evaluation 29 Some of the issues have been discussed in: Eyben R. 2003. Mainstreaming the social dimension into the Overseas Development Administration: A partial history. J. Int. Dev. 15, 879 -892


Page 13

1990s by the Comprehensive Development Framework (CDF) and more recently by the introduction of Poverty Reduction Strategy Papers (PRSPs), this new consensus emphasises key principles in the provision of development assistance in support of poverty reduction – country ownership, broad based participation, results-oriented, long term and partnership based.30

Implications for IFAD of the changing context and strategic orientation

2.24 The review presented in this chapter highlights four key trends:

• acute poverty continues, and most of it is located in the smallholder agriculture economies of the developing world. This is the target-group IFAD was founded to work with.

• development policies have changed in parallel with radical change in global political economy. Shared agreement on the sources and impediments to growth and of growth-poverty reduction linkages has grown. The variance in view about appropriate roles for state, market and society has reduced whilst attention to the national institutions which determine how well sectors, organisations and people work together has grown.

• increased harmonisation in development understanding is leading to a growing harmonisation of corporate objectives in the development world (poverty-focus; social and economic targeting; institutional focus; results-orientation; MDGs).

• from the mid-1990's, the pace of policy reflection, strategy development and process change in IFAD has increased sharply. We do not know if the subsequent changes in strategy and process have already, or will in future, increase impact on the lives of the people IFAD was created to serve.

2.25 Comparison of the evolution of strategies adopted by IFAD and the other main development agencies shows that in the 1980s IFAD was unique in its focus on the poor. Although structural adjustment programs may have had an impact on poverty reduction, it was not their main objective. Now, however, poverty reduction is a priority, if not the main focus of all the main development agencies. Measures to reduce poverty, such as equitable growth, improved access to assets and markets, empowerment, improved governance and the removal of social barriers are common to the strategies of IFAD and the other agencies. Even the language used is very similar. This means that IFAD’s 1980s niche role in poverty reduction has been filled by changes in the way the strategies of other development agencies have evolved. All are now concerned with poverty reduction. IFAD’s 2001 Strategic Framework also emphasizes innovation, knowledge sharing, partnerships and policy dialogue. Again these are not unique to IFAD. The evaluation will, therefore, focus on IFAD’s comparative advantage in providing development assistance in an environment in which IFAD’s original mandate appears to have been almost universally adopted. Key questions will include:

• Have IFAD’s processes and approach to poverty reduction changed in response to the changed international consensus and to its own changing policies and strategies?

• Had IFAD’s allocation of resources reflected and been commensurate with its strategic objectives as they have evolved?

• How effective has IFAD’s performance in poverty reduction been to its own changes in policies and strategies?

30 Recognised by IFAD as a key feature of it’s policy framework. See IFAD (2001) Partnerships for Eradicating Rural Poverty. Report of the Consultation to Review the Adequacy of the Resources Available to IFAD 2000-2002 GC 24/L.3 (Consultation on the Fifth Replenishment) Rome para 19 et seq; IFAD (2002) Enabling the Rural Poor to Overcome their Poverty: Report of the Consultation on the Sixth Replenishment of IFAD’s Resources (2004-2006) REPL.VI/5/R.2 paras. 59, 73 et seq


Page 14

• How does the changing international environment and IFAD’s past performance affect IFAD’s comparative advantage or development niche regarding rural poverty?

2.26 The evaluation will study a sample of projects effective from 1994 to 2003. Unavoidable lead times in responding to policy change means that the early part of the sample will reflect IFAD strategies and processes from the turn of the decade. Later projects should reflect some of the more recent changes in strategy and process. An important evaluation question will be the extent to which current projects have been refocused to reflect evolving strategies.

2.27 The evaluation objective asks about IFAD's relevance to borrowing Member States. That depends on the extent to which the evolution of IFAD strategy and process – in the context of the simultaneous evolution of all other development agencies and borrowers – has increased or decreased IFAD's niche-role and relevance. That will be a key question to be answered in assessing evidence obtained from across the evaluation framework.

3 Evaluation framework and questions

3.1 This chapter describes the evaluation framework used in the study. It starts by reviewing a number of technical issues about our approach. The objectives call for evaluation of corporate performance, which arises from corporate mechanisms and processes. We present a model to explain what elements of corporate performance will be evaluated. The evaluation criteria to be used are then defined and applied in a series of tables that present questions and proposed means of measurement.

Evaluation issues

3.2 Terminology We have tried to use consistent language in describing our methodology, following current practice in OE. The evaluation framework is the structure of issues to be examined – based on the study objectives of development impact and corporate performance. The presentation follows the style adopted by OE in the Methodological Framework for Evaluation. Evaluation criteria are generic and follow OECD-DAC guidelines: impact, relevance, effectiveness, efficiency and sustainability.

3.3 Evaluability The terms of reference for this evaluation raised the issue of evaluability to act as a filter for projects sampled by the evaluation.31 The implication was that projects considered not to be evaluable should be discarded from a random sample or from the country visits. What is meant by evaluability? There is an interesting literature on this subject and the concepts have been applied to development projects.32 Common features are of a process to follow a series of steps: check first if there is a theory or logic about how the project is intended to work (possibly presented as a results chain or logical framework); next examine if it has been implemented well enough to have a plausible chance to produce its intended outcomes; and last, explore possible evaluation designs in the light of these findings, the likely cost involved and the intended use of the information. The assessment is not intended as a means to exclude projects from evaluation, but rather to identify practical shortcomings and their implications for the evaluation.

31 Terms of Reference para 15(a) 32 Key references include: Wholey, Joseph et al (1994) Handbook of Practical Program Evaluation, Chapter 2. Jossey-Bass Publishers, San Francisco; Patton M Q (1997) Utilization-Focused Evaluation, Third edition: The New Century Text. Sage, Thousand Oaks, California; Rossi, PE, Freeman HE, and Lipsey MW (1999) Expressing and assessing program theory, Evaluation: A Systematic Approach, Sixth Edition. Sage, Thousand Oaks, California. For application to development projects, see Poate et al (2000)


Page 15

3.4 For example, a Sida study on evaluability of democracy and human rights projects found a number of practical weaknesses: lack of monitoring data; lack of baseline data; difficulty of attribution; low quality of stated outputs; lack of or disagreement among stakeholders on indicators; high likely cost; project purposes that were not specific, realistic or measurable; absence of broad ownership of the project purpose among stakeholders; and low or restricted access to stakeholders by evaluators.33 None make evaluation impossible, but all affect evaluation methodology and costs.

3.5 For the purposes of this evaluation we believe it is unnecessary to take a narrow view of evaluability, especially with regard to availability of evidence, as the study is funded for primary data collection. In the absence of baseline or monitoring data greater reliance will be placed on the memory recall of respondents. The approach taken will be guided by the maturity of projects, with lesser treatment of impact on projects in the early years of implementation. Fieldwork may also be affected by security concerns and the study will be guided in this regard by UN criteria. Evaluability will be examined in the broader context of how well IFAD has developed a results-based approach to strategy and projects, in support of its ambitions as a learning institution with mechanisms that allow lessons from self-evaluation and independent evaluation to be applied.

3.6 Benchmarking The question of whether and to what extent this evaluation should make use of benchmarking was raised during the period of negotiating the assignment. What is benchmarking? Two references derived from literature on quality management provide an insight: ‘Benchmarking is a tool for improving performance by learning from the best practices and understanding the processes by which they are achieved” ‘.., the process of identifying and learning from good practices in other organisations”34

3.7 Who can IFAD learn from; what types of comparisons would be useful? Obvious partners are the global and regional IFIs, UN system organizations and other major bilateral and NGO development agencies. What types of practices can be compared? Sometimes benchmarking is interpreted as comparing performance ratios: the time from concept note to loan effectiveness; the staff input per approved loan; the proportion of projects rated satisfactory or better during implementation, etc. For example, portfolio performance statistics are readily available from the World Bank.35 Information is gathered through reviews using structured enquiries such as for Quality at Entry Assessments on a sample of new projects every year; supervision missions are required to assess the performance of projects towards their development objectives and in implementation progress, using an ordinal rating scale; and the portfolio database is analysed for time-related characteristics (such as time between loan signing and effectiveness). A large and diverse body of statistics is used to assess the performance of the portfolio.

3.8 Many of these statistics could be calculated for IFAD. But is such an approach worthwhile? For valid comparisons the scope and nature of the work, and the operating environment, need to be closely aligned.36 IFAD is very different from other IFIs: it has no country presence; project supervision is done predominantly by cooperating institutions; its strategy focuses on less favoured areas and marginal groups in society, factors that are likely to increase transaction costs. A large proportion of work is undertaken by consultants. Furthermore, data for the World Bank reflect a large population of over 2000 projects,

33 Poate, Derek et al (2000) The evaluability of democracy and human rights projects. Sida Studies in Evaluation 00/3. Sida, Stockholm 34 O’Reagain, S and R. Keegan (2000) Benchmarking explained, benchmarking-in-Europe.com, 2002 EFQM (1998-99) The European Benchmarking Code of Conduct, benchmarking-in-Europe.com 35 World Bank (2002) Annual Report on Portfolio Performance. QAG, World Bank, Washington 36 According to the Assistant President of FAD, efforts to benchmark performance between World Bank and IADB in the area of financial management proved unrewarding owing to fundamental differences in cost structures (pers. comm.)


Page 16

whereas IFAD's current portfolio is less than 250. IFAD is clearly different from other IFIs though this in no ways suggests it should not be held to the same high standards as other IFIs.

3.9 If the purpose of benchmarking is learning from good practices this can be done without recourse to narrow statistical computations. Firstly, an overall comparison of portfolio performance can be made with data from comparable organizations. This can include, for example, the extent to which the portfolio has been aligned with the MDGs.37 Secondly, process comparisons can be made about the ways in which quality assurance is done, or the techniques for knowledge management. In this evaluation, comparison of narrow statistical measures will be avoided, but IFAD’s comparative position will be mapped among peer institutions. Specific examples that will be examined to the extent that data permit during the evaluation are:

• Comparable processes of quality assurance used by the World Bank, regional development banks and other development agencies for which information is available

• Quality at entry as seen from the evaluation assessment of project design and published data from the IFIs

• Ratings of project performance used by IFAD and the IFIs

• Portfolio concentration by country with respect to poverty indicators

• Aging of the portfolio (projects with delayed or extended implementation)

• Costs and duration of project cycle processes such as identification and formulation

3.10 There may be greater potential for internal benchmarking of performance among the five regions, but that is beyond the scope of this evaluation.

3.11 Evaluation criteria This evaluation will use well-established evaluation criteria as set out in Table 4. The definitions given here are derived from those adopted by IFAD’s Office of Evaluation (OE) to ensure compatibility of analysis. The ways in which the criteria are applied to different topics or levels are set out in Table 5.

Table 4 Statement of evaluation criteria

Criteria Definitions derived from IFAD’s Methodological Framework for Evaluation

Impact The changes in the lives of the rural poor, intended or unintended — as they and their partners perceive them at the time of the evaluation — to which IFAD interventions have contributed, as well as the likely sustainability of changes.

Relevance A measure of the extent to which objectives are consistent with (i) the rural poors’ perceptions of their needs, potential and aspirations; (ii) the economic, social and policy environment; (iii) IFAD’s mandate, strategic framework and policies at the time of design; (iv) IFAD’s regional strategies; (v) IFAD’s current country strategy as formulated in the COSOP; and, (vi) the country’s current poverty-reduction policies and strategies.

Effectiveness The extent to which major objectives were achieved or are expected to be achieved

Efficiency A measure of how economically inputs are converted into results

Sustainability The probability of continued, long-term benefits from a development intervention

37 See for example: Word Bank (2002) Annual Review of Development Effectiveness. Chapter 1, The MDGs as a Benchmark for Development Effectiveness. Washington


Page 17

3.12 Assessing impact Measuring changes in the lives of the rural poor is a challenging task with several levels of complexity. The nature of the change itself can be hard to measure. For example, income is a difficult concept as it can include items in cash and kind. Statisticians have found that surveys of expenditure and consumption provide a more reliable estimate by proxy than direct questions about income. But to be accurate such estimates require repeated enumerations over controlled periods of time. Characteristics such as social capital and empowerment are fraught with problems of how to define objectively verifiable indicators. Measurement itself may depend on the extent to which indicators can be quantified and exist for independent inspection, or of necessity are only reported by memory recall. The magnitude of change is also difficult. Measurement before and after an event is desirable but, unless those people who participate in a project are known beforehand, the best estimate will have to come either from a statistical average of before-the-event, or again, by memory recall. If the main element of change arises from a natural process such as crop or animal production, climatic and biological diversity can result in large year-on-year variation that masks underlying trends. Data series may need to be collected for several years to demonstrate statistically significant change. And last, in order to attribute any change to the direct effects of a project it will be necessary to compare the experiences of those participating in the project with others who did not, but were living under comparable circumstances and experienced secular changes common to the participants.38

3.13 The most robust evaluation model is quantitative data collection collected from project participants before and after a project activity, and compared with non-participants over the same period. If participants can be selected at random this is an experimental design with before and after, and with and without measurements. In practice, such an approach is very costly and little used for routine evaluation of development projects. More commonly, the rigorous experimental design requirements are relaxed in several practical ways. Some simplify data collection, the others build on the underlying theory of change.

3.14 First, a quasi-experimental design is used with a sample pre-project or baseline collected before or during the early phase of a project and a post-project sample stratified into participants and non-participants. Further methodological simplification is to use memory recall in place of baseline surveys and to ask participants to quantify their change in income or production or whatever characteristic is of interest, without objective verification. Combining such self-evaluation with participatory tools that promote triangulation of information is widely accepted as a practical approach to economical evaluation.

3.15 Project designs are based explicitly or implicitly, around a logical process by which resources are used to deliver outputs in the form of goods and services to project participants. According to the attractiveness and appropriateness of those goods and services participants will respond by adopting knowledge, attitude and practices. That adoption leads to economic and social change: new crop and animal husbandry; improved access to and productivity of assets; uptake of social services; greater voice in society, etc. Efficient evaluation design can benefit from the underlying theory in several ways. First, by assessing the relevance of a project intervention (the extent to which it tackles a demonstrated problem), the potential attractiveness can be assessed. Secondly, if the underlying theory of change is known (such as the predicted level of crop productivity benefits from a type of seed dressing, or agronomic practice), quantification of the delivery of outputs by a project and the subsequent adoption by participants (evidence of effectiveness that is easier to measure than impact) can be used to infer likely impact.

38 See inter alia Casley, Dennis and Krishna Kumar (1987) Monitoring and Evaluation of Agriculture and Rural Development Projects, World Bank; Poate, CD and PF Daplyn (1993) Data for Agrarian Development. CUP; Chambers, Robert (1990) Putting the Last First, Longmans; IIED PLA Notes Op. Cit. World Bank (1996) Designing Project Monitoring and Evaluation. Lessons and Practices No. 8 , Operations Evaluation Department, Washington.


Page 18

3.16 This evaluation will use triangulated self-evaluation described above, supported by any secondary data available at project level. This will result in quantifiable responses to participants’ own perceptions of the benefits they have gained. It will not generate data on incomes that could be compared with household surveys of expenditure and consumption. It is cost effective and relatively quick to enumerate through a combination of group and individual interviews.

Functional model of IFAD processes

3.17 In order to evaluate corporate performance we need to understand IFAD’s operating procedures and mechanisms used to develop policies and strategies, and for the management of its portfolio of projects. Figure 1 presents a model to describe the functional processes by which IFAD achieves impact. It is derived from a review of corporate documents and discussion with operations staff, and reflects the interpretation by the evaluation team. The diagram illustrates the ways by which the mandate is transformed into policies which taken together with needs analysis and the funding envelope leads to the overall strategic framework. From that, regional and country strategies give rise to a mix of projects, grants and policy dialogue that create impact. The diagram clarifies the distinction between development impact (the results of project loans, technical assistance grants and policy dialogue) and corporate performance (the processes by which IFAD organises and delivers projects and policy dialogue).

3.18 The right hand side lists the process steps that take place. These steps identify the elements of corporate performance to be evaluated: strategy formulation affects the development of strategies at corporate, regional and country levels - the evaluation therefore will examine how these processes function, as well as the conformity and coherence of strategy among the three levels; the use and management of partnership arrangements; management of the project cycle; arrangements for monitoring and evaluation; knowledge management, with it’s links to innovation, scaling-up, and policy influence; and human resource management.39

39 Administration and financial management are outside the IEE terms of reference


Page 19

VisionMDG

Mandate

Expand & improve food production systemsStrengthen related policies & institutions

Policy

Targeting GenderInnovation Partnerships

Need

Poverty assessment

Funding

Resource envelope

Strategic framework

Human & Productive assets Financial assetsSocial assets & technology & markets

Regional strategies

COSOPs

ProjectsGrants

Policy dialogue

Impact

The path to impact

Figure 1 Functional model of IFAD processes

Corporate processes

Development & formulation of policy

Resource allocation

Poverty assessment

Strategy formulation

Partnerships

Projectcycle

Monitoring &evaluation

Knowledgemanagement

Institutional servicesHuman resource management,

administration & financial management


Page 20

Evaluation framework

3.19 Table 5 sets out the evaluation framework. The topics listed are structured by the evaluation objectives and detail the corporate processes identified in Figure 1. Each topic in the framework reflects specific evaluable objectives in the original mandate of IFAD and subsequent strategies and replenishment consultations, as referenced in the table. It meets the Executive Board’s requirements of evaluating developmental impact and corporate performance, and provides a structure for development of detailed questions later in this chapter.

Table 5 Evaluation framework

Evaluation framework Related objectives from IFAD’s mandate and subsequent strategies

1 Development results and impact on rural poverty reduction

1.1 Policy influence through dialogue and advocacy

• strengthen policiesae • strengthen institutions a

1.2 Project impact of loans and TAGs • introduce improved or expanded food production in the poorest food deficit countries a

• improve the nutritional level of the poorest populations a

• strengthen capacity of the rural poor and their organizations d

• improve equitable access to productive natural resources and technology d

• increase access of the poor to financial assets and markets d

2 Corporate performance of policy

and operations

2.1 Resource allocation • funding allocated on the basis of objective economic and social criteria with regard to the needs of low income countries and fair geographic distribution b

• enhance catalytic impact c 2.2 Policy & strategy development and

implementation • leadership in rural poverty eradication c

2.3 Partnerships • development of national partnerships, regional and international coalitions cef

2.4 Project cycle management • design and implementation of innovative, cost-effective and replicable programmes ce

2.5 Knowledge management • to manage innovation, learning processes influencing and scaling up c

2.6 Human resource management • catalytic institution with highly motivated, well-trained and efficient staff c

Sources: a: IFAD Mandate Article 2; b: IFAD Mandate Article 7; c: IFAD Strategic Framework 1998-2000; d: Strategic Framework 2002-2006; e: Consultation on the Vth Replenishment; f: Consultation on the VIth Replenishment

3.20 The questions to be asked are structured in a practical way to enable the team to organize and manage their enquiries. They fully reflect the major evaluation questions set out in the terms of reference (para. 11). Table 6 lists these questions and maps them to the evaluation framework.


Page 21

Table 6 Summary of evaluation questions from the IEE terms of reference

TOR Evaluation Questions Evaluation Framework

Two fundamental questions: Is IFAD properly focused on its rural development mission? Are the skills and resources of IFAD used in the best possible way, given IFAD’s overarching goal of supporting rural development and helping countries eradicate rural poverty?

1.1, 1.2, 2.1, 2.2 2.3, 2.4, 2.5, 2.6, 2.7

(a) What is the development relevance of IFAD’s policies and programmes? What value does IFAD add to the international development community, particularly in relation to rural poverty reduction, improved food security, relevant national sector policies, national development and poverty reduction strategies, such as the PRSPs, and the international development goals, e.g. the MDGs?

1.1, 2.2

(b) What has been the sustainable–quantitative and qualitative—impact of IFAD-supported projects on the ground. And to what extent has IFAD contributed to rural poverty reduction? How successful has IFAD been in improving agricultural and rural development policies and stronger institutional capacities in partner countries?

1.1, 1.2

(c) How effectively has IFAD promoted innovative approaches in relation to policy, partnerships, project implementation, technology and other aspects of IFAD-assisted operations that are meant to impact poverty? How has IFAD made use of local knowledge and technology in promoting innovative approaches? How have IFAD’s innovative approaches been replicated and scaled up?

1.1, 1.2, 2.2, 2.4, 2.6

(d) How effective has IFAD been in pursuing its objectives? How and to what extent are IFAD policies adapted to the achievement of these objectives, and how clear, explicit and measurable are IFAD’s objectives?

2.2, 2.5

(e) How efficient has IFAD been in the use of lending and other budget resources and the deployment of skills? And how selective has it been in the allocation of its resources and the choice of institutions and partners? To what extent has IFAD promoted the ownership and partnership of relevant host country institutions, including those representing the poor?

2.1, 2.3, 2.7

(f) What explains IFAD’s performance in terms of evaluation criteria and questions such as those mentioned above?

Concluding analysis

(g) How can IFAD enhance the impact and sustainability of its development cooperation, and contribute more to poverty-reduction efforts? What are the main recommendations that IFAD should consider adopting in the near and long term?

Concluding analysis


Page 22

3.21 The following tables set out the questions to be addressed by the evaluation. They are an expansion and further development of the major evaluation questions set out in paragraph 11 of the terms of reference and reproduced in Table 6 above. Each topic in the framework has a hypothesis to be tested. These are derived from IFAD’s mandate and strategy objectives described above. They encapsulate what potential change is to be evaluated. For example: 1.1 Policy Influence hypothesis: ‘In response to strategy development during the 1990’s IFAD has become a leader and exerted policy influence from pioneering and innovative activities’. This hypothesis summarises the key intention of strategy development as stated in IFAD documents. The evaluation will aim to test the evidence for this assertion, against a ‘null hypothesis’ that strategy development failed to enable IFAD to become a leader etc. The questions are derived from the team’s analysis of issues through document review and interviews during the inception phase. Where appropriate they build on evaluation questions defined in the OE methodological framework. According to the nature of each topic, questions are structured by level of influence – global, regional and country; they are also grouped by evaluation criteria of relevance, effectiveness and efficiency. For each question or set of questions the intended evaluation method or instrument is noted. Draft questionnaire proforma for reviews of COSOPs and all project-related data appear in Annexes 3 and 4.

3.22 The evaluation test for each topic in the framework is whether or not IFAD’s stated objectives have been met. In the case of project investments, such tests will be based on measurable objectives and stated targets as set out in project documents. The primary mechanisms for assessment of impact and project design and implementation will be review questionnaires and ratings of performance by the evaluation team. In addition, for overall ratings of projects (see Annex 4, Draft Summary Form 5), contestability will be introduced by asking IFAD management and country-level project management to give their own ratings. The results will enable analysis of divergent opinions by the team and be used as an input to the final analysis.

3.23 Monitoring and evaluation is a key topic, both as a potential source of information for the IEE and as important functions for management, accountability and learning. M&E issues are covered under framework 2.2 Policy and Strategy Development, and 2.4 Project Cycle Management. The range of issues covers M&E on projects, self evaluation by PMD and independent evaluation by OE.

3.24 Corporate performance and policy influence objectives are not stated in quantifiable ways. The evaluation judgement here will be based on a composite analysis of objective evidence, the observations of key informants, and judgement of the evaluation team about performance and trends over time. Evidence from other organisations will be examined to assess how successful the approaches that IFAD is following have been in other settings. Partnerships, knowledge management and human resource management are all aspects of corporate performance for which objectives have been set out in various strategies but do not have measurable targets in the same way as projects. For these, the stated objectives have been identified and used to develop hypotheses for each topic and sets of evaluation questions in Table 7. The evaluation team will make an overall rating of progress towards stated objectives for each topic and sub-topic in the framework using the four-point rating scale adopted by OE: high, substantial, modest and negligible.

3.25 An interesting feature of IFAD’s mode of working is the use made of independent consultants. Collectively, these form an important group of stakeholders with a share in the Fund’s corporate memory, and an interest in the Fund’s corporate culture and management style. The evaluation will draw on this resource in two ways. Firstly, consultants with relevant experience will be interviewed in connection with the sample projects for country visits. Secondly, under the human resource management aspects of the framework, the IEE will evaluate management procedures for consultants.


Page 23

Table 7

Evaluation of development impact

1.1 POLICY INFLUENCE Hypothesis Policy influence evaluation questions Evaluation instruments In response to strategy development during the 1990’s IFAD has become a leader and exerted policy influence from pioneering and innovative activities

Global/Regional Did IFIs and other development agencies change policies as a result of IFAD’s influence? Did the ‘2001 Poverty report’ and other key IFAD documents influence policies in IFIs and other development agencies? Country Did national/sectoral policies affecting the rural poor change? Did the regulatory framework affecting the rural poor change? Did rural financial institutions change? Were there other changes in institutions and policies? What value can IFAD add to sector-wide approaches? Did policies and strategies of NGOs change?

Desk Study: Review of policies of IFIs, UN and other development agencies; key informant interviews with policymakers. Results from the OE evaluation of innovation. Field study: Review of policies at country level; key informant interviews with national policymakers, representatives of development agencies and of civil society organisations.


Page 24

Table 7 continued

1.2 PROJECT IMPACT Hypothesis Project impact evaluation questions for loans and TAGs Evaluation instruments IFAD has achieved measurable impact in poverty reduction and contributed to capacity development and improved access to resources by the rural poor.

Relevance To what extent is the project aligned with national strategy, the COSOP and regional strategy? Do the project outcomes address identifiable problems? Effectiveness To what extent are outputs and targets achieved? Are some components better achieved than others? To what extent to TAGS and ECP projects support country strategy and operations? Efficiency Are outputs and impacts achieved within expected cost and time? Are there major cost- or time-overruns or budget revisions? How effective are project MIS and monitoring systems? Have IFAD efforts to upgrade project monitoring been effective?

Desk Study: Review of project documentation; Interviews with Country Portfolio Managers, OE staff and consultants Field Study: Interviews with project management; Project M&E system data

Impact How many people are likely to benefit directly? Is the number of beneficiaries as expected? How much have people benefited ? Are the poor the main beneficiaries? What are the benefits to poor people? Are women benefiting? Will the flow of benefits be sustained? Is production and/or productivity increasing? Are the benefits spreading? Are new technologies and techniques being successfully introduced? Is there sustainable impact on national/local state or sector organisation and functioning? Has policy dialogue occurred, and with what result? Is there sustainable impact on private sector development? Is there sustainable impact on civil society organisation and functioning? Is there an impact on the natural environment?

Desk Study: Review of project documentation; Interviews with Country Portfolio Managers, OE staff and consultants Field Study: Interviews with project management; interviews with partners; data from project M&E system data; qualitative and quantitative data collection from beneficiaries

Judgements where there are no data The availability of baseline data from project M&E systems will directly affect the quality of information on which the IEE can base its judgements. The ERT found that few projects had good data. This evaluation will make use of self-assessment by project beneficiaries to overcome problems of data inadequacies.


Page 25

1.2 PROJECT IMPACT Hypothesis Project impact evaluation questions for loans and TAGs Evaluation instruments

Have there been unforeseen impacts? Is the aggregate impact of projects in one country greater than the parts?

Table 7 continued

Evaluation of corporate performance

2.1 RESOURCE ALLOCATION Hypothesis Performance evaluation questions Evaluation instruments Funding has been allocated on the basis of objective economic and social criteria with regard to the needs of low income countries and fair geographic distribution

Has the actual distribution of project loans and technical assistance grants been selective with respect to rural poverty needs and regional locations? How selectively has funding been matched to needs? How do these compare with the rural programmes of comparable IFIs and development agencies? What are the implications for the proposed Performance Based Allocation System agreed under IFAD VIth Replenishment?

Desk Study: Statistical analysis of IFAD’s portfolio and those of World Bank, regional banks and the EU, as available, in the context of country and regional data on rural poverty. Comparative econometric analysis of donor efficiency, e.g. Beynon (2003)


Page 26

Table 7 continued

2.2 POLICY & STRATEGY DEVELOPMENT

Hypothesis Performance evaluation questions Evaluation instruments Adoption of strategic policies has set project activity in a systematic fashion within a broad country and inter-organizational context.

Global Relevance Does the SF specify relevant objectives consistent with IFAD’s mandate and successive Replenishment rounds (particularly V & VI)? Do SF objectives adequately reflect changes in IFAD’s policy and operating context? How well aligned are SF objectives with the MDGs (and PRSP initiative)? Effectiveness How effectively has the SF contributed to the articulation of IFAD’s comparative advantage or development niche? How effectively has IFAD promoted priority themes and policy areas? How have SF objectives helped guide strategic decisions about resource allocation, the choice of institutions and partners at the global level? Efficiency How has the SF been operationalised in terms of IFAD’s human resources and the budget? How well have these been adapted to changing circumstances?

Global Desk review of IFAD V and VI replenishment documents; the Strategic Framework and associated preparation documents and follow-up working papers; Interviews with members of the SF drafting team and Assistant Presidents, PD Front Office and PT staff

Interviews with key respondents in partner organisations; and policymakers and other stakeholders in sample countries

Regional Relevance How relevant are regional strategies in the light of changing priorities for poverty reduction? What is the fit between regional strategies and SF objectives? Do regional strategies take adequate account of regional-specific analyses and characteristics? Effectiveness

Regional Desk review of the regional strategies and associated preparation documents and follow-up working papers Focus group meetings with regional staff and Divisional Directors in PMD Interviews with key respondents in regional partner organisations; and policymakers and other


Page 27

2.2 POLICY & STRATEGY DEVELOPMENT

Hypothesis Performance evaluation questions Evaluation instruments How effectively do regional strategies draw on best practice knowledge and processes? Are the objectives clearly stated and impact measurable? Are there adequate arrangements for indicators and their monitoring? Do they emphasise policy priorities consistent with IFAD comparative advantage? Efficiency What has been the influence of regional strategies on strategic decisions about resource allocation and selectivity? Have they been fully operationalised? Do they build on previous regional experience?

stakeholders in sample countries

Country COSOPs Relevance Are COSOPs relevant to country circumstances? Are they consistent with regional strategies? How well aligned are COSOPs with country-led PRSPs and the principles underlying them, and what process was used to bring that about? Effectiveness How far do COSOPs draw on best practice knowledge and processes in their preparation? Are the objectives clearly stated and impact measurable? Are there adequate arrangements for indicators and their monitoring? Efficiency Do they provide adequate guide to the allocation of resources, the choice of institutions and partners at country level?

Country

Proforma-based desk review of the COSOPs and associated preparation documents

Interviews with current and past CPM and other relevant regional staff in PMD

Semi-structured questionnaire-based interviews with key respondents in regional partner organisations; policymakers, other donors, national partners, and implementation managers and other stakeholders in sample countries


Page 28

Table 7 continued

2.3 PARTNERSHIPS Hypothesis Performance evaluation questions40 Evaluation instruments The strategy of working in partnership with others has enabled IFAD to be more effective in delivering its strategic framework and ensuring programme impact and fulfilling its catalytic role.

Relevance Do partner organizations share the same goals and operating principles as IFAD? Do partnerships bring added benefits for IFAD to contribute to the MDGs? How does IFAD’s comparative advantage inform its partnership strategy? Partnership unity Do partnerships have clear and measurable objectives? Are those objectives widely shared by partner staff?

Semi-structured interviews with key respondents in regional partner organisations; policymakers, other donors, national partners, and implementation managers and other stakeholders in sample countries

Effectiveness How successful have partnership arrangements been in contributing to: scaling up through cofinancing; complementing macroeconomic reform; sharing institutional capacity; and exchanging knowledge? To what extent has IFAD promoted the ownership and partnership of relevant host country institutions, including those representing the poor? Are there arrangements to monitor and evaluate the partnership? Does IFAD contribute to or benefit from capacity building under the partnership?

Efficiency Is there a management or coordination mechanism for the partnership?

40 Partnership is a general term widely used to describe a range of inter-organisational relationships and collaborations. It is well described in the IFAD context in the Consultations on the Vth Replenishment para. 14 et seq and the VIth Replenishment, para. 72 et seq. Partnership structures can take a variety of different forms. As a result, there are many different definitions of what constitutes a ‘partnership’. In a review of the literature, Watkins and Csaky (2003: 13) find that two common themes emerge: i) A sense of mutuality and equality between those involved, including the attitudes of the partners working together and a sense of trust between those involved; and ii) mutual commitment to agreed objectives, reciprocal obligations and accountability, whether these are laid down in a formal contract or are more informally agreed on. These two qualities can be expressed as unity and direction – the unity between individuals and organisations, and the direction (or overarching goals) of the partnership shared among its members. As a way of working between organisations, partnerships may be distinguished from both market-based relations determined by contract, and relationships in the public sector involving hierarchies based upon authority. Trust is the control mechanism in networks and partnerships (Powell 1991, Thompson et al 1991); without trust between actors, there is little to distinguish a partnership from a contractual or authority-based relationship.


Page 29

2.3 PARTNERSHIPS Hypothesis Performance evaluation questions40 Evaluation instruments

How frequently and by what means is information shared within the partnership? Are resources adequate to achieve partnership goals? Have staff been trained in partnership working?

Table 7 continued

2.4 PROJECT CYCLE MANAGEMENT

Issues Performance evaluation questions Evaluation instruments Project cycle management has enabled IFAD to design and implement innovative, cost effective and replicable programmes.

Identification (Pre-Inception) Are projects and TAGs consistent with country strategies? What are the relative merits of identification by IFAD and by other partners? Are projects well poverty-targeted? What does project identification cost?

Desk Study: Review of project documentation; Interviews with Country Portfolio Managers, PD Front Office and PT. Evaluation officers who have participated in PDTs, TRCs and OSCs. Field Study: Interviews with project management; Interviews with partners

Preparation (Inception to RRP approval) To what extent has government managed the project development process? Are designs based on sound analysis of the causes of poverty? Are designs based on sound analysis of pathways out of poverty? Is there evidence of learning in the design of interventions? Are beneficiaries involved in project design? Are designs well gendered? Are project & component objectives clear, realistic and measurable? Are M&E indicators well structured, practical and operational?? Is the project logic consistent? Are designs clearly focused? Are designs significantly altered during implementation? Are measures to promote sustainability and replicability included? How appropriate and convincing is the economic analysis?


Page 30

2.4 PROJECT CYCLE MANAGEMENT

Issues Performance evaluation questions Evaluation instruments How appropriate and convincing if the technical analysis? Are risk appraisals and mitigations clear and realistic? How effective is the quality assurance system? Are the new OE structure, methodology and products impacting on project design and process? What does project preparation cost?

Implementation (Start-up to completion) Are implementation arrangements clear and realistic? Have implementation arrangements altered from the design? Do project managers and implementers receive appropriate and required management and technical guidance and support? Is CI supervision efficient and effective? Are CI recommendations actioned appropriately? Are self evaluation systems, such as the Progress Report on the Project Portfolio effective?

Desk Study: Review of project documentation; Interviews with Country Portfolio Managers Field Study: Interviews with project management; Interviews with partners; Interviews with CI representatives

Table 7 continued

2.5 KNOWLEDGE MANAGEMENT AND LEARNING

Issues Performance evaluation questions Evaluation instruments Knowledge sharing through innovation management, learning processes, influencing and scaling-up has improved effectiveness of IFAD’s programme impact and policy influence.

Effectiveness Do staff and consultants have easy access to information about lessons and innovation from IFAD and other sources? Are staff, consultants and partners well informed about lessons and innovation? Has the dissemination of information, especially products from OE, influenced strategy and project design? Has the dissemination of information improved working practices within the country programme, in other IFAD programmes or among partners? Has the dissemination of information influenced the work of partners?

Desk Study: Findings from the ‘Evaluation of IFAD’s capacity as a promoter of replicable innovation’. Document review of reports and other means of dissemination Focus group meetings with operational level staff in PMD and OE staff Field Study: Interviews with staff of partners at global, regional and country levels


Page 31

2.5 KNOWLEDGE MANAGEMENT AND LEARNING

Issues Performance evaluation questions Evaluation instruments

Efficiency Are there clear objectives and strategies for innovation and learning in IFAD, at global, regional and country levels? Are those objectives and strategies understood by staff? Are those objectives and strategies understood by partners? Does the approach maximise IFAD’s strengths? Does the approach make use of local knowledge and technology? Does the approach cater for joint working with partners (analysis of stakeholder interests, involvement in processes)? Are there clear targets within IFAD and among other agencies? Does the monitoring include assessment of performance at different stages of the communication process? Are the new OE structure, methodology and products increasing internal learning? Does the ‘Agreement at Completion Point’ procedure improve the effectiveness of evaluation feedback? Do IFAD’s organisation and culture promote innovation within IFAD? How do IFAD’s current instruments work for innovation?


Page 32

Table 7 continued

2.6 HUMAN RESOURCE MANAGEMENT

Issues Performance evaluation questions Evaluation instruments Development of highly motivated, well-trained and efficient staff has contributed to improved programme design and impact.

Effectiveness Corporate Culture Is management supportive to the desired roles of leadership, catalysis and innovation? What are corporate attitudes to high risk innovations? How well is gender mainstreamed within the organization Human Resources Is there a long-term HRD strategy? Are consultants fully integrated into any HRD strategy? Are competencies defined for staff and consultants at different levels? Is there a register of competencies of all staff and consultants?

Desk Study: Analysis of personnel and consultant databases for evidence of turnover, stability, and HRD; document review of operational procedures and manuals; interviews with senior management, staff of HR division and the Staff Association; key informant interviews with a sample of all staff

Efficiency Corporate culture How many levels of decision making are involved in operational approval ? Do innovative projects take longer to approve? Human Resources Is there an appropriate balance between staff and consultants? Does the present distribution of effort between staff and consultants bring value for money? Do staff performance reviews lead to appropriate career development? Do training programmes adequately meet identified career development needs? Is career development linked to performance and Divisional or Departmental plans? Is there recognition of innovation and the building of partnerships? Do managers have training in HRD issues? Do staff in the HR Division have the necessary specialization in HRD?


Page 33

4 Methodology

4.1 This chapter describes the ways in which the evaluation team will conduct the evaluation using the questions and instruments as set out in the framework in Chapter 3. The tasks are dealt with in chronological order, starting with the desk review. A summary of the inputs of the team and key dates appears in Chapter 5, as part of the workplan. The first part of this chapter deals with the selection of countries and projects to be studied. The tasks under the desk review and country visits are explained next. Last is a proposal for the structure and contents of the intermediate and final reports.

Sampling of countries and projects

4.2 Annex 2 contains details of the approach to sampling countries, loan projects and technical assistance grants. The approach follows that prescribed in Annex 1 to the terms of reference. Key features are:

• The number of countries to be sampled in each region is in proportion to the share of that region in IFAD’s portfolio of loan projects. Four countries were sampled in all regions expect Asia and the Pacific, from which five were sampled.

• Countries were sampled at random with probability proportional to number of loan projects.

• A subsample of two countries in each region were drawn randomly with probability proportional to number of loan projects for the country visits.

• Two projects were sampled by simple random sample for detailed study in each country. If a country had less than two IFAD projects three projects were selected from another country and details recorded.

• All country specific technical assistance grants will be reviewed in the sample countries. An additional sample of 4 TAGs with total costs greater than US$100,000 was also sampled randomly.

4.3 The sample was drawn in the OE office using numbered balls. The procedure was witnessed by a group of 40 or more IFAD staff.

Desk study

4.4 The largest part of the work during the desk study phase is to review information available at IFAD concerning policies and strategies, projects and TAGS. Table 8 lists the documentary sources to be reviewed for each of these topics. All reviews will make use of structured review proforma drafts of which are in Annexes 3 and 4. These will ensure:

• A consistent approach by all team members and across all documents • Detailed cross-referencing of sources of information to underpin the subsequent

analysis and provide a trail of evidence • A documented summary of factual information • For each topic, a rating or summary assessment by the reviewer

4.5 Technical Assistance Grants will be evaluated in two ways. Firstly all grants that are specific to the sampled countries will be reviewed to assess consistency and complementary with the country strategy and programme. Many of these grants are small and have been used for activities such as convening a workshop or training course. In addition a sample of four large TAGs will be evaluated using the same criteria as for the loan projects.


Page 34

Table 8 Summary of documents for desk review

Documents to be reviewed

Policies and Strategies

1976 Agreement Establishing IFAD 1977 Lending Policies and Criteria. Subsequent revisions in 1994 and 1998 1992 Poverty report 1992 IFAD’s Role and Resource Requirement. IFAD IV Replenishment 1994 Rapid External Assessment 1996 Strategic Framework 2000 Replenishment V 2000 Action Plan 2000 – 2002 2001 Rural Poverty Report 2001 Strategic Framework for IFAD 2002-2006 2002 Replenishment VI 2002 Regional Strategy Papers 2001 – 2003 Regional Assessments of Poverty 2003 Achieving the Millennium Development Goals - 5 Discussion Papers 2004 Progress Report on the Project Portfolio Documents pertaining to the Special Program for Africa All Thematic Evaluations and other thematic work Corporate level evaluations For each sampled country: Current and previous COSOPs

Projects Report and Recommendation of the President Background formulation/appraisal documents as necessary Lead Adviser’s Memo Most recent supervision report Mid-term review PCR Interim evaluation report Final evaluation report

TAGs Report and Recommendation of the President Background formulation/appraisal documents as necessary Evaluations and other reports

4.6 Evaluation reports Reports produced by the Office of Evaluation constitute a specific resource for the evaluation. Project- and country-specific documents will be examined as part of the project review process. Other documents fall into two classes: those commissioned after the OE became independent in April 2003, and those commissioned before. Table 9 lists those studies that are completed and identifies their relation to the evaluation framework.

Table 9 List of Evaluation Reports

Date Title Reference to Evaluation Framework topics

Post April 2003

Annual Report on the Results and Impact of IFAD Operations, September 2003

1.2

Evaluation of IFAD’s Supervision Modalities 2.4 Marketing and Competitiveness in West and Central

Africa 2.2

Thematic evaluation on Decentralisation in PF 1.1 Country Programme Evaluations in Tunisia, Senegal and

Indonesia (and forthcoming in Bolivia and Egypt) 1.1, 1.2


Page 35

Date Title Reference to Evaluation Framework topics

Pre April 2003

Thematic Evaluation of Organic Agriculture in Latin America, December 2002

1.1, 1.2

Evaluation of IFAD’s Technical Assistance Grants Programmes for Agricultural Research, September 2002

1.1, 1.2

Evaluation of IFAD’s Capacity as a Promoter of Replicable Innovation, February 2002

2.4

Evaluation of IFAD/NGO Extended Cooperation Programme, September 2000

2.3

Promotion of Local Knowledge Systems and Innovations 2.6 Thematic evaluation on Agriculture Extension in PA 1.2 Thematic evaluation on Innovative Approaches in Peru 2.4 Thematic evaluation on Rural Financial Services in PN 1.2 Country Programme Evaluation in Tanzania 1.1, 1.2

4.7 Interviews and meetings Interviews with IFAD staff will form an important part of the desk study methodology. Many will be done on an ad hoc basis to pursue specific lines of enquiry about policies, strategies and projects. There will, in addition, be a variety of structured meetings to explore specific topics. Table 10 summarizes these.

Table 10 Summary of planned interviews during Desk Review

Evaluation framework topic

Type of interview and participants

Policy and General Executive Directors President and Vice President Senior Management Team

1.1 Policy influence Key Informant interviews with front desk staff, stakeholders in Rome-based institutions (WFP, FAO); telephone and email interviews with stakeholders in other locations

1.2 Project impact Interviews with Country Programme Managers and assistants

2.1 Resource allocation

Interview with Assistant Presidents PMD & FAD

2.2 Policy & strategy development

Focus group discussion with contributors from all five regional strategies together

Key Informant interviews with the five regional directors

2.3 Partnerships Questionnaire survey among a sample of all staff

2.4 Project cycle management

Key Informant interviews with staff from PT and OE

Group meeting with CPMs and assistants

2.5 Monitoring & evaluation

Key Informant interviews with staff from PT and OE

2.6 Knowledge management

Focus group discussions with groups of CPM in each region plus OE staff

2.7 Human resource management

Key informant interviews with staff of HR Division and the Staff Association; random sample of staff from all levels across regions and other divisions


Page 36

4.8 Details of all reports including the Desk Study report are given in a later section below.

Country visits

4.9 The main objective of the country visits is to validate and enrich the desk review and to generate new information that will confirm or refute the conclusions of the desk review.41 The work will be co-ordinated by an international consultant collaborating closely with a team of in country consultants. IFAD has undertaken to assist in identifying a national evaluation counterpart who will organize access to selected IFAD-supported projects and relevant officials and other stakeholders. We will consult with the national counterpart, and the particular CPM, on selection of the in country consultant team. We will ensure that the national evaluation team members have no conflict of interest in terms of the IEE TOR.42

4.10 The following sections describe our approach to the country visits in terms of consultation, sampling and methodology. The tasks for each phase of the study, including pre-visit preparation, are outlined with the selection criteria for the national evaluation team. Drafts of the evaluation summary forms are given in Annex 4.

4.11 Although the main focus of the evaluation is the two projects sampled in each country, the evaluation will examine the whole portfolio to understand its composition, coherence and performance. This will be done in three ways. Firstly, details of all projects: their size, objectives, location, implementation and supervision arrangements will be obtained from PPMS and by discussion with CPMs. That data will include the most recent Project Status Report. Secondly, working through the National Evaluation Counterpart at country level, the team will interview key informants among partners and project management (see below). To the extent that other projects have links to any of the sub-sectors, target groups or locations of the sampled projects, further information will be sought about their current implementation experience and performance.

4.12 Stakeholder Consultation Stakeholders are organisations, groups or individuals who have a direct or indirect interest in a particular development intervention.43 Stakeholder analyses will be conducted at an early stage of preparation for the study. The analyses will be refined during the process in order to obtain a range of perspectives and ensure that all key people and groups participate in the study. Our broad classification of stakeholders with whom we will consult is:

• Project Beneficiaries – people the project aimed to reach and who have been involved in project activities to date. IFAD also uses the term primary stakeholders to describe this group44 whose interests are key to project success. We will explore perceptions of changes in assets ownership and productivity; access to, use of, and satisfaction with physical and economic services; effects of changes in institutions, regulations and services. Changes in food security, social capital and empowerment will be given particular attention. Discussions will be held with groups, households and individuals.

• Partners – those who have knowledge of IFAD and/or its projects and beneficiaries

but who are not directly involved in project implementation. This will be a wide ranging group and includes senior government personnel, co-financiers, donors, NGOs and key informants relating to project beneficiaries and their livelihoods. The focus of interviews will be on relevance, effectiveness, impact, sustainability and replicability. Opinions will be sought on engagement in, or impact on, policy

41 Annex 1 IFAD (2003) Independent External Evaluation of IFAD, Terms of Reference, Para16 42 Ibid. paras 28(d) and 29 43 DAC/OECD (2002) Glossary of Key Terms in Evaluation and Results Based Management 44 IFAD (2002) Managing for Impact in Rural Development: A Guide for Project M&E


Page 37

processes and the relationship of the project with civil society and empowerment of the poor.

• Project Management – those who are directly involved in implementing the project

including managers, staff, TA and sub-contracted implementers. Interview topics will include both development impact and management issues.

4.13 Sampling The stakeholder analyses, for national level and the selected projects, will form the basis for planning initial meetings and contacts in country. At the project level we will take account of key elements of diversity, for example locations, components and types of beneficiaries. Sampling will then be planned within these strata taking account of the resources available. An indicative objective is to conduct interviews with at least 100 beneficiaries per project. Specific targets will be developed once the nature of the sampled projects is known.

4.14 Work with beneficiaries will include meetings with particular focus groups on whom the project may have had a differential impact and/or whose interests are a priority for IFAD. Female headed households (de jure or de facto) and other groups of women will be included in all studies; other possible groups are richer vs poorer households; distinct ethnic or socially differentiated groups; elderly vs younger groups; livelihood systems may be an important conditioning factor in some cases. Interviews with individuals and/or representatives of households will be conducted in parallel.

4.15 Methods and approaches We will draw on the experience and design of OE methodology45 as well as from other development organizations46 for country and project level evaluations. We will use a range of participatory approaches appropriate to particular circumstances taking into account lessons learned from field practice.47 The use of participatory tools will be a decision for each country team depending on the nature of stakeholders and circumstances of each project. A core list of impact questions appears in Annex 4. Participatory tools will be used where the field teams consider that their use will enhance the quality and accessibility of information. Examples would be to conduct a wealth ranking to identify groups in society from whom questions about impact can be asked. If the desire is to compare some aspect of performance today with the situation before the project a time-line diagram might be used or simple visual comparison of scale.

4.16 We intend to use a system of question checklists to guide semi-structured interviews with all types of stakeholders. In some situations other methods will be used to ensure wide participation, shared understanding and quality discussions. These could include mapping community changes, seasonal calendars for food security issues or wealth ranking within a project area. It is, however, important not to pre-judge what will be both useful and acceptable. We intend to develop detailed approaches with our national consultant team and assess their briefing/training needs during the reconnaissance visit (see below).

4.17 The information obtained through discussions and interviews will be summarised using standardized forms and a system of ratings. Draft formats will be refined during the desk review period.

45 IFAD (2002) Managing for Impact in Rural Development: A Guide for Project M&E; IFAD (2003) A Methodological Framework for Project Evaluation 46 For example the OECD/DAC guidelines; World Bank 2003 conference on Evaluating Development Effectiveness; ActionAid ALPS system; work at IDS and IIED on participatory M and E 47 For example: Cornwell and Pratt (2003) Pathways to Participation: Reflections on PRA IDS; IIED (1998) PLA Notes 31: Participatory Monitoring and Evaluation


Page 38

4.18 A set of briefing papers will be prepared on approaches and methodology for our in- country consultants. The reconnaissance visit will provide the opportunity to discuss socio-cultural acceptability of methods/questions and identify training needs of our in country teams. Preparatory workshops will be held with the international and in country consultants to ensure shared understanding and a strong team ethos.

4.19 The terms of reference for the evaluation stress the need for the team to quantify impact as far as possible (para. 11(b)). The resource envelope available for the evaluation and tight timescale means that data collection will be restricted to single -visit interviews. This means that most of the information to be collected will be based on memory recall by respondents. (There will be small but significant exceptions where interviews can be supplemented by direct observation of assets and activities.) Within those constraints the team plans to make full use of pre-coded questions drawing on beneficiaries’ self-assessment of project activities and benefits. Draft Summary Form 4 in Annex 4 contains a first draft of questions. This will be developed during the desk study and formatted for computer processing.

4.20 The design concept is to maximise the value of information to be gathered by working with groups using participatory techniques, and with individuals or households following structured questions. The topics to be covered are complementary, as shown in the draft form. Thus the same or similar questions will be asked of groups through participatory settings as are asked of individuals or household representatives through a questionnaire. Classification of the data will enable analysis to be undertaken according to location, project, age of the project, component cluster, and socio-economic characteristic of the respondents. Furthermore, the approach will enable aggregation of results within regions and for IFAD as a whole.

4.21 In addition, and to provide a degree of triangulation where possible, quantitative secondary data appropriate to particular countries/projects will be identified and sourced. For example by:

• examining records kept by micro-finance and other agencies for the period since the project was completed

• inspecting statistical data for the location or topic of the project; • seeking independent survey data from third parties such as NGO or community

organisations, researchers and other development organisations 4.22 The first country visit will be planned so that experiences, particularly on the use of the question checklists and summary forms, can be fed back in time to influence subsequent visits.

4.23 Outline workplan This section outlines how the country visit work will be divided into phases with key tasks listed for each phase.

Preparation Phase 4.24 This will take place during the desk review period. It is designed to prepare the way for maximum effectiveness and efficient use of time in country. Key tasks are:

• Consultation with the CPM – checking available data and sources; drawing up a preliminary stakeholder analysis

• Establish contact with national evaluation counterpart (who will be appointed by IFAD)

• Provisional identification of possible local consultants (see draft criteria) • Plan programme for the reconnaissance visit including initial meetings with key

stakeholders where feasible • Prepare draft briefing documents, developed as part of the country and project based

desk review, for national counterpart and in country team including


Page 39

o Summary of issues relating to regional and country strategies; partnerships, process and national poverty reduction priorities

o Summary of project level findings and issues arising from the desk review; o Checklists of questions; report structures; principles of participatory inquiry

and working approach

Reconnaissance Visit 4.25 We plan that a short visit by an IEE core team member will be made to each country in advance of the field work. This will enable initial contacts to be with key stakeholders, providing the opportunity to introduce purpose and approach of study in advance of more detailed evaluation discussions. We will also initiate the planning process for the field visits which will be continued by the national team. Key tasks are:

• meet national evaluation counterpart • recruit national consultant team (see selection criteria below) • refine/agree specific country approach and location of field work • identify training/briefing needs of consultants • develop and agree detailed workplan and timing – especially preparatory work to be

done on logistics, field studies and secondary data • draw up roles, responsibilities and contractual arrangements • validate/modify the preliminary stakeholder analysis • begin process of developing national time line of key policy changes /events relating

to development and poverty reduction • hold initial meetings/briefings for key stakeholders/partners as appropriate

(government and project agencies) • obtain or locate sources of appropriate national information/statistics and project level

information

National evaluation team criteria for selection • No previous direct connection with IFAD as per the conflict of interest in IEE TORs • Knowledge of rural poverty, livelihoods of the poor and the agriculture sector • Experience of undertaking field level, participatory evaluations • Gender balance of team • Understanding of monitoring and evaluation concepts, project, programme and policy

issues • Evidence of ability to produce good quality, concise summary reports • Good reputation for quality of work among other donors and track record of

delivering outputs on time • Ability to organise logistics and back up for international consultant, the completion

of the field work and reporting requirements 4.26 The international consultant will visit the country twice. In the intervening period the national evaluation team will continue field level work and sourcing secondary data as required.

First field visit 4.27 This visit will focus on key stakeholder meetings, briefing for the national team and initial field work. Early field work will be reviewed to identify and rectify any problems or gaps in both process and products. Key tasks are:

• Meetings with Project Management team and other key stakeholders • Conduct briefing/ orientation workshop with in country team • Initial field visits to project locations and testing of planned methods and

approaches


Page 40

• Review workshop with in country team • Detailed workplan agreed for continuation of field work and reporting

Second field visit:

4.28 This visit will take place after, or very close to, the end of the field work. The focus will be on review, analysis and feedback to stakeholders. Key tasks are:

• Progress review with in country team • Analysis of field data and estimation of beneficiary numbers • Follow up/ feedback meetings with key stakeholders • Drafting country working paper • Wrap up meetings

Analysis of data

4.29 Data analysis will be undertaken using standard PC spreadsheet and database software. Nominal and ordinal ratings and other data recorded on the document review proforma will be entered to spreadsheet by the team members as the work is done. Analysis will consist of frequency distributions of rating scores cross tabulated by characteristics such as country, region, ‘standardised component cluster’, year of effectiveness, nature of cooperating institution, and costs.

4.30 Information collected during the field visits will be processed in a similar way. The Project Management and Partner interview summaries will be entered to a spreadsheet database. Impact data collected from groups and individual beneficiaries will be entered into a database using software to be developed by the team, and tabulated using proprietary database and spreadsheet functions.

4.31 Aggregation of findings from surveys and ratings used for documentary assessment will be presented in analytical matrices that bring together IFAD’s policy and strategy objectives, plans for action and assessments made by the evaluation team. A particular feature for project impact will be comparison of ratings made by the evaluation team, by IFAD management and by project management.

Reports: desk, country, regional and final

4.32 The sequence of work during the evaluation has been planned logically to build to a final analysis and report. The stages of intermediate reporting are designed to establish a clear trail of evidence from documents and interviews that support the final conclusions. The report structure has been developed to create a set of complementary documents that avoid duplication and provide a cumulative assembly of findings. Table 11 presents the proposed tables of contents for all reports. The tabular layout is presented to show the interlinkages and complementarity among the reports. An indication of target length is also given.

• Desk study report, 50 pages, plus a Country Synopsis annex for each country in the sample

• Ten country working papers, 55 pages – a stand-alone report to be written at the end of each country visit, drawing together information from both the desk review and country visits.

• Thematic reports – four short cross-cutting reports to bring forward information from the countries that received field visits with the other countries in the region that were studied at the desk review.

• Final report that draws together the findings from the whole study


Page 41

Table 11 Proposed tables of contents for desk, field and final reports48

Desk study report (50 pages)

Country working papers (55 pages)

Thematic summary reports (10 pages)

Final report (33 pages)

Notes - Completed in-country at conclusion of each country visit. - For ten countries total. - Stand-alone documents, hence including some items from Desk Study Country Synopsis Papers

Notes - Five short summaries highlighting key lessons from Desk Study and Country Synopsis Papers

Abbreviations Table of Contents Executive Summary

Abbreviations Table of Contents

Abbreviations Table of Contents

Abbreviations Table of Contents Executive Summary

1. Introduction (2) - IFAD & IEE background - Purpose of the Desk Study - Content of the Report - Acknowledgements

1. Introduction (2) - IFAD & IEE background - How country was selected - Country situation and IFAD strategy - Purpose of the country/project study - Content of the Report - Acknowledgements

1. Introduction (1) - IFAD & IEE background - Desk & Field Study countries - Content of the Report - Acknowledgements

1. Introduction (2) - IFAD background - ToRs & Purpose of IEE - Acknowledgements

2. Work Completed & Methodologies (3) - Sampling : methodology & characteristics of the sample - Document Reviews - Meetings - IEE Consistency and Quality

2. Work Completed & Methodologies (3) - National Evaluation Counterpart and National Evaluation Team - Meetings with Project Management - Beneficiary Sampling - Field data collection - Meetings with partners and key informants

2. Work Completed & Methodology (1) - Inception Report - Evaluation Framework & Methodology - Desk Study - Country Visits - Reports completed

3. Results 3. Results 2. Findings (10) 3. Findings 3A Products (20) - Portfolio Analysis (1994-2002) - Loans: Performance and Trends - Quality at Entry

3A : Project 1 (20)

Candidate list of themes: • Overcoming rural poverty • Strengthening capacity of the rural

poor and their institutions

3A Products and Impact (10)

Portfolio Trends Selectivity and poverty focus

48 The contents and length of the Thematic reports and Final reports will be finalized between the IEE team and OE as the study progresses.


Page 42





Targeting (Poverty & Gender) Focus Logic Analyses Sustainability Replication Innovation

- Implementation - Supervision - PSR Ratings - Outcomes (Secondary data, mainly from MTRs, PCRs; project, country and thematic evaluations) - TAGs

Design Targeting (Poverty & Gender) Focus Logic

Implementation

Project Management Project Supervision Validation of PSR Ratings

Project Impact (Quantitative Data by component from Beneficiary Participatory Survey and project MIS. Qualitative data from interviews) - Number of Beneficiaries - Impact on: Income, Food, Skills, Health, Voice, Overall, Attribution - Relevance - Targeting (Poverty & Gender) - Sustainability - Innovation & Replication Institutional Impact - Policy dialogue and state reform - Private sector development - Civil society development

• Improving access to productive natural resources and technology

• Increasing access of the rural poor to financial assets and markets

Evidence from the desk study and country visits

Impact on rural poverty Summary quantitative analysis

Enabling the poor to overcome their poverty

Developing capacity and enabling access to resources and assets

3B : Project 2 (20) (ToC as above)

3B Processes (20) Policy and Strategy Development

IFAD's policies and objectives Country Strategies (COSOPs)

3C Country Programme (5)

Implications for IFAD’s mandate and strategies

3B Corporate performance (10) Policy and Strategy Development

IFAD’s relevance and niche


Page 43





Regional Strategies Sectoral/Thematic Strategies Resource Allocation Trends

Knowledge Management and Learning

Internal Learning Mechanisms Evaluation-based Learning Influencing the wider world

Organisational Partnerships

Global Regional Sectoral/Thematic

Human Resources Management

Management Processes Staff Consultants

- Impact of COSOP - Synergy: Inter-Project & with TAG - Organisational Partnerships - Programme Policy Influence

Knowledge Management and Learning Project Cycle Management Quality and efficiency Organisational Partnerships Human Resources Management

4. Conclusions (5) - Products - Processes - Issues for the Field Study

4. Conclusions (5) - Design Issues - Impact - Country Programme

3. Conclusions (2) 4. Conclus ions and recommendations to improve future performance(10)

Annexes 1. Country Synopsis (21 papers, each c.10 pages) - Summary Country Strategy (from COSOP) - Country Portfolio Table - Summary of Findings Table - TAGs - Key Issues for the Field (for 10 countries only)

Notes: Approximate page lengths (excluding preliminaries and annexes) in brackets; Italics = Explanatory Notes; Rows indicate (approximately) how issues are presented in a broadly consistent form across the four report -types.


Page 44

Quality assurance

4.33 The IEE team will manage its work for quality in the following ways:

• During the start of the desk review the approach to ratings will be standardized by a process of cross-checking assessments of individual COSOP and project document reviews. By asking all team members to rate one or more common projects and then reviewing the results the team will develop a consistent and comparable data set.

• The team will hold regular review meetings to discuss methodological issues and agree common approaches

• The team leader will review samples of work of the team members at random to identify inconsistencies.

• Draft reports will be read by the ITAD Director responsible for quality assurance (Mr Dane Rogers, who attended IFAD during the tendering process).

• The team will keep all records of interviews and meetings where notes are taken. Together with the structured report documentation and use of databases to store data these will enable verification of source notes and material.

5 Workplan

Timetable with roles and planned inputs

5.1 Table 12 sets out the detailed plan of work for the desk review. The plan covers the period from the end of the Inception Phase. A nominal delivery date of 30th March has been set for the Desk Review Report. The actual date is two months after approval of the Inception Report. Eight members of the evaluation team will be involved in the desk phase, including the team leader and all three senior experts.

5.2 The workplan presents a modified strategy for the work compared with ITAD’s technical proposal. The proposed approach is for five members of the team to be allocated with specific responsibility for the review of strategy, loans and grants in one region each. This regional responsibility will be carried through the country visit stage to preparation of the regional summary reports.

5.3 Members of the team have additional responsibilities as indicated in the table. The total effort is expected to be eight person months.

5.4 The schedule for the remainder of the assignment is shown in Table 13. Specific timing and arrangements for the visits will be worked out after the details of the countries to be visited are known and will be customised to suit the requirements of the country teams. A further change to the original ITAD proposal is that the team will not conduct a pilot visit to the first country. This is considered less necessary in view of the involvement of a larger proportion of the overall team during the desk review. The opportunity will still be taken to refine the methodology during the first visit.

5.5 Dates for submission of progress reports are also shown in Table 13.

Estimated level of effort

5.6 Table 14 presents a revised version of the estimated level of effort by members of the consultant team. This updates Attachment IV to the consultant’s contract. Information for Task 1, the Inception Phase is actual time spent. Figures for the desk review reflect the proposed approach set out in this report. Estimates for tasks 3 to 5 are indicative and depend on the specific countries that are sampled.


Page 45

5.7 Interest was expressed during review of the Inception Report to extend the scope of work to include evaluation of Human Resource Management and Financial Management over and above the current provisions. The level of effort set out in this report fully accounts for the resources made available under the IEE contract and specification in the terms of reference. Furthermore, the decline in value of the dollar since the contract was tendered already reduces the degree of flexibility available to the team. If additional work is desired the IEE team would be pleased to undertake it, but would require additional resources.

5.8 If the team identifies additional studies or follow-up to work undertaken during this evaluation these will be notified in the desk study and final reports.


Page 46

Table 12 Desk study schedule of activities & allocation of person months

Month February March April Person Months

Task Description Week # 6 7 8 9 10 11 12 13 14 15 16 DP JR IH AE MU AB CB NC OI JS TotalMonday 2 9 16 23 1 8 15 22 29 5 12

2 Desk reviewD Poate - Team leader 1.8 1.8Team management, country visit planningJohn Rowley - Senior expert 1.0 1.0Africa I region & Knowledge managementIan Hill - Senior expert 0.8 0.8Asia & Pacific, & HR managementAlison Evans - Senior expert 0.6 0.6Analysis of policy & strategy developmentMary Underwood - Core team member 0.8 0.8Latin America & Caribbean regionAndy Batkin - Core team member 0.9 0.9Africa II region, M&E and PCMChris Barnett - Core team member 1.0 1.0Near East & North Africa, & partnershipsNeil Chalmers - Core team member 0.7 0.7Analysis of portfolio data & field processing

0.0 0.2 0.2

Delivery due # ~ 8 April


Page 47

Table 13 Schedule of activities after the desk review

Month April May June July August

Task Description Week # 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36Monday 29 5 12 19 26 3 10 17 24 31 7 14 21 28 5 12 19 26 2 9 16 23 30

3 Country studiesFirst visit to one country

Set 1 - five countriesSet 2 - four countries

Completion due

4 First draft reportDelivery due

4 Draft final reportDelivery due

5 Final reportDelivery due

6 Progress reports10th April 17th June


Page 48

Table 13 Schedule of activities after the desk review (continued)

2005

Month September October November December Jan Feb Mar Apr

Task Description Week # 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52

Monday 6 13 20 27 4 11 18 25 1 8 15 22 29 6 13 20

3 Country studiesFirst visit to one country

Set 1 - five countriesSet 2 - four countries

Completion due # 17 September

4 First draft reportDelivery due # 31 October

4 Draft final reportDelivery due # 15 December

5 Final reportDelivery due (15 days after EB seminar due in February/March 2005) #

6 Progress reports1st October 10th December


Page 49

Table 14 Estimated level of effort in person monthsTask* Task Task Task Task

Name and role in team 1 2 3 4 5 Total1 Team Leader

1.1 Derek Poate 1.12 1.75 1.50 1.10 1.00 6.47

2 Senior Experts2.1 Ian Hill 0.27 0.80 0.38 1.452.2 John Rowley 0.69 1.00 1.50 0.50 0.19 3.882.3 Alison Evans 0.00 0.60 1.25 1.15 1.00 4.00

3 Other members of the core team3.1 Andrew Batkin 0.73 0.90 2.00 0.50 4.133.2 Chris Barnett 0.00 1.00 1.90 1.00 3.903.3 Charles Chandler 0.003.4 Neil Chalmers 0.00 0.70 0.65 1.353.5 Mary Underwood 0.81 0.80 1.50 3.11

International experts for Country Reviews3.6 Vijaya Shrestha 1.50 1.503.7 Monique Munting 1.50 1.503.8 Munhamo Chisvo 1.50 1.503.9 Ridley Nelson 1.50 1.503.10 Ted Rice 1.50 1.503.11 Chris Mowles 1.50 1.50

4 ITAD Administration4.1 Jane Stanton 0.14 0.23 1.30 0.10 0.10 1.87

TOTAL 3.76 7.78 20.98 4.35 2.29 39.16

*Figures for Task 1 are actuals

Independent External Evaluation of IFAD Inception Report Annex 1

Page 50

Annex 1 Bibliography

Beynon, Jonathan (2003) Poverty efficient aid allocations – Collier/Dollar revisited.

Economics and Statistics Analysis Unit Working Paper 2. ODI, London Coleman, G (1990) ‘Problems in project-level monitoring and evaluation: evidence from one

major agency,’ Journal of Agricultural Economics, 41, pp 149-161 Dabelstein, Niels, Victor Hugo Morales Melendez, Adnan Bashir Khan, Lothar Caviezel,

Osvaldo Feinstein, Gabriel Lombin, Robert van den Berg (2002) External Review of the Results and Impact of IFAD Operations. Copenhagen

Head, Ivan L., Ali Ahmed Attiga, Martin Enrique Pineiro, Yves Rovani and Djibril Sene (1994) The Challenge of Rural Poverty: The Role of IFAD. Report of the Rapid External Assessment of IFAD. Rome

IFAD (1976) Agreement Establishing the International Fund for Agricultural Development. Rome

IFAD (1977) By-laws for the Conduct of Business of IFAD (Revised 1997). Rome IFAD (1977) Financial Regulations of IFAD (Revised 1997). Rome IFAD (1977) Rules of Procedure of the Governing Council (Revised 1997). Rome IFAD (1978) Lending Policies and Criteria (3rd Revision, 1998). Rome IFAD (1998) General conditions for Agricultural Development Financing. Rome IFAD (2000) Strengthening IFAD’s support to the development of effective and efficient

monitoring and evaluation systems methodological study (Interim Paper). Evaluation Committee – Twenty-Fifth Session EC 2000/25/W.P.3/Rev.1 Agenda Item 3(b). Rome

IFAD (2001) Strategic Framework for IFAD 2002-2006. Rome IFAD (2002) Annual Report 2001. Rome IFAD (2002) Progress Report on the IFAD V: Plan of Action (2000-2002). Executive Board –

Seventy-Seventh Session EB 2002/77/R.31 Agenda Item 13. Rome IFAD (2002) Progress Report on the Project Portfolio . Executive Board – Seventy-Fifth

Session EB 2002/75/R.12/Rev.1 Agenda Item 6. Rome IFAD (2003) A methodological framework for project evaluation. Office of Evaluation and

Studies. Rome IFAD (2003) IFAD Evaluation Policy. Executive Board – Seventy-Eighth Session EB

2003/78/R.17/Rev.1 Agenda Item 8(c). Rome IFAD (2003) Progress Report on the Project Portfolio . Executive Board – Seventy-Eighth

Session EB 2003/78/R.16 Agenda Item 7. Rome Powell, Walter W. (1991). Neither Market nor Hierarchy: Network Forms of Organisation.

in Thompson, G, Frances, J, Levacic, R and Mitchell, J. (1991) Markets, Hierarchies and Networks: The Co-ordination of Social Life. London: Sage.

Thompson, G, Frances, J, Levacic, R and Mitchell, J. (1991). Markets, Hierarchies and Networks: The Co-ordination of Social Life. London: Sage.

Watkins, Francis and Csaky, Corinna (2003). Partnerships, Volume 1 - Literature Review. Edinburgh Resource Centre for DFID Evaluation Department.


Page 51

Annex 2 Sampling of countries, projects and grants

Population to be studied

1.1 The primary population for the evaluation consists of 278 projects that became effective during the period 1994-2003. It comprises 44 per cent of all projects ever undertaken by IFAD and includes every recipient country. In December 2003, 196 are still under implementation.

1.2 The primary population contains a mixture of projects at different stages in their implementation cycle. The Scope and Focus of the TOR call for the evaluation to focus on the ‘assessment of the sustainable impact and effectiveness of IFAD’s development initiatives’.49 This infers sampling from projects that can provide evidence of sustainable impact – projects that are completed or close to closure, i.e. the sample must contain an adequate number of mature projects.

1.3 By the end of CY 2004, 137 projects will have closed, representing 43 per cent of the value of the portfolio. Further inspection of the population shows that a total of 189 projects will have been under implementation for at least four years by the time of country visits in July 2004. Table 1 summarizes the population.

Table 1 Population of projects

IFAD Portfolio 1994-2003

Current closing due before end

2004

% of 1994-2003

Number of countries

97 85 88%

Value of portfolio $m

3,600 1,585 44%

Number of projects

278 137 49%

Region

Desired characteristics

• Sample from 5 regions

• Number of countries to reflect relative importance of the region in the IFAD portfolio: (volume of loans and/or number of projects) – random selection

1.4 The IEE has been requested to sample countries across the five regions in proportion to numbers of projects or value of the portfolio. In fact, the distribution of number of projects and value of the portfolio are both closely aligned across the regions. In Africa II, LAC, and NENA, numbers and value are within 1 percentage point of each other. In Africa I, the smaller size of projects results in a slightly higher proportion by number than by value; and in Asia-Pacific the larger average size of project gives a slightly higher proportion by value than number. Rounding the proportions to whole numbers we propose that four countries will be sampled in Africa I and II, LAC and NENA, but 5 in Asia & Pacific, to allow for the higher

49 Terms of Reference para 5


Page 52

average number of projects per country. From this sample, two countries will be visited in each region, totalling 10 in all.

Table 2 Regional distribution of the portfolio

Region IFAD % of Proposed Sample

1994-2003

Total No.

Africa I Number of countries 18 19% 4

Value of the portfolio $m 614 17% Number of projects 55 20% Average no. projects per country 3.1

Africa II Number of countries 18 19% 4 Value of the portfolio $m 704 19% Number of projects 57 21% Average no. projects per country 3.2

Asia & Pacific

Number of countries 17 18% 5


LAC Number of countries 25 26% 4


NENA Number of countries 19 20% 4 Value of the portfolio $m 644 17% Number of projects 50 18% Average no. projects per country 2.6

Total Number of countries 97 100% 21 Value of the portfolio $m 3689 100% Number of projects 278 100% Average no. projects per country 2.9

Country

Take into account:

• Income classification

• Size of country portfolio in 1994-2002

• Size of rural population

• Relative size of agricultural sector

• IFAD lending terms

• HDI index


Page 53

1.5 The terms of reference call for countries to be sampled taking into account the characteristics listed above. In view of limitations in the availability of data to classify countries (for example, data on poverty and in particular rural poverty are not available for all countries) and practical considerations about stratifying such a small population within each region, a simpler approach is proposed. The desired number of countries will be sampled with probability proportional to number of projects in the IFAD portfolio. This is closely correlated to the value of the portfolio, but facilitates a practical process for drawing the sample. The sample will be drawn in the OE office using numbered tokens. The procedure will be available to be witnessed by any interested party within IFAD.

Selection for the country visits

1.6 After the countries have been sampled for the desk review, two countries will be sampled in each region for the country visits, using probability proportional to size random sampling. If one of the selected countries presents security concerns it will be replaced randomly. The team will follow UN guidelines in making this judgement. Countries with security phase III and above on the whole territory or in the capital city will not be selected for country visits. Should security problems arise later in 2004 the concerned countries will be replaced. The IEE team plans to draw the country sample at the same time as the desk sample in order to allow adequate time for organising country teams. Information about the composition of the projects to be studied will also guide the design of field survey questionnaires and methods. We consider that waiting until during the desk review will be detrimental to the planning and implementation of the country visits and could ultimately jeopardise the quality of the evaluation.

Projects

Take into account:

• IFAD’s main sectors for classification

• Different financing modalities and implementation arrangements

• Implementation difficulties arising from security concerns

1.7 The twenty-one sampled countries have an expected total of 61 projects in the population.50 In order to assess the overall composition of the portfolio, basic data will be reviewed for all projects in each sampled country. But in line with the TOR and to keep the volume of work at a manageable level 42 projects will be examined in detail. Two phases of the same project will not be treated as separate projects but as part of the same intervention. In those countries with two or fewer projects, all will be reviewed. If a country has only one project, three projects will be sampled from another country. In those countries with more than two, two will be selected by simple random sample. In the event that the regional sample of countries generates fewer than eight projects the regional sample will be abandoned in its entirety and a fresh sample drawn using the same procedure.

Technical Assistance Grants

1.8 Information about the portfolio of TAGs is more difficult to obtain in IFAD than the equivalent data about loan projects. The approach to studying these will be practical and comprehensive. At country level, all grants to sampled countries will be listed and reviewed in the context of the coherence and synergy of the IFAD activities.

50 On average there are 2.9 projects per country


Page 54

1.9 A list to be provided by the Technical Advisory Department of PMD will be used to draw a simple random sample of 4 TAGs with values greater than US$100,000. The IEE will also draw on the findings of recent evaluations by OE into grants for agricultural research and the IFAD/NGO extended cooperation programme.

Addendum to Annex 2

Report of the sampling procedure and outcome

The random sampling exercise took place on Friday, 30 January, at 11am in OE. Approximately 40 IFAD staff attended the event including representatives from the regional divisions, OE and other organizational units. A number of 21 countries (4 from PA, PF, PL and PN, and 5 from PI) were randomly selected for desk study, with probability of extraction proportional to the number of projects in each country. Then ten countries (2 from each region) were again randomly selected from the initial selection of 21 countries, with probability of proportional to the number of projects in each country. None of the countries selected for field visit presented any particular security concern. Later the same morning, 42 projects from the 21 selected countries and 4 TAGs (taken from a list of IFAD TAGs approved by the Board between 1994 and 2003, with an amount above US$ 100,000) were selected through simple random extraction. Two projects were selected in each country. Project selected in countries that were extracted for field visit will be the object of field validation. In the case of PL, since only one project was present in one of the selected countries (Chile), three projects were selected randomly in Bolivia for desk study. Out of these three, two were randomly selected for field visit. The selection of projects and TAGs enjoyed a smaller audience but included representatives from OE, PMD and Information Training and Development (ITAD) – the IEE service provider. The list of selected countries, projects and TAGs is the following Countries and Projects for Desk Study

Countries in PA Projects in PA South West Rural Development Project Burkina Faso Rural Microenterprise Support Project Village Infrastructure Programme Ghana Rural Enterprise project phase II Smallholder Development Project in the Forest Region Guinea Smallholder Development Project in the North Lower Guinea Village Organisation and Management Project Senegal Agricultural Development Project in Matam phase II

Countries in PF Projects in PF

Eastern Lowlands Wadi Development Project Eritrea Gash-Barka Livestock and Agricultural Development Project Nampula Artisanal Fisheries Project Mozambique PAMA Support Project Intensified Land Use Management Project in the Buberuka Highlands Rwanda Rural Small & Micro-enterprise Promotion Project (phase I) Rural Financial Services Programme Tanzania Agricultural System Development Programme

Countries in PI Projects in PI

Bangladesh Small-scale Water Resource Development Sector Project


Page 55

Aquaculture Development Project Maharashtra Rural Credit Project India Jharkhand-Chattisgarh Tribal Development Programme Arhangai Rural Poverty Alleviation Project Mongolia Rural Poverty Reduction Project Poverty Alleviation W. Terai Nepal Western Uplands Poverty Alleviation Project Pat Feeder Command Area Development Project Pakistan North-Western Frontier Barani Area Development Project

Countries in PL Projects in PL Sustainable Development Project by Beni Indigenous People Small Farmers Tech. Asst. Service Project

Bolivia

Management of Natural Resources in the Chaco & High Valley Region Chile Agricultural Development Project for Peasant Communities and Smallholders

of the IV Region Programme for Rural Development and Reconstruction in the Quiché Department

Guatemala

Rural Developmentn Programme for Las Verapaces Management of Natural Resource in the Southern Highlands Project Peru

Puno Cusco Corridor Development Project

Countries in PN Projects in PN North-west Agric Service Project Armenia Agricultural Services Project East Delta Newlands Agricultural Service Project Egypt West Noubaria Rural Developmtn Project Agric. Resource Management Project in the Gov. of Karak & Tafilat Jordan National Programme for Rangeland Rehab. & Development phase I Southern & Eastern Region Rural Rehabilitation Project Macedonia Agricultural Financial Services Project

Name of TAGs Programme for the Establishment of the Near East and N Africa Management Training in Agriculture Global Cassava Development Strategy Validation & Delivery of New Technologies for Increasing the Productivity of Flood-prone Rice Lands in South and Southeast Asia Programme for Poverty Alleviation and Enhanced Food Availability in W Africa (Yam) Countries and Projects for field visit

Countries in PA Projects in PA South West Rural Development Project Burkina Faso

Rural Microenterprise Support Project Smallholder Development Project in the Forest Region Guinea Smallholder Development Project in the North Lower Guinea

Countries in PF Projects in PF

Mozambique Nampula Artisanal Fisheries Project


Page 56

PAMA Support Project Rural Financial Services Programme Tanzania Agricultural System Development Programme

Countries in PI Projects in PI

Small-scale Water Resource Development Sector Project Bangladesh Aquaculture Development Project Pat Feeder Command Area Pakistan North-Western Frontier Barani Areaa Development Project

Countries in PL Projects in PL Sustainable Development Project by Beni Indigenous People Bolivia

Small Farmers Tech. Asst. Service Project Management of Natural Resource in the Southern Highlands Project

Peru

Puno Cusco Corridor Development Project

Countries in PN Projects in PN North-west Agric Service Project Armenia

Agricultural Services Project East Delta Newlands Agricultural Service Project Egypt West Noubaria Rural Developmtn Project


Page 57

Annex 3 COSOP Review Proforma

COUNTRY: Region: Date of preparation: Date of review by EB: Date of any previous COSOP: Key Question: Does the document tell you what, why and how a particular intervention/initiative should be undertaken to reduce poverty – is it a good guide to strategic decision making? Overall Rating: Document Structure: Does the document comply with current IFAD requirements? Ref: EB 2002/77/R.12

Completely - All sections Mainly - Not more than 2 missing Only partially – more than 2 missing

What is missing?

Document Content Ratings: 5 Very good; 4 Good; 3 Satisfactory; 2 Poor; 1 Very Poor; n/a not applicable QN Question Rating Justification/Explanation/References Relevance: The link with IFAD strategic

framework is

The consistency with the regional strategy is

The country context analysis is The agricultural sector and rural

poverty analysis is

The governance and institutional environment analysis is

The consistency with national poverty reduction/MDG strategies and priorities is

The evidence for consultation and participation of key stakeholders in formulation is

The evidence for local ownership is The treatment of gender issues is The treatment of e nvironment issues is The treatment of empowerment issues

is

The treatment of equity issues is The analysis of UN, WB, regional

development banks, other donor strategies is

The identification of IFAD’s role and comparative advantage in country context is

The relationship of identified needs, constraints and opportunities to the analyses is


Page 58

QN Question Rating Justification/Explanation/References analyses is

Effectiveness The statement of objectives is The logical framework is The indicators of performance are The indicators of impact are The targeting of the needs of the poor

is

The integration of all IFAD activities is The rationale for innovations and

project interventions/investments is

The identification of strategic alliances and partnerships is

The recognition o f opportunities for pro poor policy dialogue is

The analysis of action areas for improving portfolio management is

The identification of opportunities for scaling up is

The strategy design for increasing agricultural productivity and sustainability is

The definition of the participation of IFAD in PRSP process is

Efficiency The evaluation of previous IFAD

experience is

The consistency with country level allocation of human and financial resources is

The opportunities/plans for leveraging additional funds and acting as a catalyst are

The M and E system at programme and strategy level is

The arrangements for communication and dissemination are


Page 59

Annex 4 Methodology, data sources and draft formats for project-related information

This Annex outlines the methodology and sources of data and information the Team will use to address the three inter-linked elements of the Evaluation Framework concerning project impact, project cycle and M&E.

The Desk Study will use three sources of data : documentation relating to project preparation; implementation stage documentation including supervision reports and the Project Status Report summaries, and where available, mid-term reviews, completion and evaluation reports; interviews with CPMs and other IFAD staff.

The Field Study will use four sources of data : Interviews with project management; interrogation of the project monitoring data; interviews with key partners; individual and focus-group interviews with project beneficiaries.

This Annex includes first drafts of the key summary forms and checklists for these sources. Each is outlined below.

Desk study sources of information

i) Loan and Grant Project Documentation review

The Desk Study will summarise and analyse the whole portfolio of loans becoming effective, and TAGs approved, during the study period (January 1994 and December 2002). The data for both portfolio reviews is available on IFAD databases. (See Evaluation Framework 2.1 Resource Allocation).

Following completion of the country and project sampling exercise, the desk study will undertake a thorough review of the sampled loan projects and TAGs.

The end-point of the loan preparation process, and the key official document defining each loan project (and which all sampled projects will have in common) is the Report and Recommendation of the President (RRP). The Desk Study will assess the individual RRP for the sampled loan projects and TAGs, leading to summary rated assessment across the whole sample. Background information will be drawn from the Formulation, Appraisal, Technical Review and OSC documents preceding the RRP, and from discussions with the CPMs and other staff concerned as required. However, because, the RRP represents the formal corporate definition of the project at entry, and marks the conclusion of an extensive process of formulation, quality assurance and loan/grant negotiation, it is appropriate that the RRP, as approved, is the central document of review.

For the desk study project documentation reviews, the Team will seek to achieve consistency of approach across Team members through: i) All Team members reviewing documentation for the same project/s at the outset ii) Requiring clear justifications for assessments made (see draft summary formats below) iii) A consistency-review of all completed Summary forms. During i), the initial joint review by the team of the same project/s, objective indicators for the five-point rating scale will be jointly developed.

ii) Implementation progress

Because different Co-operating Institutions (CIs) provide supervision reports on different formats, it is difficult to compare supervision reports across the whole sample. However,


Page 60

there will be one or more annual Project Status Reports (PSR) for all sampled projects, based in large part on the preceding Supervision Reports. The Desk Study will assess the most recent PSRs for all sampled projects, and will compare these with the preceding Supervision Reports. Where mid-term review, completion, evaluation or other documentation is available for the selected projects, this will also be reviewed.

Draft Summary Form 1 for the loan reviews is appended below. A modified version of the same form will be used for the TAG reviews.

iii) CPM and staff interviews

Clarification and elaboration on issues arising concerning individual projects will be sought from CPMs and other staff as required. Individual and seminar discussion around common issues will also be organised. A preliminary checklist of Impact, PCM and M&E issues for discussion with CPMs and other staff is appended below.

Field study sources of information

i) Project Management interviews This category includes the formally-assigned project managers, their staff and TA, together with directly implementing partners including government, local government and para-statal units and sub-contracted and other implementing entities and NGOs.

ii) Project M&E system It is anticipated that up-to-date project performance data will be more readily available through project office systems, than through supervision reports and other documents available in Rome. A key task for the field-study teams will be to use available MIS data to estimate the number of project beneficiaries for each project component over a given period of time. The draft methodology for this is appended below.

iii) Partners This category includes development partners who have knowledge of IFAD projects or who are familiar with IFAD’s role and activities past or present, but are not directly responsible for project implementation (co-financiers; central and local government units; donors; NGOs; other agencies and key informants). A draft Summary Form (3) for information from these Partner Organisations is appended below.

The views of Cooperating Institutions (CI) with a country presence will be incorporated on this form. However, the main source of CI views and perspective will be through separate interviews with the main CI offices and the recent OE evaluation report.

iv) Beneficiaries The majority of IFAD projects are multi-sectoral, with several different interventions. One IFAD typology lists over fifty categories of project intervention, several of which are likely to be included within a single sampled project. The lag-time for these different types of intervention to impact on poverty varies greatly. For example, effective irrigation should help families in the command area to eat better during the next growing season, whilst much of the benefit of increasing girls school attendance will appear amongst future generations. The linkage between different types of intervention and individual household incomes is similarly diverse. The typical small loan, well-used, enables a new household business and income stream for the borrower’s family, whilst a new bridge facilitates growth throughout the


Page 61

locality and increases opportunities for all, but provides less direct and attributable benefit for any given household.

The best means of measuring and comparing impact information across this diversity of interventions, impact timescales and linkages is to ask beneficiaries themselves how they view the relevance, benefits, sustainability and other key features of the various interventions they are involved in, and to assess their responses on a common standard.

Draft Summary Form 4 (appended below) is for use in individual and group beneficiary interviews, and for combining separate interviews and meetings into an assessment across the whole project component. Combined with the Team estimates of number of project beneficiaries, this beneficiary-perception methodology provides comparable quantitative data on the key dimensions of impact. Across a Field Study sample of 20 projects, with perhaps 100 project components, and interviews representing thousands of beneficiaries, the methodology will provide quantitative data of a statistically meaningful size, which can be aggregated and analysed as required across beneficiaries, intervention types (components), projects, countries, regions and for the field study sample as whole. Indicative comparisons of impact amongst different types of intervention will be possible. Aggregated beneficiary data on key impact questions can be compared with the information obtained through the project management and partner interviews.

Consistency amongst Field Study Teams will be developed through i) Intensive pre-mission briefing and trialling of the forms; ii) Post-mission review of the information and assessments amongst the teams. Within each country field study there is extensive triangulation (pursuing issues with different informants and with different instruments) both to compare perspectives and provide a check on the quality of information acquired.

Note : The Summary Data forms and other checklists below are presented as drafts for information in this Report. They are likely to undergo some refinement and modification during the Desk Study Phase, in terms of content, and in format and layout to enable electronic processing.


Page 62

Note : “Project Component” Because most IFAD projects include a wide variety of interventions and components, it is important to be able to assess the performance of different components within a project, and to be able to aggregate results. Throughout this Inception Report and Annex, “Project Component” refers to the standardised categorisation into thirteen component clusters, as developed by IFAD, and now approved within the Framework for a Results Management System (Executive Board 80th Session). Use of the IFAD-standardised description of component clusters (as opposed to the varied specification of components, sub-components and outputs in RRPs and preparation documents) enables comparison and aggregation of different items of information and impact by component across the whole sample.

The 13 standardised component clusters are :

1. Human capital development for rural households 2. Institutional development (institutional transformation, policy change) 3. Management and co-ordination (including M&E) 4. Marketing, storage and processing 5. Natural resources management 6. Research, extension and training 7. Rural community development 8. Rural enterprise development 9. Rural financial services 10. Rural infrastructure development 11. Smallholder agriculture development 12. Smallholder livestock development 13. Small-scale fisheries development

(During the Field Study, any components or activities which do not fit comfortably within the new standardised component categorisation will be noted.) The following Table provides a tabulated summary of the data sources to be used in addressing points 1.2, 2.4 and 2.5 of the Evaluation Framework.


Page 63

Draft Summary Form 1 Desk Study: Project and Grant Review

(For sampled loan projects and, with some modification, for TAGs) KEY DATA Country Project Title Effectiveness Date Total Project Cost IFAD Loan Amount Co-financiers Project Duration Sector Co-operating Institution Time from Inception to RRP approval (months) Estimated preparation cost ($000) Aggregation of project-defined components and components into standard Component Clusters

Component 1: Includes :- Component 2: Includes :- Component 3: Includes :- Component 4: Includes:

Variations : Significant changes during implementation to Project Design, Budget, Execution/Implementation arrangements or other key aspects of the project


Page 64

Total Project Financing by Standard Component and Expenditure Category Project Total C1 C2 C3 C4 Standard Component Cluster : Total Budget ($000) - Infrastructure/Civil Works - Equipment/supplies - TA & systems development - Training & Capacity Building - Bank/MFI credit fund - Other

IFAD financing by Standard Component and Expenditure Category

Project Total C1 C2 C3 C4 Standard Component Cluster : Current Total Budget ($000) - Infrastructure/Civil Works - Equipment - TA & systems development - Training & Capacity Building - Bank/MFI credit fund - Other


Page 65

Ratings51 5 Very Good 4 Good 3 Satisfactory 2 Poor 1 Very Poor n/a Not applicable

Rating Explanation of Rating (with references) 1 The congruence of the project approach with the COSOP (or equivalent at the time of

project inception) is:

2 The congruence of the project with the regional strategy (or equivalent at the time of project inception) is:

3 Poverty Analysis : The analysis of the reasons why the beneficiaries and/or the project area are poor is:

4 Institutional Analysis : The analysis of institutional configurations and trends is: 5 The analysis of lessons from previous loans and activities (where they exist) is: 6 The analysis of how other people in a similar situation in the country have escaped

poverty is:

7 Participation : The evidence that beneficiaries were directly involved in the project design is:

8 Targeting – Selection : The social analysis and evidence justifying the selection of beneficiaries and/or the project area is:

9 Targeting – Women : The gender analysis, assessment of women’s needs, role and benefits, and incorporation of a clear means to reach women in the project design is:

10 Objectives : The clarity, realism and measurability of the project objective/s is: 11 Outputs : The clarity, realism and measurability of the component outputs is: 12 Indicators : The definition, clarity and practicality of the objective and output

indicators is:

13 Monitoring : The definition and practicality of systems and responsibilities for monitoring activities and outputs is:

51 Objective Indicators will be developed jointly by the reviewers, as part of the initial consistency exercise (See " Desk Study - Loan and Grant Project Documentation review" above.)


Page 66

14 Risk Analysis : The assessment of significant risks and assumptions, and the presentation of mitigating measures is :

15 Logic: The consistency and quality of the project logic is: 16 Environment : The assessment of expected environmental impacts, risks and necessary

mitigations is:

17 Sustainability : The assessment of the likelihood of sustainability and the specification of measures to promote sustainability is:

18 Replicability : The assessment of the potential for replicability and the specification of measures to promote extension and up-scaling is:

19 Cost-effectiveness : The assessment of economic efficiency, expected rates of return or cost-effectiveness is:

20 Innovation : The assessment of the applicability, benefits and costs of new technologies and techniques to be introduced is:

21 Quality: The content, dialogue process, and overall impact of the QA process during preparation was:

22 Management : The definition of roles, relationships and responsibilities for project execution and implementation is:

23 Management : The analysis of current capacity and commitment amongst intended executors/implementers, and specification of requisite capacity building measures is:

24 Partners : The definition of the role, responsibilities and expected contribution of other project partners is:

25 Completeness: The consistency of the RRP with the IFAD model Proforma at the time was:

26 Supervision Reports: The quality of supervision reports as received is : 27 Supervision Actions: The evidence of action subsequent to supervision reports is: 28 PSR: The quality of the PSRs, compared to preceding supervision and other

monitoring reports is:

29 The explanation and justification for the variations and actions proposed during supervision is:

30 The actioning of the proposed variations was: 31 MTR: (when available): The overall quality is: 32 PCR (where available): The overall quality is: 33 Evaluation Report (where available): The overall quality is:


Page 67

Desk Study: Project Progress Ratings (for validation during Field Study ) 1= Problem Free 2= Minor Problems 3= Major problems but improving 4 = Major problems and not improving

DS: Rating in most recent PSR (Date:......................)

FS: Validation Rating

Explanation of Field Study Validation Rating

1. Achievement of physical targets 2. Technical Assistance Progress 3. Development Impact 4. Expected Benefits 5. Beneficiary Participation 6. Institution Building 7. Overall Assessment of Project Performance


Page 68

Desk Study : Preliminary checklist of Impact, PCM and M&E issues for discussion with CPMs and other staff

- Do global and country strategies, project and TAG designs, sufficiently address reform of state organisational and institutional functioning? - Is IFAD in a position to engage in national policy dialogue and institutional reform? - Is the sum of IFAD projects and grants in a country greater than the sum of the parts? Is there country programme coherence and synergy? - What are the advantages and disadvantages of IFAD and non-IFAD initiated projects ? - Are the resources committed to project preparation appropriate? - Are projects sufficiently focused? - Should IFAD make it’s support to countries more conditional? - What are the largest and commonest implementation problems? - Is IFAD's need to work through government, towards civil society strengthening objectives a constraint to effectiveness? - What are the strengths and weaknesses of supervision, and how can it be improved? - How essential is country-presence? - What impact has the new-methodology evaluations had?


Page 69

Draft Summary Form 2 Field Study: Project Management Interviews

- This form is for summarising data and information obtained from project management, their staff, TA and systems, and other involved or sub-contracted implementors. - Agencies and individuals with knowledge of the project, but who are not directly responsible for implementation should be summarised on the “Project Partner Interviews Summary Form” - Where sampled projects are closed, the Team will seek to convene a meeting of those mostly closely involved in implementation who are still available in-country. - Enter codes for the whole project and/or each component as appropriate - Agreement and disagreement amongst implementors on a topic will be explored and assessed in deriving the summary rating. - Assess components according to standardised IFAD component Cluster (as below) not official project component. - Use n/a for not applicable IFAD Standardised Component Clusters 1. Human capital development for rural households 2. Institutional development (institutional transformation, policy change) 3. Management and co-ordination (including M&E) 4. Marketing, storage and processing 5. Natural resources management 6. Research, extension and training 7. Rural community development 8. Rural enterprise development 9. Rural financial services 10. Rural infrastructure development 11. Smallholder agriculture development 12. Smallholder livestock development 13. Small-scale fisheries development


Page 70

KEY DATA Country Project Title Effectiveness Date Total Project Cost IFAD Loan Amount Co-financiers and amounts -

- -

Project Duration Co-operating Institution Coding Project C1 C2 C3 C4 Justification/Key issues 1 How many people are likely to

benefit directly? See Methodology Attached (By component and total for the project)

2 How many people were originally expected to benefit?

As s tated in RRP

3 M&E evidence suggests that, as a share of those involved, women are:

5 The great majority 4 The majority 3 About half 2 The minority 1 Very few

4 Evidence for the likelihood of sustainability is:

4 High 3 Substantial 2 Modest 1 Low

5 Evidence of increased production and/or productivity is:


6 Evidence for the likelihood of replicability is:

4 High 3 Substantial


Page 71

Coding Project C1 C2 C3 C4 Justification/Key issues 2 Modest 1 Low

7 Evidence for the successful introduction of new technologies and techniques is:


8 Evidence of sustainable impact on national/local state or sector organisation and functioning is:


9 The evidence of effective policy dialogue is:


10 Evidence of sustainable impact on private sector development is:


11 Evidence of sustainable impact on civil society organisation and functioning is:


12 Evidence that other IFAD projects in the country have benefited this one is:


13 Evidence that IFAD TAGs have benefited this project is:


14 Evidence that the impact on the natural environment is:

3 Positive 2 Little impact 1 Negative

15 Have there been unforeseen impacts? Specify


Page 72

Coding Project C1 C2 C3 C4 Justification/Key issues

15 How appropriate does project management think the design is for the target group:


16 Project management feels the extent of beneficiary involvement in the design is:


17 How satisfied is project management with component outputs and targets:


18 Does project management feel the project has the right number of components:

3 Not enough 2 About right 1 Too Many

19 Which components does project management feel will be most successful:

Rank in order from 1=Best to n=Weakest

20 Were project and component budgets right in the original design?

3 Too big 2 About right 1 Too small

21 Were project and component timescales right in the original design?

3 Too long 2 About right 1 Too short

22 Agreed time over-runs to date 3 0-10% of plan 2 10% to 25% of plan 1 > 25% of plan

23 Agreed budget variations to date: Planned budget vired in or out: 3 0-10% 2 10-25% 1 >25%

24 Has the planned design of the project 3 Major changes


Page 73

Coding Project C1 C2 C3 C4 Justification/Key issues or components been significantly altered during implementation:

2 Minor changes 1 No change

25 Have the planned management and implementation arrangements been significantly altered during implementation:

3 Major changes 2 Minor changes 1 No change

26 Have their been major or ongoing disputes amongst implementors?

3 Major 2 Some 1 Minor

27 Progress to date on achieving targets and outputs is:

5 Well ahead of target 4 Ahead of target 3 On target 2 Behind target 1 Well behind target

28 How effective is the project MIS in producing reliable, up-to-date, key data and reports?

5 Very good 4 Good 3 Adequate 2 Poor 1 Very poor

29 Has Project Management had effective help in developing the MIS:

3 Yes 1 No 0 No additional help required

30 Project management views the supervision arrangements as:


31 How satisfied is project management with the support, communication and performance of the CI supervisors?


32 How satisfied is project management with the support, communication and

5 Very good 4 Good


Page 74

Coding Project C1 C2 C3 C4 Justification/Key issues performance of IFAD staff? 3 Adequate

2 Poor 1 Very poor


Page 75

Field Study Draft Methodology for Estimating Beneficiary Numbers

This methodology is designed to provide a fair and practicable estimate of the number of people directly benefiting from the sampled IFAD projects and their component. Indirect beneficiaries resulting from: replication; enhanced growth and dynamic effects; or the outcome of policy or institutional reform are not included. This methodology does not attempt to scale benefits; it is simply concerned to estimate beneficiary numbers, however large or small the benefit accruing to each of them. Information on the size of benefits and other aspects of impact comes from the interviews with beneficiaries, project management and partners. The numeraire is “Individuals, who benefit directly for a period, up to 20 years from project start-up” The steps are as follows: 1. For each project component or activity, obtain the best available estimate of number of actual beneficiaries to date, from the project MIS, from field estimation or population data. 2. Adjust for household beneficiaries 3. Adjust for IFAD contribution to financing of the component or activity 4. Adjust to completion of the project 5. Adjust in the light of good evidence on sustainability 6. Add sub-components/activities to give a total for each standardised IFAD component in the project 7. Add components to give a total for the project Component or Activity Guidance notes 1. Estimate of number of beneficiaries to date

- From careful study of project monitoring data - For public infrastructure (roads, water supply, clinics etc.) estimate total number of users from population data, observation, key informants etc. - Exclude official institutional, organisational and policy reform components and activities.

2. Multiply by average household size as necessary

- Only use for activities/sub-components where it is reasonable to assume a family benefits from the activity of the individuals involved (e.g. extension beneficiaries, credit borrowers) - For education, estimate pupil numbers only.

3. Adjust for IFAD share of component financing

- Adjust by the share of IFAD financing for the activity or component. - Assess whether IFAD and co-financier financing are stable shares over time, and estimate accordingly.

4. Adjust to project completion - Assume that the component runs to completion at the current rate of progress and coverage, unless there are good reasons to think it will not. - Assume that the whole currently-allocated budget is spent - Ignore natural population increase (e.g. for the population using a new bridge) - Avoid double counting. For example, if all public capital investment is going into one area, and the total number of beneficiaries in the area has already been estimated in 1 above, do not adjust numbers if the same area and population will receive more bridges,


Page 76

water pumps etc. for the rest of the project period. 5. Where there is good evidence of sustainability, adjust for estimated number of beneficiaries up to 20 years maximum.

- “Good evidence” would include, for example, “The MFI is well-established and well-managed, with excellent systems and recovery rates”; “The school/well/irrigation scheme is well-built, and there is evidence that o&m is well-done” - Where good evidence of sustainability exists which will benefit more individuals over time, estimate the number of additional beneficiaries. - The maximum period for the estimation is 20 years from project start-up. (Three times the average project period).

6. Total activities/sub-components to obtain a total for each standard IFAD component in the project

- Use standard IFAD component typology (not necessarily the same as official project components) and enter on the FS Summary Form for Project Management Interviews

7. Total each standard component for the whole project

- Sum components and enter on the FS Summary Form for Project Management Interviews


Page 77

Draft Summary Form 3 Field Study: Partner Interviews

- This form is for summarising information obtained from development partners who have knowledge of IFAD projects or who are familiar with IFAD’s role and activities past or present, but are not directly responsible for project implementation (co-financiers; central and local government units; donors; NGOs; other agencies and key informants at national or field level). - Agreement and disagreement amongst partners on a topic will be explored and assessed in deriving the summary Partner rating. - Where CI’s have a country presence, their views will be included on this form. - Where sampled projects are closed, the Team will seek to meet with those who had knowledge of the project during implementation. - Use n/a for not applicable IFAD Standardised Component Clusters 1. Human capital development for rural households 2. Institutional development (institutional transformation, policy change) 3. Management and co-ordination (including M&E) 4. Marketing, storage and processing 5. Natural resources management 6. Research, extension and training 7. Rural community development 8. Rural enterprise development 9. Rural financial services 10. Rural infrastructure development 11. Smallholder agriculture development 12. Smallholder livestock development 13. Small-scale fisheries development KEY DATA Country Project Title Effectiveness Date


Page 78

Coding Project C1 C2 C3 C4 Justification/Explanation/Key issues Partners feel that the targeting of the

project to the poor/poor areas is: 4 High 3 Substantial 2 Modest 1 Low

Partners feel that the extent to which the project is addressing priority needs of the poor is:


Partners feel that, as a share of those involved, women are:

5 The great majority 4 The majority 3 About half 2 The minority 1 Very few

Partners feel that the likelihood of sustainability is:


Partners feel that the likelihood of increased production and/or productivity is:


Partners feel that the likelihood of replicability is:


Partners feel that the likelihood of successful introduction of new technologies and techniques is:


Partners feel that sustainable impact on national/local state or sector organisation and functioning is likely to be:


Partners feel that the likelihood of 4 High


Page 79

Coding Project C1 C2 C3 C4 Justification/Explanation/Key issues effective policy dialogue is: 3 Substantial

2 Modest 1 Low

Partners feel that sustainable impact on private sector development is:


Partners feel that sustainable impact on civil society organisation and functioning is:


Partners feel that impact on the natural environment will be:

3 Positive 2 Little impact 1 Negative

Have there been unforeseen impacts? Specify

On the number of project components, partners feel the project has: right number of components:

3 Not enough 2 About right 1 Too Many

Which components do partners feel are the most successful?

Rank in order from 1=Best to n=Weakest


Page 80

Draft Summary Form 4 Field Study: Project Beneficiaries

- This form will be used to collect the views arising from both group (participatory) and individual discussions with beneficiaries of each project component. - Beneficiaries are defined here as :- "People the project aimed to reach and who have been involved in project activities to date." - The questions are presented here in stylised form. In practice, questions will be adapted and sequenced to fit the type of component; its private/public goods nature; the interviewees and the situation. - Where, within a group interview, there are differences of opinion which are not rateable within the categorisation shown, separate forms can be completed for the conflicting sets of views. IFAD Standardised Component Clusters 1. Human capital development for rural households 2. Institutional development (institutional transformation, policy change) 3. Management and co-ordination (including M&E) 4. Marketing, storage and processing 5. Natural resources management 6. Research, extension and training 7. Rural community development 8. Rural enterprise development 9. Rural financial services 10. Rural infrastructure development 11. Smallholder agriculture development 12. Smallholder livestock development 13. Small-scale fisheries development Basic Data Country Project Component Cluster Location Date Base year for comparison – since start of project; since last year; since a nominated event/time

Data from groups Data from households or individuals Nature of the group: Numbers of males/females Socio-economic characteristics of the group

Nature of the respondent: Individual/ Household representative/ household members Socio-economic characteristics of the household

Data from groups Data from households or individuals Physical and financial assets Has community infrastructure changed? 4-high; 3-substantial; 2-modest; 1-negligible Code transport, roads, communication, storage etc. Have household assets changed ? 4-high; 3-substantial; 2-modest; 1-negligible Code houses, bicycles, radios etc Has access to financial services changed? 4-high; 3-substantial; 2-modest; 1-negligible

Physical and financial assets Has community infrastructure changed? 4-high; 3-substantial; 2-modest; 1-negligible Code transport, roads, communication, storage etc. Have household assets changed ? 4-high; 3-substantial; 2-modest; 1-negligible Code houses, bicycles, radios etc Has access to financial services changed? 4-high; 3-substantial; 2-modest; 1-negligible


Page 81

Data from groups Data from households or individuals Access to and use of social infrastructure Has people’s access to potable water changed? 4-high; 3-substantial; 2-modest; 1-negligible Code nature of water source etc. Has people’s access to health services changed ? 4-high; 3-substantial; 2-modest; 1-negligible Code nature of service etc What benefits does the group think have arisen? Code nature of benefits Has people’s access to education services changed ? 4-high; 3-substantial; 2-modest; 1-negligible Code nature of facility etc What benefits does the group think have arisen? Code nature of benefits

Access to and use of social infrastructure Has your access to potable water changed? 4-high; 3-substantial; 2-modest; 1-negligible Code nature of water source etc. Has your access to health services changed ? 4-high; 3-substantial; 2-modest; 1-negligible Code nature of service etc What health benefits have you or any member of your family gained? Code nature of benefits Has your access to education services changed ? 4-high; 3-substantial; 2-modest; 1-negligible Code nature of facility etc What educational (or literacy) benefits have you or any member of your family gained? Code nature of benefits

Data from groups Data from households or individuals Empowerment and social capital Have any organizations or institutions changed in the community? Yes/No Code nature of organizations/institutions Does the group think that there has been any change in the way people get involved with public authorities and private organizations? Yes/No Code nature of involvement

Empowerment and social capital As a result of your involvement in this activity do you belong to any organizations or institutions? Yes/No Code nature of organizations/institutions As a result of your involvement in this activity, do you feel more confident to speak out and assert your rights? 4-Much more confident; 3-A bit more confident; 2-About the same; 1-Less confident As a result of your involvement in this activity, do you feel that your skills or knowledge, or those of people in your family, have already, or will in the future: 4-Increased substantially; 3-Increased moderately; 2-Stayed about the same; 1-Decreased

Data from groups Data from households or individuals Food security Has the farm technology or practice in this community changed? Code nature of the change Has the nature and level of farm production changed? Code food crop/livestock; cash crop/livestock; other

Food security Has your farm technology or practice changed? Code nature of the change Has your farm production changed? Code food crop/livestock; cash crop/livestock; other Can you quantify the annual change? Code area/output/crop; number/livestock Do you feel you and your family: 5-Eat much better; 4-Eat a bit better; 3-No change; 2-Eat worse


Page 82

Data from groups Data from households or individuals Environment and CPR Has the status of the natural resource base change in the community? Code nature of change Have arrangements to manage common pool resources and deal with conflict changed? Code nature of change

Environment and CPR Has there been any change to the natural resources to which you have user rights? Code nature of change Have you been affected by any arrangements to manage common pool resources and deal with conflict? Code nature of change

Data from groups Data from households or individuals Income and poverty impact As a result of the community’s involvement in this activity, do you feel family income has already, or will in the future: 4-Increase substantially; 3-Increase moderately; 2-Stayed about the same; 1-Decreased In what ways has family income increased? Code food consumption, cash income, employment, time savings, other, etc. As a result of people’s involvement in this activity, do you feel people’s lives overall are already, or will be in the future: 4-Substantially better; 3-Moderately better; 2-About the same; 1-Worse Have the benefits accrued uniformly across the community, or have some specific groups benefited less or more? Code groups and extent of benefit

Income and poverty impact As a result of your involvement in this activity, do you feel your family income has already, or will in the future: 4-Increase substantially; 3-Increase moderately; 2-Stayed about the same; 1-Decreased In what ways has your income increased? Code food consumption, cash income, employment, time savings, other, etc. As a result of your involvement in this activity, do you feel your lives overall are already, or will be in the future: 4-Substantially better; 3-Moderately better; 2-About the same; 1-Worse

Data from groups Data from households or individuals Attribution and other evaluation criteria Attribution Do you feel the benefits you've described are mainly the result of your involvement in this activity, or do you think you would have got similar benefit some other way? 4-Benefits are entirely due to this project; 3-Benefits are mainly due to this project; 2-Would perhaps have benefited without the project; 1-Would definitely have benefited without the project Relevance Of all the help you need to improve your lives, how important is this activity to you? 4-Top priority for nearly all of us; 3-High priority for many of us; 2-Quite important for some of us; 1-Not really a priority for most people Sustainability Looking to the future, after the project has finished, will you continue this activity/do you feel the benefit to you of this activity is:

Attribution and other evaluation criteria Attribution Do you feel the benefits you've described are mainly the result of your involvement in this activity, or do you think you would have got similar benefit some other way? 4-Benefits are entirely due to this project; 3-Benefits are mainly due to this project; 2-Would perhaps have benefited without the project; 1-Would definitely have benefited without the project Relevance Of all the help you need to improve your lives, how important is this activity to you? 4-Top priority; 3-High priority; 2-Quite important; 1-Not really a priority for most people Sustainability Looking to the future, after the project has finished, will you continue this activity/do you feel the benefit to you of this activity is: 4-Certain to continue; 3-Likely to continue; 2-Likely to stop; 1-Certain to stop


Page 83

Data from groups Data from households or individuals 4-Certain to continue; 3-Likely to continue; 2-Likely to stop; 1-Certain to stop Poverty targeting Compared to other people in this area, do you think the majority of people involved in this activity are: 5- Much poorer; 4-Poorer; 3-About the same; 2-Richer; 1-Much richer Women Of all the people directly involved in this activity, do you think women are: 5-The great majority; 4-The majority; 3-About half; 2-The minority; 1-Very few Innovation Are the activities you’ve been involved with in this activity: 4-Very new to most of you; 3-Mostly new to many of you; 2-Mostly familiar to many of you; 1-Very familiar to most of you Replication Looking forward, how likely do you think it is that other people in the area will adopt what you’re doing under this activity? 4-Very likely; 3-Likely; 2-Unlikely; 1-Very unlikely

Poverty targeting Compared to other people in this area, do you think the majority of people involved in this activity are: 5- Much poorer; 4-Poorer; 3-About the same; 2-Richer; 1-Much richer Women Of all the people directly involved in this activity, do you think women are: 5-The great majority; 4-The majority; 3-About half; 2-The minority; 1-Very few Innovation Are the activities you’ve been involved with in this activity: 4-Very new to most of you; 3-Mostly new to many of you; 2-Mostly familiar to many of you; 1-Very familiar to most of you Replication Looking forward, how likely do you think it is that other people in the area will adopt what you’re doing under this activity? 4-Very likely; 3-Likely; 2-Unlikely; 1-Very unlikely

Data from groups Data from households or individuals Unexpected Have any things happened you hadn’t expected when you first became involved in this activity?

Unexpected Have any things happened you hadn’t expected when you first became involved in this activity?


Page 84

Draft Summary Form 5 Overall 12-Point Project Assessment

This Form summarises collects overall assessments on ten key points of sampled projects from CPMs (collected during the Desk Study), Project Management (collected during the Field Study) and the IEE Team. Ratings:

4= High; 3= Substantial; 2= Modest; 1= Negligible

IFAD/CPM’s Rating (For all 42 projects)

Project Mant’s Rating (For 20 Field Study Projects only)

IEE Rating Comments

Identification 1 - Relevance: The extent to which the project fits country development priorities, IFAD

strategy and beneficiary needs is:

Design 2 - Targeting: The extent to which the design targeted the right people with appropriate

activities was:

3 - Complexity: The extent to which the design was implementable, without major change, was:

4 - Riskiness: The quality of the assessment of risks and mitigations in the design was: Implementation 5 - Efficiency: the economical conversion of inputs/resources into outputs and outcomes

has been:

6 - Performance by borrower: Overall, the performance of project management and implementors has been:

7 - Performance by IFAD & CI: Overall, the performance of IFAD and the supervisors has been:

8 - Effectiveness: (taking into account their relative importance) achievement of development intervention’s objectives were , or are expected to be:

Outcomes 9 - Impact on poverty: Considering both the total number of people benefiting and the

size of the benefits accruing, impact on poverty is likely to be

10 - Impact on women: specific impact on women was, or is likely to be: 11 - Sustainability of Impact: The likelihood of project-induced benefits continuing after

project closure is:

12 - Sustainability of institutional impact: The likelihood of institutional (state, private, civil) changes induced by the project, continuing after project closure is:


Page 85