document of the world bank · document of the world bank ... pkh program keluarga harapan ... pto...

Document of The World Bank

Report No: ICR00003359

IMPLEMENTATION COMPLETION AND RESULTS REPORT (TF-97410)

ON A

GRANT

IN THE AMOUNT OF US$73.323 MILLION

TO THE

REPUBLIC OF INDONESIA

FOR A

ADDITIONAL FINANCING TO PNPM-RURAL III FOR

PNPM GENERASI SCALE-UP

June 9, 2015

Global Practice for Social, Urban and Rural Resilience Indonesia Country Management Unit East Asia and Pacific Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS

(Exchange Rate Effective June 9, 2015)

Currency Unit = Rupiah US$ 1.00 = IDR 13,330

FISCAL YEAR

January 1 – December 31

ABBREVIATIONS AND ACRONYMS

AF Additional Financing AM Aide Memoire ASF Administrative Services Firm AusAid Australian Aid BOS Bantuan Operasionel Sekolah (School Operational Grants) BSM Bantuan Siswa Miskin (Scholarships for Poor Students) BAPPENAS Badan Perencanaan Pembangunan Nasional (National Planning Board) CCT Conditional Cash Transfers CPS Country Partnership Strategy CDD Community-driven Development CSO Civil Society Organization DFAT Department of Foreign Affairs and Trade EA Environmental Assessment ECED Early Childhood Educational Development EIRR Economic Internal Rate of Return FM Financial Management GoI Government of Indonesia HNP Health, Nutrition, and Population IBRD International Bank for Reconstruction and Development ICR Implementation Completion and Results Report IDR Indonesian Rupiah IE Impact Evaluation IFR Interim Financial Report IGSES Implementation Guidelines for Social and Environmental Safeguards IOI Intermediate Outcome Indicator IP Indigenous Peoples ISM Implementation Support Mission ISR Implementation Status and Results Report IGSES Implementation Guidelines for Social and Environmental Safeguards JMC Joint Management Committee KDP Kecamatan Development Project (IBRD Loan 4330-IND) KPI Key Performance Indicator MMR Maternal Mortality Rate MCA-I Millennium Challenge Account-Indonesia

M&E Monitoring and Evaluation MDG Millennium Development Goal MDST Musyarawah Desa Serah Terima (Village Accountability Meeting) MDTF Multi Donor Trust Fund MIS Management Information System MoEC Ministry of Education and Culture MoF Ministry of Finance MoH Ministry of Health MoHA Ministry of Home Affairs MoV Ministry of Villages, Disadvantaged Areas and Transmigration MP3KI Masterplan for Acceleration of Poverty Reduction NGO Non-Government Organization NMC National Management Consultant NTB Nusa Tenggara Barat Province NTT Nusa Tenggara Timur Province PAD Project Appraisal Document PIU Project Implementation Unit PKH Program Keluarga Harapan (Hopeful Family Program) PDO Project Development Objective PMD Pemberdayaan Masyarakat Desa (Directorate for Village Community

Empowerment in the Ministry of Home Affairs) PMS Performance Monitoring Survey PNPM Program Nasional Pemberdayaan Masyarakat (National Community

Empowerment Program) PRODIS Program Disbursement (MIS application) PROGEN Program Generasi (MIS application) PROTAK Program Tahapan Kegiatan (MIS application) PSF PNPM Support Facility PTO Petunjuk Teknis Operasional (Operations Manual) RF Results Framework RISKESDAS Riset Kesehatan Dasar (Basic Health Survey) RPJMN Rencana Pembangunan Jangka Menengah Nasional (National Medium Term

Development Plan) SMS Short Message Service Sosbud Directorate of Traditional and Community Social and Cultural Empowerment

(Pemberdayaan Adat dan Sosial Budaya Masyarakat) SP2D Disbursement Order SUSENAS Survei Sosial Ekonomi Nasional (National Social Economic Survey) TA Technical Assistance TF Trust Fund TPK Tim Pengelola Kegiatan (Village Implementation Team) UPK Unit Pengelola Keuangan (Financial Management Unit)

Regional Vice President: Axel van Trotsenburg

Country Director: Rodrigo A. Chaves

Senior Global Practice Director Ede Jorge Ijjasz-Vasquez

Acting Practice Manager: Kevin Tomlinson

Project Team Leader: Robert Wrobel

ICR Team Leader: Robert Wrobel

INDONESIA ADDITIONAL FINANCING TO PNPM-RURAL III FOR PNPM GENERASI

SCALE-UP (P122032, TF097410)

CONTENTS

Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph

1. Project Context, Development Objectives and Design ................................................... 12. Key Factors Affecting Implementation and Outcomes .................................................. 63. Assessment of Outcomes .............................................................................................. 134. Assessment of Risk to Development Outcome ............................................................. 195. Assessment of Bank and Borrower Performance ......................................................... 206. Lessons Learned............................................................................................................ 237. Comments on Issues Raised by Grantee/Implementing Agencies/Donors................... 24Annex 1: Project Costs and Financing .............................................................................. 25Annex 2: Outputs by Component ..................................................................................... 26Annex 3: Economic and Financial Analysis ..................................................................... 34Annex 4: Grant Preparation and Implementation Support/Supervision Processes ........... 35Annex 5: Beneficiary Survey Results ............................................................................... 37Annex 6: Stakeholder Workshop Report and Results....................................................... 38Annex 7: Summary of Borrower’s ICR and/or Comments on Draft ICR ........................ 40Annex 8: Comments of Co-financiers and Other Partners/Stakeholders .......................... 44Annex 9: List of Supporting Documents .......................................................................... 46

A. Basic Information

Country: Indonesia Project Name:

ID-TF ADDITIONAL FINANCING OF PNPM RURAL III FOR GENERASI (COMM.CCT) SCALING-UP PROGRAM

Project ID: P122032 L/C/TF Number(s): TF-97410

ICR Date: 04/28/2015 ICR Type: Core ICR

Lending Instrument: SIL Grantee: GOVERNMENT OF INDONESIA

Original Total Commitment:

USD 10.20M Disbursed Amount: USD 73.32M

Revised Amount: USD 73.32M

Environmental Category: B

Implementing Agencies: DG Community and Village Empowerment, Ministry of Home Affairs (MOHA)

Cofinanciers and Other External Partners: B. Key Dates

Process Date Process Original Date Revised / Actual

Date(s)

Concept Review: 11/18/2009 Effectiveness: 08/16/2010

Appraisal: 02/10/2010 Restructuring(s): 11/08/2013

Approval: 08/16/2010 Mid-term Review:

Closing: 12/31/2012 12/31/2014 C. Ratings Summary C.1 Performance Rating by ICR

Outcomes: Moderately Satisfactory

Risk to Development Outcome: Moderate

Bank Performance: Moderately Satisfactory

Grantee Performance: Moderately Satisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings

Quality at Entry: Moderately Unsatisfactory

Government: Moderately Satisfactory

Quality of Supervision: Satisfactory Implementing Agency/Agencies:

Moderately Satisfactory

Overall Bank Performance:

Moderately SatisfactoryOverall Borrower Performance:


C.3 Quality at Entry and Implementation Performance Indicators

Implementation Performance

Indicators QAG Assessments

(if any) Rating

Potential Problem Project at any time (Yes/No):

No Quality at Entry (QEA):

None

Problem Project at any time (Yes/No):

No Quality of Supervision (QSA):

None

DO rating before Closing/Inactive status:


D. Sector and Theme Codes

Original Actual

Sector Code (as % of total Bank financing)

Health 34 34

Other social services 33 33

Primary education 33 33

Theme Code (as % of total Bank financing)

Child health 40 40

Other human development 30 30

Other social development 30 30 E. Bank Staff

Positions At ICR At Approval

Vice President: Axel van Trotsenburg James W. Adams

Country Director: Rodrigo A. Chaves Stefan G. Koeberle

Practice Manager/Manager:

Kevin A Tomlinson Franz R. Drees-Gross

Project Team Leader: Robert Wrobel John Victor Bottini

ICR Team Leader: Robert Wrobel

ICR Primary Author: Paul Gerard M Boon

F. Results Framework Analysis

Project Development Objectives (from Project Appraisal Document) The project development objective (PDO) is for villagers in PNPM-Rural locations to benefit from improved socio-economic and local governance conditions. Revised Project Development Objectives (as approved by original approving authority) (a) PDO Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1 : Immunization coverage for 12-23 month olds increases by 10% points from 65% in 2007 to 75% in 2010.

Value quantitative or Qualitative)

65.00 75.00 75.00

Date achieved 09/03/2007 12/31/2010 12/31/2014

Comments (incl. % achievement)

The randomized impact evaluation found that PNPM Generasi's impacts were largest in locations with low baseline health and education performance. As a result, government prioritized sub-districts with low baseline health and education performance for proj

Indicator 2 : Prenatal care visits increase by 10% points from 71% in 2007 to 81% in 2010. Value quantitative or Qualitative)

71.00 82.00 82.00

Date achieved 09/03/2007 12/31/2010 12/31/2014



Indicator 3 : Deliveries assisted by trained professionals increase by 10% points from 67% in 2007 to 77% in 2010.

Value quantitative or Qualitative)

67.00 77.00

Date achieved 09/03/2007 12/31/2014



(b) Intermediate Outcome Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised

Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1 : Percentage of pregnant women receiving four prenatal care visits Value (quantitative or Qualitative)

61.40 91.22

Date achieved 12/31/2010 12/31/2014 Comments (incl. % achievement)

Source: RISKESDAS/MoH

Indicator 2 : Percentage of pregnant women receiving 90 iron tablets Value (quantitative or Qualitative)

18.00 32.43


Source: RISKESDAS/MoH; Percentage of women that report consuming 90 or more iron supplements during pregnancy.

Indicator 3 : Percentage of deliveries assisted by a trained professional Value (quantitative or Qualitative)

82.20 88.89



Indicator 4 : Percentage of women receiving two postnatal care visits Value (quantitative or Qualitative)

46.80 47.50



Indicator 5 : Percentage of children under one year receiving complete childhood immunizations

Value (quantitative or Qualitative)

75.00 83.82

Date achieved 12/31/2014 Comments (incl. % achievement)

Source: 2010 IE Endline Survey

Indicator 6 : Percentage of infants for which weight increases each month


N/A 75.36


Indicator 7 : Percentage of children attending monthly weight checks for under threes and biannually for under-fives


75.00 73.49


Source: SUSENAS

Indicator 8 : Percentage of children under-five receiving vitamin A twice a year Value (quantitative or Qualitative)

71.50 52.12


Below expectations. 2007 RISKESDAS data showed 71.5 of children under 5 received Vitamin A supplements twice annually.

Indicator 9 : Primary school enrollment of all children 6 to 12 years old Value (quantitative or Qualitative)

98.50 98.67


Source: PNPM Generasi Impact Evaluation endline survey.

Indicator 10 : Minimum attendance rate of 85% for all primary school aged children Value (quantitative or Qualitative)

96.00 89.64



Indicator 11 : Junior secondary school enrollment of all 13 to 15 years old Value (quantitative or Qualitative)

87.40 93.75



Indicator 12 : Minimum attendance rate of 85% for all junior secondary school-aged children Value (quantitative or Qualitative)

86.00 93.20



G. Ratings of Project Performance in ISRs

No. Date ISR Archived

DO IP Actual

Disbursements (USD millions)

2 05/30/2011 Moderately Satisfactory Moderately Satisfactory 10.20 3 05/02/2012 Moderately Satisfactory Moderately Satisfactory 32.84 4 12/25/2012 Satisfactory Moderately Satisfactory 42.34 5 04/15/2013 Satisfactory Moderately Satisfactory 42.34 6 10/22/2013 Satisfactory Moderately Satisfactory 61.73 7 05/07/2014 Satisfactory Satisfactory 73.32 8 12/03/2014 Moderately Satisfactory Satisfactory 73.32

H. Restructuring (if any)

Restructuring Date(s)

Board Approved

PDO Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in USD millions

Reason for Restructuring & Key Changes Made

DO IP

11/08/2013 S MS 61.73 Extend TF closing by 24 months.

09/16/2014 MS S 73.323 Reallocate Grant proceeds across expenditure categories

I. Disbursement Profile

1

1. Project Context, Development Objectives and Design

1.1 Context at Appraisal

Over the past several decades, Indonesia has made great strides in reducing poverty, with the official poverty rate declining from 23.4 percent in 1999 to 11.4 percent in 2013. Indonesia has also achieved significant successes in improving a number of key human development indicators. The under-five mortality rate in the country has declined from 164 per 1,000 live births in 1970 to 85 in 1990; and to 31 in 2012. Similarly, the percentage of children under five that were moderately or severely underweight decreased from 29.8 percent in 1992 to 17.9 percent in 2012. The proportion of birth deliveries attended by qualified attendants increased from 40 percent in 2002 to 63 percent in 2012.

However, Indonesia still lags behind other countries in the region in terms of maternal and infant mortality; child malnutrition; junior secondary school enrollment; and education learning quality. Indonesia’s 2013 maternal mortality rate (MMR) of 190 (per 100,000 live births) means it will miss its Millennium Development Goal (MDG) target of 102. Indonesia has the firth highest incidence of stunting in the world, with 8.4 million children under five suffering from chronic malnutrition. Across all of these measures, outcomes were worse in rural areas than in urban areas, with the quality and availability of basic services lagging in the former. A range of demand- and supply-side constraints continue to impact health and educational outcomes. Demand-side constraints included high out-of-pocket health and education expenditures, particularly in the case of the latter; high transportation costs to facilities due to infrastructure gaps; low awareness of and motivation to utilize services; and concerns over poor quality of services that are not worth the cost. Supply-side constraints included shortages of basic medical inputs in primary health care facilities; poor coverage of doctors and midwives in rural areas; and teacher absenteeism in rural areas.

In 2007, the Government of Indonesia (GoI) launched two large-scale pilot projects to address demand-side constraints in the area of health and education, as follows: (i) the Hopeful Family Project (Program Keluarga Harapan), which involved the provision of conditional cash transfers (CCTs) to individual households; and (ii) PNPM Generasi,1an incentivized community block grant program implemented through the National Program for Community Empowerment (Program Nasional Pemberdayaan Masyarakat, or PNPM-Rural). PNPM-Rural was the GoI’s flagship community empowerment program, underpinning Pillar Two of the GoI's three-pronged poverty reduction strategy, as defined in Presidential Decree 15/2010 on accelerating poverty reduction. At the same time, the GoI established the PNPM Support Facility (PSF) to harmonize and coordinate the efforts of development partners engaged in community empowerment-focused poverty reduction efforts. The PSF was managed by a Joint Management Committee (JMC) with representatives from GoI agencies and development partners, with the Bank serving as the PSF trustee.

The pilot phase of Generasi, which ran from 2007 to 2009, was accompanied by a rigorous impact evaluation2 (IE), which randomly assigned Project treatment and control areas. This IE found that Generasi had a statistically significant positive impact in terms of its 12 defined health and education indicators. In particular, the IE showed that the Project facilitated an average reduction in the incidence of childhood malnutrition of 2.2 percentage points. In addition, it facilitated an increase in the frequency of weight checks by young children; an increase in the number of pregnant women receiving iron supplements; and an increase in the rate of school participation of 7 to 12-

1 In this document, PNPM Generasi will be referred to as either “Generasi” or “the Project.” 2 Olken, B.A., J. Onishi, and S. Wong - Indonesia’s PNPM Generasi Program: Final Impact Evaluation Report, World Bank, June 2011.

2

year-olds, with this increase being most significant in areas with the lowest baseline of participation. Based on this evidence, the GoI took the decision to scale up Generasi as part of its suite of demand-side health and education programs. With the scaling up of the Project, the GoI committed to expanding coverage from the original 164 pilot sub-districts to at least 369 poor rural sub-districts3 by 2014. PNPM Generasi was appraised as Additional Financing (AF) to PNPM-Rural III. This AF was approved on August 3, 2010, with funding of US$ 10.2 million.

Generasi directly supported the GoI’s efforts to accelerate the achievement of the Millennium Development Goal (MDG) targets. It achieved this by increasing the utilization of basic services that contribute to reductions in maternal and child mortality and to the achievement of universal primary education. Generasi was fully consistent with the Bank’s 2009-2012 Country Partnership Strategy (CPS) for Indonesia (Report No. 44845, dated July 22, 2008). In particular, Generasi aligned with the pro-poor pillar, which focused on promoting community development; protecting the vulnerable; and improving health outcomes. Generasi was fully aligned with the CPS’s goals of improving local-level governance; facilitating a greater level of participation by women; increasing access to health and education services; and facilitating a higher rate of participation in early childhood education and development services.

1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved)

The Project Development Objective (PDO) was for villagers in PNPM-Rural locations to benefit from improved socio-economic and local governance conditions. The PDO in the Grant Agreement dated August 16, 2010 remains the same.

The expected outcomes for Generasi were included in the Loan’s Results Framework (RF) (see PNPM-Rural III PAD, Annex 3: Results Framework and Monitoring). The PNPM-Rural III PAD defines key performance indicators (KPIs) at the PDO level through Generasi, as improved health indicators in a minimum of 200 sub-districts in six provinces (East Java, West Java, North Sulawesi, East Nusa Tenggara, Gorontalo, West Nusa Tenggara). Specifically, these KPI are as follows:

To increase the rate of immunization coverage for infants aged 12-23 month by 10 percent, from 38 percent in 2005 to 48 percent in 2010;

To increase the proportion of women participating in prenatal care visits by 10 percent, from 56 percent in 2005 to 66 percent in 2010; and

To increase the proportion of births assisted by trained professionals by 10 percent, from 40 percent in 2005 to 50 percent in 2010.

The Project Paper for the AF to PNPM-Rural III revised the RF to add a total of 12 indicators, of which eight related to health and four related to education. These indicators were included as Intermediate Outcome Indicators (IOIs) under Component One.

Health Targets:

Pregnant women participate in at least four prenatal care visits;

Pregnant women take iron tablets during their pregnancy;

All birth deliveries are assisted by a trained professional;

Women participate in at least two postnatal care visits following deliveries;

3Plans for Generasi expansion are outlined in the original PNPM Generasi Project Paper (Report No. 56112-ID), and are reflected in the PAD for PNPM-Rural III, approved March 31, 2010, and in the PAD for PNPM-Rural IV, approved July 14, 2011.

3

All children complete the full childhood immunization schedule;

Infants achieve monthly weight increases;

Children under three are weighed monthly and children aged 3 to 5 are weighed biannually;

Children under the age of five take vitamin A supplementation biannually.

Education Targets:

All children 6 to 12 years old are enrolled in primary school;

The minimum attendance rate for all primary school-aged children is 85 percent;

All children aged 13 to 15 are enrolled in junior secondary school;

The minimum attendance for all junior secondary school-aged children is 85 percent.

1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification

The PDO and key indicators remained unchanged during project implementation.

1.4 Main Beneficiaries

The AF provided additional financing to PNPM-Rural III to support the expansion of Generasi from 164 sub-districts in six provinces to 369 sub-districts across 42 districts in eight provinces (East Java, West Java, North Sulawesi, East Nusa Tenggara, Gorontalo, West Nusa Tenggara, Maluku and West Sulawesi) through 2014, and to strengthen the capacity of the implementing agency to manage the scale-up of the program. Generasi's target was to achieve an estimated coverage of 5.4 million women, infants, and school-aged children in approximately 3,600 villages. The Project was expanded on an annual basis, as described in Table 1.

Table 1: Expanding Coverage through Additional Financing (AF)

Coverage Approval Date Amount (US$ ) AF 212 sub-districts in six provinces 3 August 2010 10,200,000 AF 1 290 sub-districts in eight provinces 2 June 2011 28,123,000 AF 2 369 sub-districts in eight provinces 17 April 2012 35,000,000 Total RETF 73,323,000

Secondary beneficiaries included: (i) the Directorate General for Community and Village Empowerment in the Ministry of Home Affairs (MoHA, the executing agency), which benefited from increased project implementation capacity; (ii) village facilitators and partner agencies providing implementation support; (iii) district governments in which the targeted sub-districts were located, which benefited from training, field studies, and institutional strengthening; and (iv) local service providers and community members active in facilitating the delivery of basic services (including midwives, community health volunteers, contract teachers, and tutors), who benefited from the operational support provided through Generasi.

1.5 Original Components

Component 1: Sub-District Grants (US$ 9.81 million): This component contributed to “improved socio-economic conditions” by providing resources for poor rural communities to finance an open menu of local health- and education-related development activities that addressed small-scale supply or demand-side constraints to service utilization, depending on where the

4

community perceived the need to be greatest. An incentivized portion4 of the grant rewarded villages that performed well relative to other villages, with these villages receiving a larger block grant allocation in the following year of implementation. The budget for this component eventually constituted approximately 87 percent of the total Project budget.

To support the achievement of “improved governance conditions,” this component provided operational funds (calculated as 4 percent of the Sub-District Grant allocation) to support local-level development planning, training and capacity-building, and mapping of early childhood education services. Activities funded through operational funds included: (i) village participatory needs assessments through which villagers worked with local health and education service providers to identify lags in access to specific services; to identify the neediest beneficiaries; and to determine which investments would be most effective to facilitate the achievement of specific targets; (ii) information and planning meetings at the hamlet, village, and sub-district levels, during which local service providers gained an understanding of community health and education needs; and (iii) annual village accountability meetings (MDST), through which the previous year’s funds were accounted for and performance bonuses were allocated.

Component 2: Technical Assistance (TA) (US$ 79,000): This component provided TA to strengthen the capacity of GoI entities responsible for implementation at the national, provincial, and district levels. Specifically, funds provided under this component covered the salaries and operating costs of national and provincial level specialists; the salaries and operating costs of district and sub-district facilitators; and MIS database related costs, including server costs and the salaries of computer operators.

Component 3: Training and Capacity Building (US$ 233,000): This component provided pre-service and refresher training for Generasi facilitators at the district and sub-district levels. This component also financed evaluation workshops; training for MIS staff; and the development and dissemination of communication and socialization materials to ensure Generasi and its approach was well understood at different levels. It also financed National Specialists (NMC) responsible for implementing the complaints handling system.

Component 4: Incremental Costs and Goods (US$ 78,000): This component provided support to strengthen and expand the operations of the Secretariat under the Ministry of Home Affairs (MoHA) including through workshops; through measures to improve coordination with other ministries; and through supervision. The Secretariat was responsible for the implementation of due diligence practices and for the provision of support to the implementing agency government staff in day-to-day management tasks.

1.6 Revised Components

The components remained unchanged during implementation. However, with two more AFs, the total budget allocation and the allocations for each component changed as follows:

4Within a sub-district, in the Project’s first year funds are divided among villages in proportion to the number of target beneficiaries in each village; i.e., the number of children of varying ages and the expected number of pregnant women. In the Project’s second and subsequent years, 80 percent of the sub-district’s funds are divided among villages in proportion to the number of target beneficiaries. The remaining 20 percent forms a performance bonus pool to be divided among villages, structured as relative competition between villages within the same sub-district, based upon their previous year’s performance on the 12 Generasi indicators.

5

Table 2: Generasi total budget and reallocation following second Restructuring

Components AF AF1 AF2 Original Amount

(US$ )

Reallocation (US$ )

%

1 Sub-District Grants

9,810,000 25,283,000 25,640,000 60,733,000 64,121,448 87.4

2 TA 79,000 2,243,000 7,640,000 9,962,000 6,904,831 9.4 3 Training and

Capacity Building

233,000 517,000 1,640,000 2,390,000 1,994,015 2.8

4 Incremental Costs and Goods

78,000 80,000 80,000 238,000 302,706 0.4

Total 10,200,000 28,123,000 35,000,000 73,323,000 73,323,000

1.7 Other significant changes

1.7.1 Institutional Arrangements

Generasi was originally implemented through the same Secretariat and TA structure as PNPM-Rural. Recognizing that the rapid expansion of PNPM-Rural to national scale had strained Project management capacity, the GoI decided that, from 2012 onwards, Generasi would be managed by the Directorate of Traditional and Community Social and Cultural Empowerment (Pemberdayaan Adat dan Sosial Budaya Masyarakat, or Sosbud), while PNPM-Rural would be managed by the Directorate of Institutional and Community Training (Kelembagaan dan Pelatihan Masyarakat, or KPM) under the Directorate General for Village Community Empowerment (Pemberdayaan Masyarakat Desa, or PMD). PMD established a stand-alone Generasi Secretariat in the Sosbud directorate responsible for implementing Generasi.

1.7.2 Restructurings and Delinking from PNPM-Rural III

Restructurings: The Project was restructured twice during implementation. The first restructuring was carried out to extend the closing date of the TF to December 31, 2014. Generasi was originally intended to operate at least until the end of 2014, in line with GoI’s implementation timeframe for PNPM-Rural. However, at the time of appraisal, the PSF Multi Donor Trust Fund (MDTF, TF070967), the TF from which Generasi was financed, had a closing date of June 30, 2013. As such, Generasi had to be implemented with the original closing date of December 31, 2012, thereby necessitating a later restructuring. The second restructuring was carried out to reallocate Grant proceeds across expenditure categories. As part of both restructuring processes, the Bank recommended to the GoI that the RF be revised to better reflect the outputs and outcomes specific to Generasi. The GoI did not act upon this recommendation.

Table 3: Project Restructurings

Project changes Approval Authority Approval Date First Restructuring

Extend TF closing by 24 months. RVP November 28, 2013

Second Restructuring

Reallocate Grant proceeds across expenditure categories

CD September 16, 2014

Delinking PNPM-Rural and Generasi: As described above, the GoI separated Generasi from PNPM-Rural III due to the lack of capacity of the implementing agency to manage both programs following the nation-wide scale-up in 2010. In December 2012, the Bank delinked Generasi from

6

PNPM-Rural III to reflect the GoI implementing arrangements for the two projects.5 Furthermore, it was decided that Generasi would not be linked to PNPM-Rural IV, a decision formalized during the appraisal process for PNPM-Rural IV. From March 2013 onwards, stand-alone Implementation Status Reports (ISRs) were produced for Generasi.

2. Key Factors Affecting Implementation and Outcomes

2.1 Project Preparation, Design and Quality at Entry

Soundness of background analysis and lessons learned from previous projects.

Generasi was designed to function through the robust KDP/PNPM-Rural implementation platform developed over the previous ten years and to build on the findings of a robust randomized CT impact evaluation (IE)6 which provided strong evidence regarding the Project’s strengths and weaknesses and the type of locations in which the Project was most effective. With evidence of the Project’s success, the GoI’s PNPM Steering Committee took the decision to expand Generasi to additional sub-districts with poor health and education outcomes and to mainstream the incentivized block grant system in all Generasi locations from 2010 onwards.

Assessment of the Project design

Generasi benefited from being able to draw on robust IE findings. Major findings and related design implications included: (i) the incentivized community grant approach was more effective than the non-incentivized approach, and therefore was mainstreamed in all Generasi locations; (ii) Generasi was most effective in areas with low health and education performance, which enabled the GoI to prioritize sub-districts with poor health and education performance when expanding the Project; (iii) the Project had greater impacts in health than in education ; and (iv) the GoI’s existing CDD architecture was an effective implementation platform to enable communities to overcome constraints to accessing health and education services, but required specialized facilitation in order to be effective. As such, while all Generasi locations participated in PNPM-Rural from 2010 onwards, the Generasi TA and facilitation structure remained in place, rather than relying on existing PNPM-Rural facilitators.

The design enabled Generasi: (i) to build upon the achievements of the PNPM-Rural system to achieve significant economies of scale; (ii) to take advantage of the social capital built through the implementation of PNPM-Rural for communities to plan and implement local development sub-projects; (iii) to implement a community-based poverty targeting mechanism that enabled flexible adjustments to beneficiary lists as required to mitigate the impact of economic shocks; (iv) to create opportunities for communities to take action on the basis of a collective action model, in which villagers meet with local service providers to achieve improvements in terms of the specified health and education indicators by addressing small-scale supply-side constraints; and (v) to maintain the necessary flexibility to add financing every year to support the phased expansion.

However, the Project design had several shortcomings, including: (i) PDO-level KPIs had an end-of-project target date of December 2010, which did not correspond with the December 2012 closing date of the Grant; (ii) KPIs and IOIs specific to Generasi did not capture aspects of improved local governance in health and education; (iii) AF cycles necessitated midstream revisions to Generasi’s budget, which caused implementation delays and distracted the GoI and Bank teams from focusing on implementation quality; (iv) the capacity of a single Project Implementation Unit (PIU) and TA

5Implementation Completion and Results Report (ICR No. 3281 of January 27, 2015) for the Fourth National Program for Community Empowerment in Rural Areas (Loan. 8079). 6Olken, B.A., J. Onishi, and S. Wong - Indonesia’s PNPM Generasi Program: Final Impact Evaluation Report, World Bank, June 2011.

7

structure to manage Generasi and PNPM-Rural during the national scale-up in 2010 was overestimated; and (v) the implementing agency had limited technical capacity in health and education.

GoI commitment, stakeholder involvement, and participatory processes: The National Planning Board (Badan Perencanaan Pembangunan Nasional, or BAPPENAS) led the preparation of the Project by proactively convening meetings between MoHA, Australian Aid (AusAID), and the Bank to reach agreement on a multi-year financing plan for the use of PSF MDTF funds to co-finance the Generasi scale-up. The design was formalized through the signing of a Memorandum of Understanding (MoU) dated March 17, 2010, and was the basis for the 2010 Project Paper.

Risks and risk mitigation measures: At appraisal, after mitigation, the overall risk level was rated as Moderate. This rating was deemed appropriate given the implementing agency's capacity limitations and the overall high risk fiduciary environment that exists in Indonesia. Many of the critical risks identified in the PNPM-Rural III PAD also applied to Generasi. Three risks were rated as Substantial, as follows: (i) the capacity risk resulting from the implementation of the scale-up of Generasi concurrently with the scale up of PNPM-Rural; (ii) governance risk; and (iii) financial management risk, including in the area of procurement.

The risk analysis did not identify the need for a legally-binding mechanism to ensure that the GoI abided by its co-financing commitments. This risk factor had an impact on the quality of planning and budgeting, as the amount of GoI co-financing varied from year to year, and necessitated time-intensive budget revisions. In addition, the Bank underestimated risks associated with the poor quality and availability of basic health and education services, such as the availability of vaccines, which affected the ability of communities to achieve Project targets.

Given the above, Quality at Entry is rated as Moderately Unsatisfactory. The technical design was sound and built on a robust IE of the pilot phase. The primary drawback to Quality at Entry was the RF, which did not define PDO-level indicators beyond 2010. Most critical risks were identified upstream, but two were missed. Weaknesses in the upstream risk assessment did not negatively impact implementation, as the GoI and the Bank were able to tap PSF resources to mitigate risks underestimated at appraisal during implementation.

2.2 Implementation

Overall implementation progressed well, with geographic expansion occurring on an annual basis and according to schedule (see Table 1 above). Operational research and frequent implementation support missions (ISMs) provided a basis for the GoI to continuously adapt and fine-tune Generasi during implementation to respond to varied and evolving community needs and priorities.

Factors Subject to the Control of the Government or the Implementing Agency

Supporting factors

Strong political commitment: The Project secured support at the highest levels of government, as demonstrated by the GoI's significant financial contributions and its commitment to the establishment of the stand-alone Generasi Secretariat to facilitate the achievement of better programming, monitoring and oversight.

Willingness to expand and strengthen Generasi’s TA and facilitation structures: The mobilization of resources including dedicated national and provincial specialists; district consultants, including dedicated FM consultants; and sub-district facilitators proved to be an effective measure to improve Project management and oversight and to mitigate governance risks.

Leveraging external resources and partnerships to strengthen Project systems and impacts: The GoI tapped PSF resources to strengthen the Project’s M&E systems, including through regular ISMs, performance monitoring surveys, and qualitative operational research. M&E was further

8

supported by the development of a web-based MIS sufficiently adaptable to meet emerging needs. By 2013 the implementing agency began reaching out to local civil society organizations with expertise in local basic service delivery and existing relationships with Project communities and local governments as a means to maximize impacts.

Challenging factors encountered during implementation

GoI budgeting and multiple AF processes: The need to appraise AFs annually, despite the existence of a multi-year Project work plan, created implementation delays. The need to do so was related to the restrictions of PSF donors to allocate funds to the MDTF on an annual basis only, which in turn required annual amendments to the Project Grant Agreement. Furthermore, the GoI required signed legal agreements to be in hand before it finalized and executed its annual budget. By 2014, the Bank had adjusted its appraisal requirements to allow for a multi-year Project with estimated funding requirements to be appraised up front and Grant amendments to be handled through an exchange of letters with the GoI rather than through annual appraisals.

Cross institutional coordination impeded linking demand-and supply-side interventions: The Project’s institutional set-up involved a partnership between MoHA and line ministries, with MoHA responsible for community empowerment tasks and the respective line ministries responsible for technical health and education issues. Despite the efforts of the Coordinating Ministry for People's Welfare (Menkokesra) through the PNPM Steering Committee, coordination with the Ministry of Health (MoH) and the Ministry of Education and Culture (MoEC) remained a challenge throughout implementation. Thus, complementary supply-side interventions were not designed specifically for Generasi target communities until 2013 when Millennium Challenge Account-Indonesia (MCA-I) support was made available to the MoH to prepare a package of interventions. These included the provision of training for health workers; the provision of micronutrients; and the implementation of a communications campaign to improve the quality of maternal and child nutrition services in Generasi target communities.

Role of local governments: Under Indonesia’s decentralized system of service delivery, district governments are responsible for the delivery of basic health and education services. As a national project, Generasi did not include mechanisms for district governments to influence sub-district selection or to determine the specific health and education targets. As a result, district governments had limited incentives to adapt local development plans and budgets to complement Generasi’s investments.

Communities expressed a high level of demand for ECED services (See Section 6.b): It was difficult to accommodate this demand, given that there was no reference to ECED services in the 12 defined health and education targets.

Gender: A qualitative assessment7 noted that Generasi was successful in involving women in local health and education decision-making and in targeting sub-project investments to women. It also noted that, in order to achieve behavioral change related to health and nutrition practices, greater efforts were needed to provide men with information on maternal and child health, nutrition, and education issues. (See Section 3.5.a and Section 6.d)

Quality and retention of facilitators: Another challenge related to the quality of facilitators and the Project’s ability to retain facilitators with a background in basic health and/or education. In general, this challenge was less significant in Java than on the outer islands, where competition with the private and other sectors and/or a general lack of human resources were more significant factors. The Project had to adapt to the lack of human resources outside of Java by pursuing a strategy of

7Shatifan, N. (2011) “Gender equality as a key dimension for improved maternal and child nutrition to reduce stunting,” Unpublished report prepared for the Millennium Challenge Corporation.

9

mobilizing generalists with knowledge of health and education issues, rather than sector specialists. (See Section 6.c). However, facilitators' heavy administrative workload limited the time they could spend with community members in the field.

Effectiveness of risk mitigation

Executing Agency Implementation Capacity Risk: This risk was successfully mitigated by providing technical support under Components Three and Four to improve project management; to strengthen the Project Secretariat; to strengthen the web-based MIS and complaints handling system; and to monitor implementation progress. Technical risks were mitigated in part through the use of Bank-executed resources provided through the PSF. The use of these resources enabled Bank teams to provide technical health and education inputs and to facilitate linkages with MoH and MoEC.

Governance risks: Measures to mitigate governance risks were generally successful. However, ensuring implementation of an effective anti-corruption strategy8 to identify and resolve cases required ongoing attention.

Financial Management (FM) risks: These risks were successfully9 mitigated by: (i) expanding the external audit coverage reach to a minimum level of 20 percent of Project sub-districts; (ii) mobilizing full-time district FM consultants; (iii) mobilizing dedicated FM Specialists and Complaints Handling Specialists at the national level; (iv) strengthening the oversight of disbursement and complaints handling processes through the rolling out of a web-based system; and (v) revising the internal audit manual.

Risks related to planning and budgeting: These risks were exacerbated by requirements for a full appraisal for each annual tranche of AF. The Bank was proactive in mitigating these risks, which were addressed by streamlining the appraisal procedure for the follow-on Generasi project.

2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization

(a) M&E design

Generasi’s PDO was the same as that of its parent project, PNPM-Rural III. However, the three PDO-level KPIs specific to Generasi had 2010 end dates, despite the fact that Generasi’s implementation schedule ran to the end of 2012. Contrary to general practice for AF Project Papers, by August 2, 2010, the AF Project Paper lacked a detailed M&E section. However, Annex 1 to the AF paper did expand the PNPM-Rural III RF to include the 12 health and education related targets at the IOI level. The IOIs were appropriate for measure progress towards achieving the PDO, even if annualized values were not assigned in the RF. Additional M&E activities were implemented to measure IOI achievement beyond what was defined in the 2010 Project Paper.

(b) M&E implementation

The GoI and Bank dedicated a substantial effort to correcting the original weaknesses of the RF to ensure that the GoI had the data needed to improve Project performance. With Bank support, the

8Generasi used multiple sources to detect and report issues of leakage, fraud and corruption; including: (a) complaints by community members and other program stakeholders through available complaint channels (web, SMS, email, mail, telephone, and in-person). The web based Complaints Handling System (CHS), SMS and email gateway are used jointly by Generasi and PNPM-Rural; (b) reports from PNPM facilitators and consultants; (c) audits by the GoI audit agency (BPKP); (d) PSF and joint implementation support missions; (e) NGO reports; and (f) media reports. All complaints received are recorded in the CHS Application/Database. PSF regularly tracks cases based on information from media, field missions, the CHS, and other sources and shares it with GoI to ensure consistency in reporting and follow-up. 9Audit findings by BPKP for FY 2013 amounted to IDR 317 million or less than US$ 32,000 or 0.07 percent of IDR 446.9 billion total disbursements in 2013.

10

GoI developed an M&E strategy specific to Generasi during Project implementation. This strategy included: (i) the strengthening of the web-based MIS, with the Bank providing substantial support to strengthen and expand the system as the Project expanded geographically; (ii) the implementation of two qualitative studies; 10 and (iii) the implementation of a Performance Monitoring Survey in 2013 and 2014.

Given that the process of uploading MIS data to the web-based application experienced time lags, the Bank and the GoI established and monitored a target for the proportion of villages uploading data to the MIS (80% of Project villages by July of each calendar year). The level reached was 75 percent in 2010; 70 percent in 2011; 76 percent in 2012; 76 percent in 2013; and (at the time of the writing of this ICR) around 40 percent for 2014.11 As anticipated, MIS quality assurance practices continued to require improvement, even though efforts were undertaken to regularly assess the bottlenecks in the functioning of the MIS.

By 2013, the growth in the scale and profile of Generasi resulted in an increased demand for data on implementation progress from key stakeholders, which the MIS alone could not provide. To address this, PMD requested Bank support to implement the Performance Monitoring Survey (PMS).12 This survey used a small cluster sampling methodology that enabled an assessment of how the Project was performing on average across all participating villages. The PMS complemented the MIS, providing an additional means by which Generasi could support the GoI’s medium-term strategy of strengthening the collection and analysis of performance data.

(c) M&E utilization

Data collected from different sources, including the MIS, the PMS and ISMs, assisted GoI and the Bank to (i) identify and remedy quality issues and to incorporate better controls as Generasi progressed; (ii) improve the control of fund flows and the level of accountability for sub-grants; (iii) escalate issues to PMD management when appropriate; and (iv) provide information on project performance to stakeholders.

Generasi’s M&E system was not designed to substitute for health and education data collected by the respective line ministries and local government agencies. Instead, the system had an explicit community empowerment focus, with community members being trained to collect and analyze health and education data on a monthly basis. The use of this methodology proved to be an important tool to enable community members and facilitators to engage with local service providers to discuss issues related to service delivery gaps and constraints. Overall, M&E Design is rated as Modest.

2.4 Safeguard and Fiduciary Compliance

Safeguards.

Generasi was rightly classed as a Category B project. As was the case with PNPM-Rural III, four safeguard policies were triggered: (i) environmental assessment (OP/BP 4.01); (ii) natural habitats (OP/BP 4.04); (iii) indigenous peoples (OP/BP 4.10); and (iv) involuntary resettlement (OP/BP

10 (i) Mc Laughlin K., Indonesia Village Health Institutions – A Diagnostic - Unpublished manuscript. 2013; and (ii) Opportunities and Approaches for Better Nutrition Outcomes through PNPM Generasi - A Qualitative Study. PNPM Support Facility (PSF) and Poverty Reduction Support Facility (PRSF). February 2014. 11 At the time of writing 2014 MIS data is in the process of being uploaded. 12 Objectives of the PMS include: (i) To obtain reliable estimates of community performance indicators; (ii) To gather information about beneficiaries’ utilization of health and education services; (iii)To gather information about mothers’ health and nutrition knowledge and practices; (iv) To assess the community empowerment aspect of Generasi e.g. beneficiaries’ participation in planning and implementation; beneficiaries’ knowledge and perception and (v) To obtain qualitative assessment on Generasi implementation focusing on aspects such as service providers’ capacity, facilitation process, local level coordination, supervision and monitoring and accountability mechanism.

11

4.12). Generasi AFs continued to apply the PNPM-Rural Environmental and Social Safeguards, Indigenous and Vulnerable Peoples, and Land Acquisition and Resettlement Policy frameworks. These instruments were embedded in the PNPM-Rural and Generasi Project Guidelines and Operational Manuals and were used to trigger safeguards policies.

Environmental Assessment (EA) and Natural Habitats: No stand-alone EA (including an Environmental Management Plan) was necessary. An Environmental Management Framework, incorporating Natural Habitats screening and mitigation requirements was developed and was incorporated into the PNPM-Rural operational process. In country disclosure took place on July 21, 2010, and disclosure through Infoshop took place on September 12, 2011. Generasi sub-projects were small in scale, with Project funds of a relatively low value invested in infrastructure sub-projects. ISMs conducted throughout Project implementation found no evidence of adverse environmental impacts as a result of the implementation of Generasi sub-projects, either in general or on natural habitats.

Indigenous Peoples (IP): An Isolated and Vulnerable Peoples Planning Framework was developed under PNPM-Rural III, with the principles of participation and inclusion being embedded in Generasi’s design. In country disclosure took place on July 21, 2010 and disclosure through Infoshop took place on September 12, 2011. Generasi provided incentives for communities and facilitators to include all community members in Generasi activities, including IP. Implementation support missions carried out over the period of the Project included consultations with IP communities in various provinces. These missions found no evidence of adverse social impacts.

Involuntary Resettlement: The Land Acquisition and Resettlement Policy Framework developed under PNPM-Rural III was applied to Generasi. In country disclosure took place on July 21, 2010 and disclosure through Infoshop took place on September 12, 2011. The Project financed the rehabilitation and expansion of local health facilities and schools on land already owned. In the event that additional land was needed in order to facilitate the expansion of the facility, such land would be donated and the donation process documented. Thus, no land acquisition was financed under the Project. Project consultants and facilitators at different levels ensured implementation in accordance with this framework according to procedures specified in the Operations Manual.

The Project’s Safeguards instruments were revised in 2013 to maintain consistency with the simplified instruments applied under PNPM-Rural IV. Safeguard guidelines for PNPM-Generasi were defined in the new Implementation Guidelines for Social and Environmental Safeguards (IGSES), which was satisfactorily prepared and disclosed on November 20, 2013, and was re-disclosed on December 16, 2013. While the IGSES has been distributed to facilitators, some additional efforts were necessary to ensure that these guidelines are well understood and applied. A safeguards thematic review was conducted by the Bank in mid-2014, with this review finding no serious adverse environmental or social impacts in Project locations.

Fiduciary

Better Governance Action Plan: Generasi utilized the PNPM-Rural III Better Governance Action Plan (BGAP). These measures built on more than a decade of field experience and specific research trials. The key features of the framework13 were solid, with the guiding principle underlying the anti-corruption program encouraging multiple stakeholder involvement, not just the GoI or the Bank, in oversight and action. The BGAP included a system to identify and resolve cases of suspec-ted mismanagement, fraud or corruption, including mainly embezzlement, budget mark-ups and

13Local control of funds, beneficiaries targeting by village cadres, Generasi internal control, internal and external audits, village level social control, code of ethics, complaints handling and sanctions

12

price-fixing by Financial Management Unit (Unit Pengelola Keuangan, or UPK) personnel and Village Implementation Team (Tim Pengelola Kegiatan, or TPK) members.

In the 2013-201414 period, there were 59 recorded cases of suspected mismanagement,15 of which 54 involved cases of fraud and/or corruption, four cases related to non-compliance and one case related to force majeure. The fraud and/or corruption cases involved a total value of misused funds of IDR 3.09 billion, of which IDR 1.86 billion (60%) has been recovered. Twenty-five cases (46%) were resolved,16 while the others are currently being dealt with by Project consultants, facilitators, and communities.

An effective sanction was the suspension of Project disbursements to sub-districts with unresolved cases of fraud and corruption. Finally, it should be noted that Generasi did not include a revolving loan fund component, the highest risk component in PNPM-Rural.

Financial Management (FM): The Project complied with FM policies. It developed an online disbursement application database system at the district level, with this system enabling users to submit the disbursement order (SP2D) to the Project Secretariat. This resulted in a better managed IFR process, with the Project producing complete and on-time IFRs without backlogs (outstanding claims). The Project received unqualified opinions from the external auditor in 2011, 2012 and 2013.17 Audit coverage reached 29 percent in 2011; 54 percent in 2012; and 32 percent in 2013. This coverage is considerably above the target of a minimum of 20 percent of Generasi locations.

The Project’s FM performance improved over time, due in part to the mobilization of additional FM consultants at the national and district levels. Quarterly internal audit reports were generally submitted on time, but the implementation of internal audits remained a challenge. This was largely due to the fact that FM facilitators had difficulties in achieving the target sample size due to sub-optimal time management and the difficulties faced by PMD in allocating sufficient time for internal audit activities at the sub-district and village level.

Procurement

The procurement of the first Administrative Services Firm (ASF) was conducted in 2012, at the point when the responsibility for the implementation of Generasi was shifted between Directorates within PMD. However, the process was delayed as a result of unsatisfactory Bid Evaluation Reports. The delay was exacerbated by processing and response times within the Bank. Following the resolution of this issue, procurement at the national level has been generally carried out in compliance with the Procurement Plan and was rated as Moderately Satisfactory in the latest ISR (November 2014). Improvements were recorded, with the transition to applying the improved procurement procedures under the revised Operational Technical Guideline (PTO)18 underway, and community procurement carried out in compliance with the existing PTO. Furthermore, no significant issues were noted during annual ex-post reviews conducted by the Bank.

14 Reporting on cases of fraud and corruption directly related to Generasi began being reported separately from PNPM-Rural after the Project was formally delinked in late 2012. 15The National Management Consultant’s October 2014 Report 16 There were six cases resolved through litigation with average days to resolve 476 days, and 19 cases resolved through mediation / non litigation with average days to resolve 288 days. 172014 Independent audit result from BPKP (Indonesia’s central auditing agency) will be available by June 30, 2015. 18 The revised PTO of the PNPM-Rural 2012-2015 project became effective in April 2014 and was applicable to Generasi. The revised version built on lessons learned related to non-compliance against procedures in community procurements.

13

Generasi initiated improvements19 in the bidding process were used by the implementing agency, with an increased number of items bid competitively. This resulted in improvements in economy of contract and improved transparency in the bidding process, as it enabled quicker processing and a simplification of the tasks of both the bidder and the bid assessors. The revised bidding documents are now also used by PNPM-Rural 2012-2015 (P128832, IBRD Loan No-8217).

2.5 Post-completion Operation/Next Phase

Operation and maintenance of health and education infrastructure: In education, Generasi mainly financed the rehabilitation of existing classrooms; the construction of additional classrooms and improved sanitation facilities in existing schools; and the establishment of satellite classrooms to enable existing schools to serve remote hamlets. Operational costs and maintenance were the responsibility of the district education departments. In health, the Project financed the construction of new village-level health facilities; the rehabilitation of existing village-level health facilities; and the construction of improved sanitation facilities at health facilities. The maintenance of integrated community health post (Posyandu) activities was the responsibility of communities, whether through village government or other funding sources, while the operation and maintenance of other village-level health facilities were the responsibility of district health departments.

The PNPM Generasi Project (TF014769)20 is the follow-on to this Project. It was appraised at a value greater than US$ 300 million, including up to US$ 151.7 million in Bank co-financing. The Project will continue to utilize the incentivized sub-district grant model, operating in the eight existing Project provinces and in three new provinces through the end of 2017. The Project attracted more than US$ 81 million in grant financing from the MCA-I. As a stand-alone Project, the Project’s PDO is “to empower local communities in poor, rural sub-districts in project provinces to increase the utilization of health and education services.” This PDO has been tailored specifically to PNPM Generasi outputs and outcomes. The above-mentioned long-term IE will identify the extent to which community-driven approaches can address shortcomings in service delivery outcomes, independent of investments in service delivery systems. It will measure Generasi’s impacts on priority outcomes over a seven or eight-year period, and inform Ministry of Villages, Disadvantaged Areas, and Transmigration (MoV), MoHA, and MoF on CDD mechanisms that can be implemented systematically and in a cost-effective manner through the Village Law.

3. Assessment of Outcomes

3.1 Relevance of Objectives, Design and Implementation

Rating for Relevance of Objective: Substantial

The Project’s development objectives are relevant both to current GoI poverty reduction objectives and to the Bank’s 2013-2015 Country Partnership Strategy (CPS) for Indonesia. The 2013-2015 CPS identifies six thematic areas that form the core of the Bank’s engagement in Indonesia. Of these six thematic areas, Generasi directly contributes to two: (i) promoting communities, by protecting the vulnerable and improving health outcomes, and (ii) gender and governance. Generasi is the Bank’s only operation that directly addresses the causes of maternal mortality and childhood stunting in Indonesia. Similarly, Generasi directly contributes to GoI objectives of reducing poverty and inequality by improving access to quality basic health and education services for the poor and

19Previous to this, only the management fee of the contractors was competitive whereas other items were fixed in the RfT. Improvements were also made to criteria as well as scope of service requirements. A number of items are still defined in the RfT to ensure quality implementation including salaries for consultants as well as travel expenses. 20Refer to Report No.: 829, dated June 23, 2014 for details.

14

vulnerable. The GoI’s strategy emphasizes a multi-sectoral approach to addressing the multiple factors contributing to stubbornly high levels of maternal mortality and childhood malnutrition. The strategy recognizes the important role that community empowerment has in overcoming barriers to service utilization.

Rating for Relevance of Design and Implementation: Modest

As described in detail in section 3.2, Project components financed inputs that produced outputs and intermediary outcomes that ultimately led to improvements in the two-part development objective in what constitutes a robust causal chain. However, as described in Section 2.3, the original design of the RF had several shortcomings, and the relevance of design is therefore rated Modest.

The Project’s multi-faceted M&E system enabled the GoI to make regular improvements to Project design during implementation, thus ensuring that the Project remained highly relevant to GoI priorities. Over the lifetime of the Project, improvements included: (i) the addition of two community-level target indicators that incentivize pregnant women, mothers, fathers and caregivers to participate in enhanced nutrition counseling sessions delivered through integrated community health posts; (ii) changes to community-level education target indicators to incentivize communities to focus Project resources on hard-to-reach school-aged children, including drop outs and children with disabilities, and on ensuring that students make the transition from primary school to junior secondary school; and (iii) a refined gender strategy that focuses on engaging men in discussions and decision making processes related to investments and behaviors in health, nutrition, and education. These design changes came into effect in early 2014 through Bank no objection to the revised Generasi Operations Manual (PTO).

3.2 Achievement of Project Development Objectives

Rating: Substantial

Generasi had a two-part PDO: Part (a) was to improve socio-economic conditions and Part (b) was to improve local governance conditions. This assessment gives the two parts equal weight.

Generasi’s contribution to “Improved Socio-economic Conditions”: Rating: Substantial

There are two mechanisms through which Generasi improved socio-economic conditions. First, Generasi inputs (including the provision of scholarships, uniforms and books to poor students; the provision of antenatal care and care for children; and subsidies for transportation costs for health visits and schools) reduced beneficiaries’ out of pocket expenses, leaving them with additional funds for other needs, thus improving their socio-economic conditions. Second, Generasi improved the rate of utilization of health and education service among beneficiaries (attendance at school; antenatal care; immunization; growth monitoring; etc.), which is likely to improve beneficiaries’ socio-economic outcomes in the long run.

Communities allocated Generasi block grants for a wide variety of purposes that directly reduce communities’ out of pocket health and education expenditures. As an illustration, school fees of different types account for at least 20 percent of total household education expenditure; the cost of uniforms for primary school account for up to a third of households’ education expenditures in rural areas; while the share of transport costs expenditure almost triples in rural areas between primary and junior secondary education (from 6% to 16%) (UNICEF 2011). In 2011, communities allocated on average 20 percent of the total value of Generasi block grants towards the provision of school uniforms and books and to scholarships and transportation to school for poor students.

Second, Generasi was oriented towards improving the utilization rate of 12 priority health and education services. These were selected to represent health and educational behaviors that are under the direct control of villagers. By improving the rate of utilization of basic services among communities, Generasi was likely to improve beneficiaries’ socio-economic conditions both in the

15

short term and the long term. In the short term, improvements in the rate of utilization of health services such as immunization were likely to reduce the spread of communicable diseases, which would reduce community members’ out of pocket health expenditures. Deliveries assisted by skilled health personnel (a Generasi sub-project investment) are less likely to result in the death of the mother or child during or immediately following childbirth. In the long run, the increased rate of utilization of services such as weight checks, Vitamin A supplements and immunizations should translate into a reduction of acute and chronic malnutrition (stunting and wasting), which would directly improve individuals’ socio-economic circumstances in the long term. Further, the rehabilitation of health infrastructure; funding for transportation; the dissemination of information of the benefits of health and education services, particularly related to maternal and child health; and the provision of supplemental nutrition were likely to reduce malnutrition.

The likelihood that, over time, Generasi will reduce acute malnutrition and chronic malnutrition, especially in the poorest provinces, was also high. According to the results of the 2007-2009 IE,21 after 2.5 years of program implementation, the rate of occurrence of childhood malnutrition declined by 2.2 percent, about a 10 percent reduction from the control level. This reduction in malnutrition was strongest in areas with a higher rate of malnutrition prior to project implementation, most notably in Nusa Tenggara Timur (NTT), where underweight rates were reduced by 8.8 percentage points, a 20 percent decline compared to control areas; severe underweight rates were reduced by 5.5 percentage points, a 33 percent decline; and severe stunting was reduced by 6.6 percentage points, a 21 percent decline compared to control areas.

By improving the overall health of pregnant women and by reducing acute malnutrition among children in their first 1000 days of life, Generasi was likely to contribute to children’s ability to participate in education. There is an abundance of empirical evidence linking maternal and early child health and nutrition interventions to educational outcomes in primary school (for example, Grantham-McGregor et al. 2013). Severe malnutrition in early childhood can result in impaired cognitive development (Wai-Poi; Pollitt; Granthan-McGregor et al. 1999), with poorer cognitive function, poorer motor skills, lower activity levels, less interaction with their environment and a lower rate of skill acquisition (Lasky et al. 1981; Johnston et al. 1987; Granthan-McGregor et al. 1997, 1999). Glewwe and King (2001) argue that malnutrition that persists into the second year of life is most critical for cognitive development. Analysis based on longitudinal data provides evidence of the direct effect of child malnutrition on education, wages and income (Hoddinott et al. 2008; Wai Poi 2008).

By including school enrolment as a target, Generasi incentivized communities to identify and assist out-of-school children not benefiting from other forms of assistance. Communities allocated a portion of the Generasi block grant to help poor students with transportation to school, fees, and uniforms and books. In addition, communities allocated funds to help pay the salaries of contract teachers, which helps to ensure that schools are staffed appropriately. Through these sub-project investments, Generasi increased the likelihood that youth complete secondary education.

Generasi’s contribution to “Improved Local Governance Conditions”: Rating: Substantial

While not defined explicitly in the PNPM-Rural III RF, it was expected that Generasi would improve local governance conditions by: (i) increasing the level of participation of women and the poor in health and education-related decision making processes; (ii) equipping communities with the skills, resources, and knowledge necessary to engage local service providers collaboratively in the identification of solutions to barriers to accessing services; (iii) increasing awareness among local service providers (teachers, midwives, health workers, school principals) of the health and

21 See Olken, Onishi, and Wong 2011

16

education-related needs of poor community members and being more responsive to these needs; and (iv) improving the planning, utilization, oversight, and accountability for Project resources at all levels of implementation.

This section draws on the following data sources to demonstrate the Project’s contribution to these areas of improved local governance conditions: (i) Project MIS data on participation and direct beneficiaries; (ii) secondary quantitative data sets; and (iii) qualitative research.

Increased participation of women and poor in health and education decision-making processes: MIS data suggests that women played an important role in Project decision-making processes and benefited substantially from Project investments. Women comprised 63 percent of participants in Project socialization, planning, and monitoring meetings in 2010; 69 percent in 2011; 69 percent in 2012; and 71 percent in 2013. Over the period from 2010-2013, 57-58 percent of direct Project beneficiaries were women or girls, which suggests that relatively high levels of women’s participation in Project decision-making resulted in a relatively high proportion of Project investments being targeted directly to benefit women and girls.

Improved ability of communities to influence investment decisions by the GoI and village and district governments: The 2011 Potensi Desa (PODES) survey, a national census conducted once every three years, shows that in Generasi villages, on average, PNPM-Rural and other GoI sources of funding external to Generasi were used to make significantly higher levels of investment in health and education infrastructure than in control villages. For example, 68 percent of Generasi villages had benefited from investments in health infrastructure, compared to 58 percent of control villages. Seventy percent of Generasi villages had benefited from investments in education infrastructure, compared to 50 percent of control villages. This suggests that in Generasi villages, communities and frontline service providers were more effective at lobbying village and district governments to make investments in health and education.

Findings from two qualitative studies, (i) Indonesia Village Health Institutions – A Diagnostic; and (ii) Opportunities and Approaches for Better Nutrition Outcomes through PNPM Generasi - A Qualitative Study show that Generasi facilitators improved coordination with service providers and village governments, which resulted in these stakeholders having more information regarding communities’ health and education needs. The Local Level Institutions (LLI) III study22 found that the level of participation in PNPM-Rural remained strong, with high satisfaction rates being recorded, with a widely-held perception that the PNPM-Rural was more transparent than sectoral programs at the village level. These findings suggest that there were a number of mechanisms through which Generasi was able to strengthen the capacity of rural communities to influence the decisions and actions of local service providers, village, and district governments.

Improved planning, utilization, oversight, and accountability for Project resources at all levels of implementation: Community members were empowered to own and manage the monitoring of achievements in terms of the Project’s indicators, with reduced dependence on third parties. Through this process, communities developed the necessary governance skills and improved abilities to plan, implement and oversee frontline service delivery activities. As described in Section 2.4, the Project’s complaints handling mechanism enabled communities to identify cases of fraud and corruption and to implement protocols to resolve such cases with support from Project facilitators and the PNPM Oversight Team’s governance unit. Furthermore, the variation over time in the type of sub-project investments suggests that community members were able to adjust annual plans in line with evolving community needs, rather than simply replicating the previous year’s activities.

22The Local Level Institutions III: Overview Report. World Bank. 2013.

17

Conclusion: The analysis above shows that the Project, through the provision of incentivized block grants, was effective in increasing the rate of utilization of most targeted health and education services, thereby improving socio-economic conditions. The analysis also provides evidence of the Project’s substantial contribution to improved local governance conditions through the increased participation of women and the poor in local health and education decision-making processes; the strengthened capacity of rural communities to influence resource allocations controlled by local service providers, village, and district governments; and improved community capacity to diagnose and monitor local health and education conditions, and therefore to make better use of locally available resources to address these over time.

3.3 Efficiency

Rating: Substantial

The rating is based on the findings of four analyses of the level of efficiency of Generasi. These findings show that: (i) the cost of Generasi is comparable to a similar program in Indonesia (see below); (ii) Generasi creates employment opportunities for poor members of communities and others and has the potential to increase beneficiaries’ income in the long term; (iii) the Economic Rate of Return (ERR) of the expanded version of Generasi yields a cost benefit ratio of 1.90. In other words, for every dollar invested, the return for beneficiaries is US$ 1.90; and (iv) Generasi infrastructure sub-projects, as in PNPM-Rural, are less expensive and of the same or better quality than line agency funded projects.

Cost efficiency of Generasi compared to Hopeful Family Program (Program Keluarga Harapan, or PKH): A cost effectiveness analysis of Generasi was published in 2014.23 The analysis suggests that the incentivized version of Generasi had a real cost per point of about US$ 8, while the non-incentivized version had a real cost per point of about US$ 11.24 The cost estimates for the incentivized version of the health component was US$ 7 per point, while for the education component it was US$ 13 per point. The cost of additional child weight checks ranged from US$ 16-22; the cost of preventing one case of child malnutrition ranged from US$ 384-528; the cost of ensuring that a single child received the recommended supplementation regime of Vitamin A ranged from US$ 160-220; while the cost of enrolling an additional child in primary school ranged from US$ 200-275. In these terms, the cost of Generasi is roughly comparable to an alternative household-based conditional cash transfer program, PKH, which was piloted and evaluated at the same time as Generasi. While the data that was used to calculate the cost effectiveness is from 2007 to 2009, because of relatively similar unit costs and levels of economic growth, it can be confidently stated that these figures are still relevant.

The ERR of Generasi: Impact on Beneficiaries’ Income and Employment:25 Beginning in early 2013, the Bank, in collaboration with Monash University (Victoria, Australia), undertook an analysis of the ERR of Generasi and additional supply- and demand-side components aimed at improving maternal and child nutrition program, referred to as the “Community-Based Stunting Reduction Program”. As part of the analysis, the researchers modeled the ERR of Generasi to find that benefits from Generasi grants include reductions in rates of child malnutrition and stunting and increases in rates of school participation, both of which have a positive effect on future earnings.

Results from the Generasi IE show a 9.6 percent reduction in the rate of incidence of severe malnutrition amongst 0-3 year olds and a 5.6 percent increase in the rate of participation of children

23 Ben Olken, Junko Onishi and Susan Wong (2014). Should Aid Reward Performance? Evidence from a field experiment on health and education in Indonesia. Online Appendices. Available at: http://economics.mit.edu/files/9380 24 The assumptions and analysis strategy are described in detail in Annex 3. 25This and subsequent section come from the Additional Financing Project Paper, World Bank (2014), “Project Paper on a Proposed Additional PNPM Support Facility Grant.”

18

aged 13-15 years in junior secondary school. Estimates from Alderman and Behrman (2006) show that averting stunting and low birth rate results in an average 7.5 percent increase in income once a child is of earning age (15 years old). Furthermore, a Mincerian wage equation found that an extra year of education increases earnings in Indonesia by an average of 6 percent (Mincer, 1974). The increase in education and earnings results in an extra US$ 4.82 per year per adult from when they start earning until retirement.26

The Generasi sub-district grant modeled in the ERR calculation was also found to have an additional stimulus effect by generating employment through the construction of health and education infrastructure.

ERR of MCA-I-Generasi “Community Based Stunting Reduction Program”: Cost Benefit Ratio: According to an analysis of the ERR of the Community-Based Stunting Reduction Program, the benefit-cost ratio associated with Generasi was found to be US$ 1.90 for every dollar invested. This means that for every US$ 1 spent, individuals and communities in the integrated program receive a return of US$ 1.90 that is derived primarily from: (i) income gains from increased education; (ii) savings from chronic disease averted; (iii) gains from the prevention of diarrhea; (iv) income gains for those who would have lived; and (v) income gains for those who would have died. The benefits of this US$ 1.90 were distributed as follows: US$ 1.14 to extremely poor and poor members of the community; US$ 0.49 to the near poor; and US$ 0.25 to the non-poor.

Efficiency of health and education infrastructure investments: A technical evaluation of infrastructure financed through PNPM-Rural shows that, in general, PNPM infrastructure sub-projects are at least 15 percent to 25 percent less expensive to design and implement than similar line Ministry-funded projects.27 Table 4 shows the unit costs for health and education infrastructure. Generasi’s unit costs are the same or similar to PNPM-Rural costs.

Table 4: PNPM vs. Government Unit Cost Analysis

PNPM Costs (IDR/units) Government Costs (IDR/Units) Sub-project Unit Low Average High Low Average High PNPM/

Gov’t (%)

School rehabilitation

M2 540,000 810,000 1,080,000 930,000 87

School furniture

M2 50,000 138,000 280,000 103,000 138,500 174,000 100

Public latrines M2 530,000 1,040,000 1,370,000 590,000 1,672,500 2,755,000 62

3.4 Justification of Overall Outcome Rating

Rating: Moderately Satisfactory

Relevance: Generasi’s PDO is relevant to the Bank’s CPS and the GoI’s aim of reducing poverty through improved access to health and education services. Generasi remains the Bank’s largest operation directly focused on causes of maternal mortality in Indonesia. However, shortcomings in the design of the original RF reduced the Project’s overall relevance.

Efficacy: The analysis above, drawing from a range of quantitative and qualitative and primary and secondary data sources, shows that Generasi has demonstrated substantial progress towards achieving both parts of its PDO.

26Note that this increase in school outcomes was assumed to occur at age 13, thus the labor productivity effect was realized two years later at age 15 in the modeled ERR. 27Technical Evaluation of Infrastructure – PNPM-Rural & Other Funding Sources. PSF. July 2012.

19

Efficiency: Several studies indicate that Generasi achieved a substantial level of efficiency and cost-effectiveness.

3.5 Overarching Themes, Other Outcomes and Impacts

(a) Gender Aspects. Worldwide, interventions intended to improve health and education outcomes tend to emphasize the role of mothers, who are regarded as the single most important caregiver in a child’s life. The majority of participants in Project activities and the majority of direct Project beneficiaries were women. However, it is recognized that household decision-making on health and education is influenced by male members of the household. Therefore, Generasi adjusted some of its processes and interventions to raise men’s awareness of maternal health issues so that they would support household investments in maternal and child health and nutrition services. In the most recent revisions to Generasi, the GoI has recognized the importance of the role that men may play in determining a child’s health and nutritional status by introducing an indicator that incentivizes male caregivers’ participation in pregnancy, parenting and nutrition classes.

(b) Institutional Change/Strengthening. Generasi has firmly established the viability of the CDD approach and developed significant assets that may enable it to influence the MoV in the implementation of Village Law. These include sub-district and district facilitators who have been trained in basic health and education issues; governance and anti-corruption practices; and social and environmental Safeguards screening procedures. In addition, the provincial and district government implementing units, the PMD, and the Sub-district Project Officer, based within the sub-district governor’s (Camat) office, have also gained exposure to basic health and education issues and experience in coordinating across local government departments responsible for health and education service delivery.

3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops

Field visits and beneficiary consultations: In addition to meetings, interviews and workshops at the national level, a joint team of the Bank and the GoI conducted visits to three provinces (Gorontalo, West Java and NTT). The mission consulted with provincial, district, sub-district and village level implementers and beneficiaries to address issues of relevance in the local context, including the impact of Generasi; the sustainability of Project efforts; and the compilation of lessons learned for expansion or replication. The missions met with a total of 233 people, of whom 124 were women.

Stakeholder workshops: On March 20, 2015, PMD invited key stakeholders, including representatives of the Ministry of Finance (MoF), MoEC, MoH, and other MoHA agencies to attend a workshop in Jakarta to comment on the initial findings presented through this ICR. The majority of participants agreed that Generasi has a substantial, direct positive impact on beneficiaries; that it generates lessons learned for demand-supply interaction at the local level; and that it can serve as a strategy to improve frontline service delivery in the future. A summary of issues discussed at this workshop is included in Annex 6.

An important message to come out of this and other workshops related to the need to prepare for the phase-out of Generasi in 2017 to ensure that national and sub-national governments have the budgetary space to accommodate some of the activities. This includes providing assistance to sub-national governments to prepare villages to fund a number of project activities and strengthening the role of the local governments in the further taking over, adapting and replicating Generasi activities to ensure on-going sustainable impacts on both demand- and supply-side.

4. Assessment of Risk to Development Outcome

Rating: Moderate

20

Risks related to the development outcome include:

Political Risk and Financing Implications: These risks are Modest, as the new government is supportive of CDD efforts and has made health and education an integral component of the fifth priority of its nine-priority agenda (Nawa Cita) for the 2014-2019 period. On the other hand, the GoI’s commitment to implementing the Village Law may place significant pressure on the national budget, which may affect the availability of funds for other village development programs. Nonetheless, the GoI has included Generasi in its 2015-2019 National Medium-Term Development Plan, a document which strongly influences internal funding decisions.

Institutional changes: These risks are rated Modest. Generasi is impacted by the institutional changes that followed the promulgation of the Village Law 6/2014 and by the shifting of the responsibility for the management of Generasi from MoHA to MoV. GoI has committed to rehiring and back paying all project staff, thereby mitigating the risk of losing capacity built up through previous years of implementation. The Bank task team will continue to provide enhanced implementation support, drawing on PSF resources to ensure that Project implementation gets back on track.

Technical risks: The technical risk is rated as Low, as the capacities of village facilitators have improved significantly, enabling them to lead communities in areas such as activity identification, beneficiary selection, the implementation of project activities, and interactions with frontline service providers. The relatively small size of Generasi sub-district grants relative to anticipated levels of transfers through the Village Law scheduled to begin in 2015 means that village facilitators and Project facilitators will require an expanded set of skills to ensure that Generasi’s problem diagnosis and participatory planning processes feed into village development plans.

Social risks: These risks are rated as Low. They relate to communities suffering from fatigue and failing to maintain enthusiasm for the Project. These risks can be mitigated by ensuring that Generasi is closely linked to local governments; technical agencies are more intensely involved in fostering a sense of community ownership; and that volunteer cadres and facilitators are provided with funds from the increased village budgets.

Based on the above, the overall risk to development outcome is rated Moderate.

5. Assessment of Bank and Borrower Performance

5.1 Bank Performance

(a) Bank Performance in Ensuring Quality at Entry

Rating: Moderately Unsatisfactory

The Bank introduced a set of design improvements over the pilot phase based on rigorous IE findings. However, the lack of a dedicated TTL for PNPM Generasi until after the original AF was designed and approved meant that there were design weaknesses, most notably: (i) an RF that included PDO-level indicators with target end dates that did not correspond with the Project’s implementation period; and (ii) the limited capacity of the implementing agency to manage PNPM-Rural support projects such as Generasi while simultaneously overseeing the nationwide scale-up of PNPM-Rural.

(b) Quality of Supervision

Rating: Satisfactory

After early delays, the Bank mobilized a multi-disciplinary team with substantial experience in supervising previous Bank-financed CDD projects in Indonesia operating in a widely dispersed area. The team was based in Jakarta to enable it to provide rapid support. Its location in Jakarta had

21

the added advantage of enabling it to draw from PSF resources to provide enhanced implementation support focused on technical and fiduciary issues.28 PSF resources enabled the Bank to mobilize support from other sections of the Bank, including Health, Nutrition, and Population (HNP) and Education, and from donor partners, most notably DFAT (formerly AusAid), to provide TA and enhanced support to the implementing agency and to other key Project stakeholders to mitigate design shortcomings during implementation. The team was able to identify and to address proactively emerging issues that might have affected the achievement of the PDO by providing support to the GoI. Amongst other actions, this support was utilized to develop a separate project management structure for Generasi within the implementing agency; to mobilize separate firms from those under PNPM-Rural to manage national and provincial Generasi specialists; to tailor internal audit protocols to the nature of Generasi sub-project investments; to develop an index of health and education supply-readiness that was used to prioritize project expansion; to provide TA to revise Generasi community health and education targets; and to further develop its gender strategy.

The Project Financial Management Specialist drew on PSF resources to mobilize a team of dedicated FM specialists to provide intensive implementation support during all ISMs; to closely monitor the implementing agency’s compliance with Bank fiduciary policies; to identify relevant actions, including the mobilization of additional FM TA within the project; and to revise internal audit protocols.

Similarly, the Project Procurement Specialist provided regular support to the task team and the implementing agency. While some delays were experienced in responding to certain GoI prior review requests related to the procurement of the first ASF contract, response times improved substantially over the period of implementation, resulting in a strong working relationship with the implementing agency. This contributed to the fast, effective procurement of the second ASF contract in 2013.

Compliance with Social and Environmental Safeguards was also a focus of regular ISMs. When shortcomings in the documentation of voluntary land donations under PNPM-Rural III were identified in 2011-2012, Safeguards specialists worked closely with the PNPM-Rural and Generasi task teams to develop the IGSES and to support the implementing agency to apply it through the Project TA and facilitation structure.

(c) Justification of Rating for Overall Bank Performance


The team took an extensive set of actions to address design weaknesses which included strengthening the implementing agency’s Project Management capacity and implementing a range of M&E activities needed to monitor Project achievements and provide relevant information to Project stakeholders on strengths, weaknesses and required follow-up actions. In addition, during Project implementation, the task team successfully drew on expertise from colleagues in HNP and Education to improve Generasi’s design to focus on maternal and child nutrition; children with disabilities; and the role of men in health and education decision-making. This process resulted in the approval of the stand-alone Project (P132585) and of Additional Financing to a value of more than US$300 million in the period from 2013-2017. It also resulted in the inclusion of Generasi as an integral component of the GoI’s 2015-2019 RPJMN.

5.2 Grantee Performance

28No Bank Budget was provided for implementation support for Generasi throughout the life of the Project.

22

(a) Government Performance


The GoI demonstrated a strong commitment to CDD and to Generasi more specifically, which it perceived as a means of fulfilling its commitment to the achievement of the MDGs. The GoI appreciated Generasi’s successes in linking demand and lower-level supply issues, as demonstrated by its inclusion in the RPJMN 2015-2019 and by the annual allocation of GoI financing, which exceeded 42 percent of the total value of the budget in the period from 2010 to 2013.

BAPPENAS provided proactive oversight and troubleshooting for project implementation, and the state audit agency (BPKP) expanding its audit coverage to more than 20 percent of targeted sub-districts. In addition, the level of collaboration with MoF was good. The GoI proactively requested an extension to the closing date to December 31, 2014, as it anticipated that a portion of the sub-district grant funds would only be utilized by July 2014. However, MoF did not act on Bank suggestions, documented in the AF Project Papers and in early drafts of the Restructuring Paper (Report No: RES12300), to revise the RF during Restructuring.

There were coordination challenges at the national and sub-national levels. Menkokesra, which served as the chair of the PNPM Steering Committee, attempted to coordinate MoHA’s community empowerment programs with sectoral line agencies. However, the effectiveness of these attempts was limited, as structurally MoHA was overseen by the Coordinating Ministry for Politics, Law, and Security.

Delays in budget revisions and execution, a perennial problem in Indonesia, negatively impacted Generasi implementation, despite assurance from BAPPENAS and MoF that budget revisions involving Grant funds could be completed quickly and out of cycle. As budget allocations were confirmed on an annual basis and only applied to a single year, there was no mechanism for securing upfront agreement on multi-year contributions to Generasi. Mid-year budget cuts to MoHA also negatively affected Project implementation.

Despite these issues, the GoI was able to ensure a reasonable level of quality at preparation; relatively smooth implementation; and compliance with Grant covenants and agreements. Thus, the performance of GoI was rated as Moderately Satisfactory.

(b) Implementing Agency Performance


The performance of the implementing agency, PMD, improved considerably after the delinking of Generasi from PNPM-Rural III and after the appointment of the Directorate of Traditional and Community Social and Cultural Empowerment as the implementing agency. The implementing agency acted quickly to hire National and Provincial Specialists during the period when the procurement of the Administrative Service Firm (ASF) responsible for managing Specialist contracts was delayed in 2012; to ensure the secretariat was fully staffed and functioning; to ensure monthly program progress reports were prepared and submitted on time; and to arrange regular monitoring missions to cover all provinces and districts on a rotating basis. The results of joint monitoring activities with the Bank were documented in Aide Memoires, with mutually agreed follow-up action plans. PMD prioritized financial oversight and controls over technical health and education issues, which resulted in improved FM ratings over the life of the Project.

The most significant management challenges for the implementing agency related to: (a) delays in budget effectiveness; (b) allocating budgets internally in ways that were not in line with BAPPENAS’s proposed allocations; (c) updating and distributing the PTO in a timely manner; (d) conducting post-procurement oversight. Nevertheless, PMD did support the implementation of

23

additional M&E efforts needed to provide information on Project performance as measured in terms of an increased number of indicators.

Taking all these factors into account, the performance of PMD was rated Moderately Satisfactory.

(c) Justification of Rating for Overall Recipient Performance


The rating of Moderately Satisfactory reflects the support provided by the GoI, both in its capacity as the recipient and as the implementing agency, in the implementation of the Project. Despite repeated delays in budget execution, the Project was disbursed fully and in a timely manner in alignment with AFs. Nevertheless, the lack of coordination between ministries, despite the efforts of a dedicated coordinating ministry and of the PNPM Steering Committee, and the failure to adapt the RF negatively impacted the rating.

6. Lessons Learned

a. Demand-side efforts are sufficient to increase the rate of utilization of health and educational services. However, complementary supply-side interventions are needed for higher level impacts: Generasi was effective in increasing the rate of utilization of services and in increasing service provider and community efforts to provide basic services. These are key steps to achieving high level outcomes, such as reductions in the rate of maternal mortality. However, communities and frontline service providers are limited in their ability to improve aspects of service provision quality, which is affected by factors such as the procurement of inputs such as vaccines; the district budgets for operations and maintenance; and so on. For the improved provision of frontline services over the long term, it is necessary to ensure engagement between demand and supply-side reform efforts.

b. Service providers can leverage social capital and community skills built through Generasi: Community health volunteers and ECED volunteer teachers play a critical role in facilitating the achievement of the “final mile” of basic service delivery. A key factor underpinning Generasi’s success has been the ability of the Project to provide resources to strengthen ongoing community efforts in the areas of outreach and service provision. These efforts represent a major asset on which local government efforts can build during the implementation of the Village Law.

c. Generasi invested in building the capacities of facilitators as generalists with strong networking capabilities, rather than as sector specialists: Given Generasi's institutional arrangements, it was not possible to expect facilitators to serve as experts on basic health and education issues. Rather, they needed to become effective generalists, with sufficient knowledge of health and education issues and Indonesia’s health and education programs, while seeking to network with local governments and CSOs to obtain technical guidance on specific issues.

d. An effective gender strategy must focus on both women and men, rather than exclusively on women: Specific Project interventions increased the level of participation of girls and women in community development initiatives. However, men play a key role in decisions that affect maternal and child welfare, such as directing household expenditures towards health and education services and nutritious foods. Efforts must therefore be made to involve men in health and education awareness raising, decision-making, and project implementation.

e. Persistence and measures to build trust were required before the national government was willing to engage in partnerships with CSOs: Through efforts of the Bank and other partners, including DFAT, the implementing agency became convinced of the potential value of involving CSOs, as these organizations had well-developed networks; a good understanding of the local context; and the established goodwill required to operate in the local environment. This

24

process provided an opportunity for the CSOs to demonstrate their unique role as development actors in their own right, enhancing community awareness raising and mobilizing and linking communities to front-line service providers.

f. Greater local government engagement is needed to institutionalize and sustain Project mechanisms: Generasi was integrated in the national government budget cycle, but only to a limited extent in district budgets. This resulted in the absence of an exit-strategy or sustainable financing strategy at the local level, and limited the opportunity for local governments to build on the Project's achievements or optimize the use of the assets invested in by the Project.

7. Comments on Issues Raised by Grantee/Implementing Agencies/Donors

(a) Grantee/Implementing agencies

No comments provided.

(b) Co-financiers/Donors

DFAT provided comments on the draft ICR by email on May 28, 2015. In summary, this email stated: “DFAT supports the report’s findings. It is a comprehensive and balanced assessment of a program that is highly relevant to the Indonesian context and has achieved good outcomes.”

The complete comments from DFAT on the Draft ICR report are attached in Annex 8.

(c) Other partners and stakeholders

On March 20, 2015, PMD organized a multi-stakeholder workshop attended by representatives of BAPPENAS, MoF, MoHA, MoEC and MoH and the implementing agencies to discuss the experiences and lessons learned during the implementation of the Project. Refer to Annex 6 for a summary of the issues raised during this workshop.

No further comments on the draft report were provided by other national stakeholders.

25

Annex 1: Project Costs and Financing

(a) Project Cost by Component (in USD Million equivalent)

Components Appraisal Estimate

(USD millions)Actual/Latest Estimate

(USD millions)Percentage of

Appraisal

Generasi Kecamatan Grants 60.73 58.62 96.52Consultants services and training and workshops for facilitation support under Part 2 of the Project

9.96 6.04 60.65

Consultants services, non-consultants services, and training workshops *for training and capacity building under Part 3 of the Project

2.39 2.27 94.93

Goods and incremental operating costs 0.24 0.24 102.29 Total Baseline Cost 73.32 67.18 91.62

Physical Contingencies 0.00 0.00 0.00

Price Contingencies 0.00 0.00 0.00

Total Project Costs 73.32 67.18 91.62 Project Preparation Costs 0.00 0.00 .00 0.00 0.00 .00

Total Financing Required 73.32 67.18 91.62

(b) Financing

Source of Funds Type of

CofinancingAppraisal Estimate

(USD millions)

Actual/Latest Estimate

(USD millions)

Percentage of Appraisal

Trust Funds 0.00 0.00 Indonesia - Program for Community Empowerment 73.32 67.18 91.62

26

Annex 2: Outputs by Component

The total Generasi budget in the period from 2010 to 2013 was approximately US$ 115.7 million. Of this, US$ 73.3 million was financed through TF097410. A portion of the 2013 Project budget was carried over for execution in 2014. From a total allocation of US$ 73.3 million, US$ 67.3 million was disbursed by the closing date of the TF. The remaining US$ 6.06 million has been allocated to the stand-alone project (TF14769) for execution in 2015. Table 1 below shows allocations and disbursements as of December 31, 2014.

Table 1: Allocations and Disbursements as of December 2014

Component Allocated Disbursement as of December 31, 2014

Balance as of December 31, 2014

Component 1 64,121,448.00 58,622,382.34 5,499,065.66 Component 2 6,904,831,00 6,702,891.43 201,939.57 Component 3 1,994,015,00 1,662,933.95 331,081.05 Component 4 302,706.00 302,706.00 0.00 Total 73,323,000.00 67,290,913.72 6,062,086.28

This annex demonstrates how the Project inputs produced outputs that contributed to the achievement of the PDO, which was for villagers in PNPM-Rural locations to benefit from improved socio-economic and local governance conditions. The causal chain is described in detail in section 3.2 of the main report. In the area of health, poor maternal and child health and nutrition leads to increased mortality and morbidity. In turn, this may lead to loss of economic output and increased expenditure on health, thus perpetuating the intergenerational cycle of poverty. In the same way, low levels of education deprive individuals of the skills they need to compete in national and international labor markets, leading to a lifetime of low skill, poorly paying, often labor-intensive jobs, which can accelerate physical decline. They also represent lost opportunities to gain access to better paying jobs. International evidence shows that increased utilization of the services targeted by this project improves socio-economic conditions.29 Thus, Generasi aimed to break the intergenerational cycle of poverty and to improve socio-economic conditions through the means described in the following paragraphs.

Project facilitators met with the target communities to explain the importance of the project’s 12 targeted health and education services. Following this socialization process, the facilitators then helped communities to identify ways to increase utilization of those services. Then they assisted them in the preparation of sub-project proposals to utilize the block grants provided under Component 1 to overcome demand-side and/or small-scale supply constraints, as determined by the community on the basis of their perceived needs. Demand-side investments included transportation subsidies for poor households to health and education facilities; community awareness-raising and training sessions related to priority health and education services; supplementary feeding for poor households; scholarships, books, and uniforms for poor students; etc. Supply-side investments included transportation support for community health volunteers, midwives, and doctors; the

29See, for example, Alderman H., Horton S., Rivera J. (2008), Copenhagen Consensus Challenge Paper: Hunger and Malnutrition; Alderman H. (2010). The economic cost to a poor start to life. Journal of Developmental Origins of Health and Disease, 1(1), pp. 19-25.

27

purchase of equipment for local health facilities and schools; the hiring of contract teachers; the rehabilitation and expansion of local health and education facilities; etc.

The inputs financed through Component 1, as described in Table 2 below, led to an increase in the rate of utilization of most target health and education services. MIS data in Table 3 shows the number of beneficiaries benefiting from the increased rate of utilization of services (as also shown in Section 3.2), while Table 4 shows certain supply-side outputs. Given the community driven nature of the Project, it was not possible to establish target outputs beforehand, so only actuals are shown. Component 1 financed the provision of block grants during the period from 2010 to 2013. In 2014, block grants were financed fully from the stand-alone Project (TF014769). By contrast, Components 2, 3, and 4 of the Project financed activities in the period from 2010 to 2014.

IOI Achievements According to MIS Data

Data from the MIS is used to demonstrate Generasi’s contribution to improved socio-economic conditions by measuring: (a) the percentage of villages that achieved all health and education targets annually (Table 8); and (b) the Project’s contribution to several Bank Core Sector Indicators (Tables 3-6).

Component 1: Spending Categories for Sub-Districts Grants (US$ 102 million; US$ 64.1 million from the Grant)

Table 2: Component 1 Investments by Sector and Sub-project Category, Cumulative 2010-2013

Sector Sub-project Category Total (US$ ) % of TotalHealth Infrastructure (build and rehabilitation) 4,783,586 4.7

Equipment for health facilities 4,945,335 4.9 Medicines 484,794 0.5 Financial assistance and transportation for pregnant, children to use services 6,066,518 6.0 House rental for assisted deliveries 111,312 0.1 Transportation for service providers (midwife, doctor, community health volunteers) 3,977,706 3.9 Financial incentives for health workers 5,123,141 5.0 Counseling and training 6,660,449 6.5 Supplementary feeding (malnutrition, school feeding) 29,996,985 29.5

Education Infrastructure (build and rehabilitation) 2,660,361 2.6 Equipment for education facilities 5,459,880 5.4 Financial assistance to poor students (uniforms, books) 12,299,726 12.1 Financial assistance for poor students (scholarships, transport) 10,454,549 10.3 Financial incentives and transport for education workers 6,349,790 6.2 Community awareness raising on education 1,463,315 1.4 Extracurricular activities 519,763 0.5

Other 337,824 0.3

Total 101,695,034

Table 3: Health Demand-Side Outputs, Cumulative 2010-2013

Indicator Number

28

Children between the age of 6 and 59 months receiving Vitamin A supplementation 1,692,047 Pregnant women receiving iron and folic acid (IFA) supplements (number) 1,889,546 Children under age five treated for moderate or severe acute malnutrition (number) 326,066 Pregnant/lactating women, adolescent girls and/or children under age five- reached by basic nutrition services

1,886,742

Births (deliveries) attended by skilled health personnel (number) 327,066 Children immunized - under 5 years against Polio 693,250

Table 4: Health Supply-Side Outputs, Cumulative 2010-2013

Indicator Number Health volunteers receiving training 155,457 Health volunteers receiving operational support 155,457 New health facilities constructed (community health costs, village health posts) 633 Health facilities rehabilitated 143 Health facilities equipped with improved sanitation facilities 107

Table 5: Education Demand-Side Outputs, Cumulative 2010-2013

Indicator Number Students receiving financial support (scholarships, uniforms, transportation subsidies) 727,803 Students receiving nutrition support (school feeding programs) 123,679 Students receiving textbooks 177,816

Table 6: Education Supply-Side Outputs, Cumulative 2010-2013

Indicator Number Additional classrooms built or rehabilitated at the primary level resulting from project interventions

269

Schools equipped with improved sanitation facilities 251 Contract teachers and teaching assistants receiving operational support 6205

Tables 3-6 underestimate the total number of health and education outputs, as on average, data from only 75 percent of the total number of target villages was uploaded to the web-based MIS annually.

As the body of this report makes clear, over time the proportion of project funds invested in education relative to health declined, with an increased preference for communities to make investments in the health sector (see Figure 1).

29

Figure 1: Changing Spending Pattern Prioritizing Health

Table 7 below shows the results of the PMS in 2013 and 2014 compared to baseline data from different sources for the Project, which enables a monitoring of the Project's achievements against previous measured indicators. Interestingly, in terms of a number of these indicators, lower levels of achievement were recorded in 2014 than in 2013. It is hoped that the planned IE will determine the reasons for this.

IOI Achievements According to 2013 and 2014 Performance Monitoring Survey Findings

PMS data for 2013 and 2014 showed that on average, Generasi increased the rate of utilization of nine of 12 targeted health and education services over baseline values (Table 7). The rate of utilization of services including percentage of pregnant women receiving four prenatal care visits, percentage of pregnant women receiving 90 iron tablets, percentage of infants for which weight increases each month saw relatively large increases. On the other hand, the percentage of children under-five receiving vitamin A twice a year and minimum attendance rate of 85% for all primary school-aged children declined. Qualitative research is currently being conducted to identify the factors influencing variation in performance across the targeted services, but ISM findings suggest that availability of services, such as Vitamin A tablets, is a constraint on community performance. In addition, given high levels of funding from other sources for primary education30, communities tended to used Generasi resources on health services.

30 Twenty percent of Indonesia’s state budget by law must be spent on education, through programs such as School Operations Grants (BOS—Bantuan Operasionel Sekolah) and Scholarships for Poor Children (BSM—Bantuan Siswa Miskin).

30

Table 7: 2013 and 2014 PMS Results

Targeted Health Services Baseline PMS 2013 2013 Target from

P132585

PMS 2014

2014 Target from

P132585

% change from

baseline to 2014

Percentage of pregnant women receiving four prenatal care visits

61.40 (2010 RISKESDAS)

70.03 >70 91.22 >75 +29.82

Percentage of pregnant women receiving 90 iron tablets


39.40 N/A 32.43 N/A +14.43

Percentage of deliveries assisted by a trained professional


89.17 75 88.89 80 +6.69

Percentage of women receiving two postnatal care visits


50.83 N/A 47.50 N/A +0.7

Percentage of children under one year receiving complete childhood immunizations

75.00 (2010 IE Endline Survey)

77.15 N/A 83.82 N/A +8.82

Percentage of infants for which weight increases each month31

N/A 64.77

N/A 75.36 N/A +10.58

Percentage of children attending monthly weight checks for under threes and biannually for under-fives

75.00 (2012 SUSENAS)

74.62 75 73.49 80 -1.51

Percentage of children under-five receiving vitamin A twice a year


56.19 N/A 52.12 N/A -19.38

Targeted Education Services Primary school enrollment of all children 6 to 12 years old


99.55 N/A 98.67 N/A +0.17

Minimum attendance rate of 85% for all primary school-aged children


91.18 N/A 89.64 N/A -6.36

Junior secondary school enrollment of all 13 to 15 years old


95.83 65 93.75 70 +6.35

Minimum attendance rate of 85% for all junior secondary school-aged children


93.36 N/A 93.20 N/A +7.2

Table 8 shows that in the period from 2010 to 2013, the percentage of villages achieving each of the specific targets increased by the rate of one percent to 26 percent, depending upon the indicator. The proportion of villages that achieved the individual indicators varied from between 72 percent to 92 percent. Throughout this period, the target related to immunization coverage has been the most difficult for villages to achieve. While the average coverage increased from 63 percent to 72 percent, this was still below the targeted level of 80 percent. The failure to achieve this target was partly due to the fact that supplying vaccines is the responsibility of district health offices, which was beyond the control of the Project. The percentage of Generasi villages achieving all 12 indicator targets increased from 23 percent in 2010 to 39.7 percent in 2011; and to 48.5 percent in 2013.

31 Measured as weight increases each month over three consecutive months.

31

Table 8: Percentage of Project Villages Achieving Target for Improved Service Utilization

Targeted Health Services 2010 2013 % Change Percentage of pregnant women receiving four prenatal care visits

73 84 +11

Percentage of pregnant women receiving 90 iron tablets

91 92 +1

Percentage of deliveries assisted by a trained professional

54 80 +26

Percentage of women receiving two postnatal care visits

76 85 +9

Percentage of children under one year receiving complete childhood immunizations

63 72 +9

Percentage of infants for which weight increases each month

75 81

+6

Percentage of children attending monthly weight checks for under threes and biannually for under-fives

77 86 +9

Percentage of children under-five receiving vitamin A twice a year

80 86 +6

Targeted Education Services Primary school enrollment of all children 6 to 12 years old

77 88 +11

Minimum attendance rate of 85% for all primary school-aged children

84 90 +6

Junior secondary school enrollment of all 13 to 15 years old

80 89 +9

Minimum attendance rate of 85% for all junior secondary school-aged children

90 91 +1

Component 2: Technical Assistance (US$ 8.5 million; US$ 6.9 million from the Grant)

This component financed salaries and operational costs for the National and Provincial Specialists whose contacts were managed by an Administrative Services Firm (ASF) contracted by the implementing agency. At the National level, these specialists included Complaints Handling, Financial Management, MIS, Health, Education and Training Specialists. In addition, in each province, Project Management and MIS Specialists were deployed. Together, the National and Provincial Specialists were responsible for Project planning; producing technical documents such as the Operations Manual, training modules, internal audit manual, and Safeguards guidelines; and capacity building and oversight of District and Sub-District Facilitators. The ASF was responsible for delivering operational and administrative support to the National and Provincial Specialists, and for organizing the national and regional workshops.

This component also financed the operational costs and salaries of District and Sub-District Facilitators and Computer Operators. These facilitators provided direct TA to Project communities, local governments, and local service providers in support of achieving the 12 health and education services.

32

Table 9: Number of Project Facilitators and Specialists Financed

Year Sub-district facilitators

District facilitators

Provincial Specialists

National Specialists

Total (US$ )

2010 212 27 12 4 By Loan PNPM III

2011 290 27 12 4 1,794,667 2012 369 83 16 4 1,422,949 2013 369 83 16 27 2,928,433 2014 499 127 22 47 307,666

Total 6,453,715

Component 3: Capacity Building (US$ 4.5 million; from US$ 1.99 million from the Grant)

This component financed training for District and Sub-District Facilitators and District Computer Operators; for the production of communication and the socialization materials for use by Sub-District facilitators when working with Project communities; and for the printing and distribution of the Operations Manual (PTO). These activities were delivered by the Event Organizers (EO) contracted by the implementing agency.

Year Activity Description Personnel Trained

Amount (US$ )

2012 EO for Training Services (Pre-Service Training for District and Sub-District Facilitators; Refresher Training for District Facilitators and Computer Operators)

233,031

2013 Printing and Distribution (PTO, Posters and Leaflets) N/A 248,385 2013 EO for Training Services (ToT; Refresher Training for

District and Financial Facilitators, Computer Operator and Sub-District Facilitators; Training on Mitigation of Corruption for District and Financial District Facilitators, National Specialists and National Project Management Specialists (Secretariat)

782,031

2014 EO for training for ToT, Pre-Service for Facilitators and Computer Operator, workshop for Training Module Development

410,820

Total 1,674,268

Component 4: Project Management Support (US$ 0.7 million; US$ 0.3 million from the Grant)

This component financed the provision of salaries to certain Secretariat and Secretariat support staff and of travel and operating costs. The Secretariat is responsible for budgeting, administration, invoicing, procurement and disbursement. It consists of one Coordinator, six Specialists (Planning and Program Development; Budgeting; Human Resource Development (HRD); Procurement; Invoice and Disbursement Specialist; Policy Analyst and Reporting Management) and is assisted by six Junior Specialists. Fifteen support staff are also deployed to conduct technical and administration tasks. A small fraction of overall Secretariat costs are financed by the Project.

This component, which was fully disbursed by the end of 2014, enabled PMD to realign the PNPM Generasi Secretariat to reduce overlap with the national TA structure (National Specialists). The realignment of the Secretariat also helped to improve the overall quality of project management, facilitating the provision of support for multi-stakeholder workshops at the national level and improved coordination with the National Audit Agency in the area of the

33

implementation of Generasi-related external audits. The total value of funds from TF097410 used to finance this category stands at US$ 348,262.

34

Annex 3: Economic and Financial Analysis

A detailed explanation of the methodology that was used to calculate the cost effectiveness of Generasi compared to PKH and similar global programs is laid out in Olken et al. 2014 (http://economics.mit.edu/files/9380). To calculate the cost effectiveness of the program, Olken, Onishi and Wong (2014) calculated the total number of “points” the program created, using the weighting scheme agreed upon in advance and shown in Table 1 (Olken at al. p. 15) and the point estimates for the impact of the program from Table 3 (Olken et al. p. 22-25). They then divide the total cost of the program by the total number of points created to generate a “cost per point”, which can then be interpreted using the point values in Table 1 (Olken et al. p. 15).

To summarize, the points were created using the following formula. For each of the 12 Generasi indicators, the program established a constant predicted minimum attainment level in a particular village to be equal to 70 percent of the average achievement level for villages with similar levels of access to health and education providers and numbers of beneficiaries, calculated from the 2004 SUSENAS household survey and 2003 PODES census of villages. They then calculated how many bonus points a particular village and beneficiaries in the village earned.

To calculate the costs of the program, the authors divide the expenditures into transfers to households and real expenditures (i.e. real allocation of resources). They count school supplies, school fee subsidies, health care subsidies, and supplementary food as transfers, and all other expenditures as real expenditures. About 75% of the block grant is spent on transfers by these definitions.

35

Annex 4: Grant Preparation and Implementation Support/Supervision Processes

(a) Task Team members

Names Title Unit Responsibility/

Specialty Lending/Grant Preparation Jan Weetjens Lead Social Development

Specialist GSURR Sector Manager

John Victor Bottini Senior Social Development Specialist GSURR Task Team Leader

Susan Wong Lead Social Development Specialist GSURR Operations Yogana Prasta Operations Adviser EACIF FM & Disbursement Unggul Suprayitno Senior Financial Management

Specialist EAPFM Financial Management

Bisma Husen Senior Procurement Specialist EAPPR Procurement Melinda Good Senior Counsel LEGES Legal Indira Dharmapatni Senior Operations Officer EASIS Social Safeguard Juliana Wilson Consultant EASIS Communication Sadwanto Purnomo Social Development Specialist GSURR Operations Gerda M. Gulo Operations Analyst GSURR Operations

Supervision/ICR

Robert Wrobel Senior Social Development Specialist GSURR

Task Team Leader from November 2010 to project closing in December 2014

Sadwanto Purnomo Social Development Specialist GSURR Operations Gerda Gulo Operations Analyst GSURR Operations Ali Winoto Subandoro Health Specialist GHNDR Operations (Health) Yasmeen Nasser Al Tabbaa Social Development Specialist EASER Operations

Unggul Suprayitno Senior Financial Management Specialist GGODR Financial Management

Alexander B. Setiadji Social Development Specialist GSURR Governance

Festina Lavida Operations Officer GSURR Financial Management (Audit)

Hanggar Irawan Financial Management Specialist GSURR Budget and Disbursement

Audrey Sacks Social Development Specialist GSURR Monitoring and Evaluation

Yulia Herawati Social Development Specialist GSURR Monitoring and Evaluation

Junko Onishi Senior Social Protection Specialist GSPDR Monitoring and Evaluation

Juan Martinez Senior Social Scientist GSURR Social Safeguards Dennie Stenly Mamonto Consultant GSURR Safeguards Juliana Wilson Consultant EASID Communication Maureen Tuahatu Rustandi Consultant GSURR Communication

36

Yohanes Susilo Consultant GSURR Training Specialist Zejd Muhamad Consultant EASID MIS Specialist Budi Wijoyo Temporary GSURR MIS Assistant Paul Boon Consultant GSURR ICR Team Leader

(b) Staff Time and Cost

Stage of Project Cycle Staff Time and Cost (Bank Budget Only)*

No. of staff weeks USD Thousands (including travel and consultant costs)

Lending FY11 20.40 93.81 FY12 111.58 346.66 FY13 130.50 544.53 FY14 156.96 383.67 FY15 291.44 102.81

Total: 522.25 1,660.11 Supervision/ICR

FY11 0.00 11.65 FY12 0.00 101.33 FY13 0.00 166.68 FY14 0.00 80.63 FY15 0.00 46.99

Total: 0.00 407.29

*Staff time and cost are funded through separate TF and different P number. PNPM Generasi BETF is TF092194 under P111966.

37

Annex 5: Beneficiary Survey Results

N/A

38

Annex 6: Stakeholder Workshop Report and Results

On March 20, 2015, the implementing agency, PMD, invited key stakeholders to participate in a workshop in Jakarta to discuss the initial findings of the ICR process. Attending the workshop were representatives of the MoF, MoEC, MoH, BAPPENAS and MoHA (PMD); WB/PSF Generasi staff and consultants; PMD Secretariat staff; and NMC consultants.

The majority of participants agreed that Generasi has had a substantial direct impact on beneficiaries, providing learning for demand-supply interactions at the local level and playing a significant role in the development of a strategy to improve frontline service delivery in the national level strategy currently being designed by BAPPENAS.

The participants agreed that there was a need to prepare for the phase-out of Generasi in 2017 to ensure that the GoI and Local Governments (LGs) have both the necessary strategies and the budgetary space to accommodate the continuation of a number of activities developed under the project and also to help LGs prepare the villages and communities to take over and fund a number of these activities. Strengthening the role and capacity of the LGs to take over and implement a number of Generasi activities adapted to local circumstances was regarded as the single most important activity for ensuring that impacts on both the demand and supply-side are sustained.

Observations regarding the Bank’s initial findings:

The implementation of Generasi has demonstrated that an incentivized community empowerment approach with a clear focus and with clearly defined indicators determine achievement of sectoral targets can be effective. Although shortcomings in implementation are recognized, the project demonstrated that the provision of financial resources and good facilitation is a recipe for success.

Generasi was successful in facilitating the achievement of 10 out of the original 12 targets. In addition, it was sufficiently flexible to enable the adaption of the indicators in line with lessons learnt from implementation and evolving government priorities. For example, indicators related to education changed to place an emphasis on those with the lowest level of access to education, rather than continuing to focus on attendance rates that had already been achieved. The fact that close to 50 percent of all the villages involved achieved all 12 targets provides additional evidence of the success of collective action in both education and health sectors.

Generasi provides input to assist in the drafting of GoI’s frontline service delivery agenda, with this agenda mandating the provision of a minimum level of basic services for the poor by 2019. In this regard, Generasi can act as a vehicle for implementation that links demand and supply-side interventions. There is emerging evidence that sectors linking their activities to Generasi achieve a higher level of coordination in terms of specific targets. However, in the long run, such an approach requires a strong multi-sectoral approach to ensure good coordination and effective collaboration.

The link between Generasi and LG sectoral programs needs improvement, given that focuses on the provision of basic services that are the responsibility of LGs. The institutionalization of such efforts should be prioritized with a focus on preparing systems at both the local government and village level. This will become particularly important in the context of the increased financial resources villages will receive with the full implementation of the Village Law.

39

Facilitators play a vital role in determining the success of Generasi, with both the quality and quantity of the project facilitators being decisive for success. LGs can learn from Generasi's systems for the deployment, management, and recruitment of facilitators in their endeavors to replicate Generasi activities in the context of the phase-out of the project by the end of 2017.

Observations regarding lessons learned:

LGs and local institutions will determine the extent to which the impact and implementation of Generasi initiated activities are sustained in the long term. Hence, Generasi needs to work increasingly closely with these institutions to build their capacities and to strengthen their performance.

With the Generasi project scheduled to run for almost three more years and with the current level of interaction between sectoral ministries and BAPPENAS, there is space to work closely with BAPPENAS and LGs to define a clear phase-out strategy and to ensure uptake and replication by LGs through national policy development through integration with the GoI's agenda for frontline service delivery for the poor.

The GoI decision to shift the responsibility for the implementation of Generasi agency from MoHA to MoV effects implementation in the field. There is a need to encourage the two institutions to engage in discussions related to future implementation to ensure that these initiatives benefit from MoHA's experience while at the same time fulfilling MoV's mandate.

Improvements must be made to systems for monitoring and evaluation to ensure the availability of evidence to adjust implementation. The recurring problems with the timeliness and reliability of MIS data has influenced PMD’s capability to monitor and redress problems in a timely manner. Improved data collection and to facilitate monitoring and evaluation could be achieved by linking data collection and submission to the facilitator remuneration merit system or through other means.

Facilitators and village cadres can play an important role in linking demand and supply. There improved understanding of sectoral issues facilitates community mobilization and action. Adding a measurement for community empowerment processes and collective action and performance might help indicate the influence of support to build social capital capacity on the achievement of sectoral targets.

Some thought should be given the means for generating learning from the implementation of the Generasi approach and for the adaption of successful initiatives to on-going GoI efforts, such as MP3KI (the Master Plan for Accelerating and Expanding Poverty Reduction in Indonesia) and P2B (the Livelihood program).

40

Annex 7: Summary of Borrower’s ICR and/or Comments on Draft ICR32

Lessons Learned: (+) include -

PNPM MPd Generasi supported participatory village planning and uptake of education and health in the mid-term village development plans;

PNPM MPd Generasi was able to map the poor households;

PNPM MPd Generasi design allowed members of communities to work together to develop solutions at the community level to address issues that were previously seen as household or private domain issues. The increased dynamics in the community could explain a spill-over effect, as these villages seemed to attract a greater number of volunteers involved in the health sector and an increased number of active school committee members. They also facilitated the greater involvement of women. This is an important lesson for the future, particularly in the area of the allocation of village level funds for education and health purposes;

Providing Block Grants to communities to be used for self-determined activities to stimulate better services in the health and education sector is an effective means of addressing both demand- and supply-side issues. By allocating a major portion of the block grants to performance, a positive impact was achieved in both sectors, especially in the area of malnutrition. Using block grants as an incentive to stimulate better performance has been shown to have a statistically significant impact;

The Musyawarah Desa Serah Terima (MDST) improves block grant management and transparency;

Targets achieved by PNPM MPd Generasi include an increase in the proportion of pregnancies assisted by profession personnel; a decrease in the rate of maternal mortality; and a decrease in the rate of infant mortality. Although still fluctuative, the proportion of students dropping out of education has decreased and participation rates have increased (although not yet to a statistically significant level);

The involvement of national, provincial and district government is rated as Satisfactory. However, greater efforts are needed to strengthen the village apparatus and community groups, especially in relation to better service delivery;

The World Bank role is very supportive of measures to improve program management;

The solving of cases in PNPM MPd Generasi through litigation and non-litigation measures helps to ensure that people understand the paradigm of community responsibility and be need to solve the problems by leverage in local customs and through other means.

32 This Annex is the ICR Author’s translation of the Lessons Learned and the Recommendations based on the “PROJECT COMPLETION REPORT - Implementation of PNPM MPd Generasi - Grant Number TF-097410. 2010 – 2014” compiled by DG PMD in MoHA.

41

Communities still hope for further assistance under PNPM MPd Generasi. Thus, it is important to prepare a clear Exit Strategy that can ensure sustainability in terms of human resources, organizational structure and systems and regulations;

There is a need to improve the quality of data and to facilitate a more in-depth discussion regarding education and health amongst the technical ministries;

Facilitator skills and knowledge related to cross-program work need to be strengthened;

Empowerment is still more visible at the individual level than the community level. Hence, there is a need to improve the skills of communities to implement monitoring and evaluation;

Local governments need to be involved in determining PNPM MPd Generasi locations;

PNPM MPd Generasi indicators relate to scope of reach. There is a need to coordinate health service delivery at the village level, although these indicators currently focus on the sub-district level;

Continuous changes to the composition of government staff at the district and provincial levels hamper the implementation of programs;

To address malnutrition, the recurring issue is the need for resources to support family members accompanying babies or children. The lack of such resources means that caregivers often return to their places of residence early, before the issue has been fully addressed;

The allocation of health staff and teachers is still not even, with many villages being severely under resourced;

Local customs often mean that after completing primary school, children fail to make the transition to junior secondary school because they are required to provide labor to assist their families;

Measures need to be taken to build the capacities of village apparatus and community institutions;

Preventive and promotive activities have to be increased to ensure impact;

The main problem in achieving Indicator 5 (complete immunization) is the lack of vaccine storage facilities and the fact that vitally necessary supporting infrastructure, including electricity, is often available only intermittently. In the case of indicator 6 (babies and infants weighed every month and showing weight increases), the main constraint is the limited community understanding of what constitutes nutritious food;

PNPM MPd Generasi still only possesses limited promotional materials. The development of an improved communication strategy will increase the reach of the program;

There is a need to adapt the Safeguards implementation guidelines for PNPM MPd Generasi and for procurement, with emerging Post Review needs to be analyzed and addressed.

Recommendations:

1. PNPM MPd Generasi has already made a positive impact in terms of building the capacity of the community, program implementers and government staff related to participatory planning and implementation of programs to ensure that these programs are transparent

42

and accountable. Nevertheless, there is a need to reconstruct the planning stage and to ensure that it is integrated with the regular planning system and that it supports the implementation of the Village Law;

2. The commitment of the local government to participatory planning is not yet maximal. Thus, there is a need to approach local parliaments to raise their awareness of activities and activities to encourage the passage of legislation that provides a supportive environment for the implementation of PNPM MPd Generasi and/or associated activities. PNPM MPd Generasi offers an approach which local parliaments can adopt to prepare legislation that supports a similar strategy, including the follow-up management of PNPM MPd Generasi assets and activities and measures to ensure that midwives are stationed and operate at the village level with the appropriate support resources;

3. The capacity of facilitators is an extremely important factor in the implementation of innovative approaches to identifying the problems, assets and needs and to ensure effective and efficient use of the block grants. The planning methodology utilized to support the implementation of the new Village Law must be understood by all program actors to ensure an integrated approach to development;

4. As there has not yet been an official hand-over of the management of PNPM MPd Generasi activities, there is need to develop technical operational guidelines for program implementation;

5. ECED activities should be implemented in all PNPM MPd Generasi locations, given the fundamental importance of these to a child’s development (0 – 6 Tahun). This could be achieved through the optimization of roles in Posyandu and through a low-cost implementation approach, given that these activities do not contribute to the achievement of the program's indicators;

6. Priority in use of PNPM MPd Generasi BG in the future includes:

Development of village health posts and clinic;

The provision of support and mentoring for the Posyandu; and

The provision of support for the mentoring and management of ECED.

7. The involvement of the village apparatus (village head and other leaders and village institutions / BPD) in the implementation of PNPM MPd Generasi is necessary to ensure the sustainability of impacts on education and health so they become an integral part of the agenda included in the village medium term development plan (RPJMDes) and the annual implementation plan (RKPDes), including on matters related to infrastructure, health workers, service quality and improved access by the community;

8. Complaints handling in PNPM MPd Generasi has already resulted in the resolution of a number of cases and has supported the audit processes significantly. This was achieved through the regular coordination between the central and local government, supported by BPKP (central and provincial representatives) and other program implementers. There is a need to compile and update cases and to determine their status;

9. To the greatest extent possible, it is necessary to ensure that the national budget (DIPA) for PNPM MPd Generasi is not delayed, as delays of this sort have a serious negative impact on the quality of performance;

43

10. The regular mapping of consultants and facilitators (monthly) needs to pro-actively anticipate the number of facilitators to be recruited to avoid shortages in certain locations;

11. There is a need to mobilize national and provincial Financial Mangement Spesialists (FMS) to optimize support for administration and financial management; and

12. Measures should be implemented to improve coordination between the national actors involved in PNPM MPd Generasi to ensure that decision-making and program policy decision can be taken without delay to program implementation.

44

Annex 8: Comments of Co-financiers and Other Partners/Stakeholders

Comments from Elena Rose, First Secretary, Governance and Social Development, Department of Foreign Affairs and Trade (DFAT) on the May 8th Draft ICR report.

DFAT supports the report’s findings. It’s a comprehensive and balanced assessment of a program that is highly relevant to the Indonesian context and has achieved good outcomes.

The report could present clearer recommendations upfront to highlight challenges and lessons learned for next phase of programming and other related programs, given how rich Generasi is in these lessons. A few recommendations drawing on points in the text might include:

o Noting the report’s point on the importance of collaborating with supply-side stakeholders (MOH, MOEC) and the lack of this collaboration being one of the challenges the program faced, one recommendation could address how Generasi could improve collaboration with other stakeholders and in what area of the program it could/should pursue collaboration (e.g. better networking, between Generasi facilitators and school etc).

o The report notes that Generasi benefited from being ‘co-located’ in areas where other investments were operating because of the strengthened enabling environment (e.g. ACCESS in NTB and AIPMNH in NTT), a recommendation on how to pursue this in future could be useful, particularly given the delays faced with central government in seeking formal agreement for programs to collaborate.

o Including recommendations around facilitating a smooth transition of the program to MoV, including as relates to improving the quality of Generasi going forward (building on some of the points under (b) on p 24), could be useful.

Challenges to sustainability could be more emphasised, giving greater foothold to the substantial work that needs to follow from 2015-17 to institutionalise aspects of Generasi into Village Law implementation and potentially through line ministries. Supporting the formulation of a clear GOI vision for this and preparing for a smoother transition than has occurred for PNPM-Rural is critical to avoid both: losing the ground made up by Generasi; and the same issue of communities not allocating sufficient resources to health and education coming up again under the VL.

The report does not fully address some particular challenges that confronted Generasi such as:

o Excessive accumulation of money in the multiyear account, which was apparently one of the main causes of fraud/misuse of funds and compromised the quality of the program as it increased pressure to spend;

o How the workload of PNPM facilitators affecting the quality of their work – PSF did a study in late 2013 on facilitators’ workload, which showed most of their time was spent on administrative work, reducing their scope to conduct better quality facilitation; and

o The variable quality of facilitators, which was not uniform across program locations.

45

The definition of “improved local governance conditions” seemed somewhat narrow and the rating of substantial (pp 20-21) perhaps a bit too generous without further evidence. For instance, governance is predominantly focused on demand-side aspects and participation in processes is not necessarily a strong indicator of a sustained improvement in governance.

While Generasi achieved much, the assessment that it substantially improved socio-economic conditions of the community may require further evidence than is presented in the report. The argument that it averagely increased the rate of utilisation of nine of 12 targeted health and education indicators (i.e. it met output indicators) doesn’t – from what is outlined in the report – show a clear link to how the program achieved an outcome-level goal like improving socio-economic conditions.

46

Annex 9: List of Supporting Documents

Bhutta, Zulfiqar A., et al. “Evidence-based Interventions for Improvement of Maternal and Child Nutrition: What Can Be Done and at What Cost?” The Lancet 382.9890, (2013): 452-477.

Glewwe, Paul, and Elizabeth M. King. “The Impact of Early Childhood Nutritional Status on Cognitive Development: Does the Timing of Malnutrition Matter?” World Bank Economic Review 15.1, (2001): 81-113.

Grantham-McGregor, Sally M., et al. “Nutritional Supplementation, Psychosocial Stimulation, and Mental Development of Stunted Children: The Jamaican Study,” The Lancet 338.8758, (1991): 1-5.

Grantham-McGregor, Sally M., et al. “Effects of Early Childhood Supplementation With and Without Stimulation on Later Development in Stunted Jamaican Children,” The American Journal of Clinical Nutrition 66.2, (1997): 247-253.

Grayman, J. (2014), Opportunities and Approaches for Better Nutrition Outcomes through PNPM Generasi - A Qualitative Study, PNPM Support Facility (PSF) and Poverty Reduction Support Facility (PRSF).

Hoddinott, John, et al. “Effect of a Nutrition Intervention during Early Childhood on Economic Productivity in Guatemalan Adults,” The Lancet 371.9610, (2008): 411-416.

MIS Data Analysis to Support the Feasibility Study of PNPM Generasi Long Term Evaluation, Internal Project Report, (2014).

Johnston, Francis E., et al. “Interaction of Nutritional and Socioeconomic Status as Determinants of Cognitive Development in Disadvantaged Urban Guatemalan Children,” American Journal of Physical Anthropology 73.4, (1987): 501-506.

McLaughlin, K. Indonesian Village Health Institutions, A Diagnostic. Unpublished manuscript, (undated).

Olken, B.A., Onishi, J. and Wong, S. (2013), “Should Aid Reward Performance? Evidence from a field experiment on health and education in Indonesia”.

PNPM Support Facility (2013). Performance Monitoring Survey for PNPM Generasi.

PNPM Support Facility (2014). Summary of Findings from the Piloting of Integrating Early Childhood Education and Development Indicator into PNPM Generasi Program, Internal Project Report.

Pollitt, Ernesto, William Mueller, and Rudolph L. Leibel. “The Relation of Growth to Cognition in a Well-Nourished Preschool Population,” Child Development, (1982): 1157-1163.

Qureshy, Lubina F., et al. “Positive Returns: Cost-Benefit Analysis of a Stunting Intervention in Indonesia.” Journal of Development Effectiveness 5.4, (2013): 447-465.

UNICEF (2011). Indonesia Country Study on Out of School Children.

Wai-Poi, Matthew Grant, “Chapter 2: Large-scale Child Health and Nutrition Interventions: Indonesia’s Posyandu,” Doctoral Dissertation, Columbia University, (2011).

World Bank (2008). Country Partnership Strategy for Indonesia FY2009-2012.

47

World Bank (2010). Indonesia’s PNPM Generasi Program. Interim Impact Evaluation Report, PNPM Support Facility (PSF)

World Bank (2010). Project Paper on a Proposed Grant Amendment in the Amount of USD 10.2 Million to the Republic of Indonesia for Additional Financing for the Third National Program for Community Empowerment in Rural Areas, Report No: 56112-ID.

World Bank (2011). Indonesia’s PNPM Generasi Program. Final Impact Evaluation Report, PNPM Support Facility (PSF).

World Bank (2011). Project Paper on a Proposed Grant Amendment in an Amount of USD 28.123 million to the Republic of Indonesia for Additional Financing for PNPM Generasi Scale-Up,).

World Bank (2012). Country Partnership Strategy for Indonesia FY2013-2015.

World Bank (2012). Project Paper on a proposed Grant Amendment in an Amount of USD 35 Million to the Republic of Indonesia for Additional Financing for PNPM Generasi Scale-Up – Updated for Third Installment of Additional Financing. April 27, 2012.

World Bank and Monash University (2013). A Cost Benefit Analysis of a Community Based Nutrition Project in Indonesia: Calculating an Economic Rate of Return.

World Bank (2013). Implementation Completion and Results Report – Third National Program for Community Empowerment in Rural Areas.

World Bank (2013). Project Appraisal Document on a Proposed PNPM Support Facility Grant in the amount of USUS$ 31.70 million to the Republic of Indonesia for the National Program for Community Empowerment in Rural Areas – Healthy and Bright Generation (PNPM Generasi), Report No: 497, (April 9, 2013).

World Bank (2013). Restructuring Paper of ID-TF Additional Financing of PNPM Rural III for Generasi (Comm CCT) Scaling-up Program (P122032) August 2, 2010 to the Republic of Indonesia, World Bank Report No: RES12300, (November 8, 2013).

World Bank (2014). Project Paper on a Proposed Additional PNPM Support Facility Grant in the Amount of US151.7 million to the Republic of Indonesia for the National Program for the Community Empowerment in Rural Areas – Healthy and Bright Generation (PNPM Generasi), Report No: 829, (June 23, 2014).

... '"' '"' INDONESIA o CffiE$ JIN!) IO'WtlS

• l'RO'o'IN<t <APITAlS ... * t"-'>.t~ ... l CAl'ffAl -....__ _,, RNei::S

--~HROA(>S

-- "'""""' -- l'll()VIN(f fi(XJNIW!lf$

••• ,. - · - INTERNA1JONJ\l &OUNDo\BES

,,. •

, ... """'"""

PACIFIC OCEAN

• •

• 0

h INDIAN OCEAN •

' Ii~ ' ;. . • ~ '( I • •• - " " ~

w

AUS RALIA w

'f's· l?S' ~ ,,.. '" , .. ~ w

(