pcr evaluation note for public sector operations · basic project data project title: zanzibar...

1. BASIC INFORMATION a. Basic project data Project title: Zanzibar WATER SUPPLY AND SANITATION PROJECT Project code: Instrument number(s): 2100150017993 & 5800155000251 Project type: Project Operation

Sector: Water and Sanitation

Country: Tanzania Environmental categorization (1-3) : 2Processing Milestones Key Events Disbursement and Closing

date Date approved: 11 Nov 2008

Cancelled amount: Original disbursement deadline: 31 Dec 2013

Date signed: 22 Dec 2008

Supplementary financing: Original closing date: 31 Dec 2013

Date of entry into force : 23 Jul 2009 (Initial Grant); 22 Dec 2008 (Supplementary Grant)

Restructuring: Revised disbursement deadline: 31 Mar 2015 (Initial Grant); 31 Dec 2004(Supplementary Grant)

Date effective for 1st disbursement: 14 Oct 2009

Extensions (specify dates):Initial GrantExtension 1: 03 Oct 2013Extension 2: 24 Nov 2014Extension 3: 30 Apr 2015Supplementary GrantExtension 1: 03 Oct 2013Extension 2: 01 Apr 2015

Revised closing date: 30 Jun 2015

Date of actual 1st : 24 Apr 2010 (Initial Grant); 24 Jul 2012 (Supplementary Grant)b. Financing sources

Financing source/instrument (MUA)

Approved amount(MUA) :

Disbursed amount(MUA) :

Percentage disbursed(%):

Initial Grant 25,000,000.00 Supplementary Grant 2,925,285.00 Government: 2,840,000.00 Other (ex. Co-financiers): Beneficiaries

1,230,000.00

UN Habitat 1,230,000.00 TOTAL : 33,225,285.00 Co-financiers and other external partners:

Execution and implementation agencies:c. Responsible Bank staf

Position At approval At completion Regional Director Aloysius ORDU Gabriel NEGATU Country Manager Tonia KANDIERO

PCR EVALUATION NOTE FOR PUBLIC SECTOR OPERATIONS

Sector Director Andian Rakotobe Mohamed El AZIZI Sector Manager Sering JALLOW Osward M. CHANDA Task Manager Paul Dzimiri SABAS Thomas MARANDU Alternate Task Manager Christopher MUTASA PCR Team Leader Ephrem RUTABOBA PCR Team Members Ephrem RUTABOBA and

Alex Gomanid. Report data PCR Date : 08 Jan 2016 PCR Mission Date: From: 26 Dec 2015 To: 02 Jan 2016 PCR-EN Date: Evaluator/consultant : Peer Reviewer/Task Manager:

2. PROJECT DESCRIPTIONSummary from Appraisal Report including addendum/corrigendum or loan agreement, and taking into account any modification that occurred during the implementation phase.

a. Rationale and expected impacts:Provide a brief and precise description on the project/programme rationale (concerns/questions raised), expected impacts and the intended beneficiaries (directly or indirectly impacted by the project/programme). Highlight any change that occurred during the execution phase.

The appraisal report suggests that the rational for the project comes from The Zanzibar Poverty Reduction Plan (ZPRP 2007-2010), Cluster 2, Social Services and Well Being, which includes amongst others goals i) increased access to clean, safe and affordable water and ii) improved sanitation and sustainable environment.

Unfortunately the narratives in the appraisal report and the PCR on relevance do not provide an overview of the water and sanitation conditions and problems in the project areas, nor do they discuss the condition and reliability of the current systems, and the quality of services provided. Thelogical framework of the appraisal report indicates that in urban areas, 75% of the population has water supply, and in rural areas 59% have access to clean water. It also indicates that 25% of the population in 2008 practiced good hygiene/sanitation.

The provision of water supply of adequate quality and quantity in Zanzibar within 250 meters of consumer homes was expected to improve health condition of the people as well as free up time for women to engage in other productive activities and for girls to go to school. The provision of low cost on-site household sanitation and hygiene promotion would advance the sanitary environment for the rural and poor households. This was expected to reduce U5 child mortality rate from 101/1000 in 2008 to 50/1000 by 2020.

In addition, the project was projected to assist in reducing water borne diseases from 315/1000 to 164/1000 by 2015. This is in line with the existing successful Zanzibari efforts of reducing malaria. These interventions will also increase household income by reducing repeated cases of illness,

reducing the health bills, and alleviate the financial and social burden of population, especially for heads of households and women.

The project would ncrease access to sanitation and safe drinking water for about 276,000 students and reducing the ratio of students per sanitation facilities from 171 per facility to 40 will reduce girls’ absences from schools from 5 days to 2 days per months.

The expected impacts are taken from the appraisal report log-frame outcome. These are

1. Urban and rural water supply and sanitation facilities capable to satisfy demand2. Urban and rural water supply and sanitation facilities satisfactory operating3. Enhanced sanitation and hygiene practices4. Applied Integrated Water Resources Management

The beneficiaries are as anticipated in the logical framework are: Population of Chaka Chaka, Wete, Mkoani Towns, The 9 rural project areas Students of rural schools

4. Population of Zanzibar

b. Objectives/Expected Outcomes:Provide a clear and concise description of the project objectives, expected outcomes, and intended beneficiaries. In so doing, highlight any revision/amendment.

The project objective or purpose is taken from the logical framework: To improve water supply andsanitation services in rural and urban communities supported by integrated water resource management.

The four expected outcomes are listed in the appraisal report:1. Urban and rural water supply and sanitation facilities capable to satisfy demand2. Urban and rural water supply and sanitation facilities satisfactory operating3. Enhanced sanitation and hygiene practices4. Applied Integrated Water Resources Management

As noted above, the expected beneficiaries, from the logical framework are:

Population of Chaka Chaka, Wete, Mkoani Towns, The 9 rural project areas Students of rural schools

4. Population of Zanzibar

While some project activities were dropped because of lack of funds, the PCR does not mention any revisions or amendments to objectives or outcomes.

c. Outputs and intended beneficiaries:Provide a clear and concise description the expected outputs and intended beneficiaries. In so doing, highlight any revision/amendment.

The logical framework of the appraisal report lists the following outputs ZAWA’s staff trained Sustainable sanitation and hygiene promotion Rural water supply systems rehabilitated and newly constructed Community mobilized and continue to sustain the system School water supply and sanitation systems constructed and rehabilitated Urban water supply systems rehabilitated, expanded Integrated water resources management awareness Monitoring and evaluation system put in place linked with MKUZA national

monitoring system

The expected beneficiaries are listed as government institutions, local committees and CBOs, communities in the project areas, and the population of Zanzibar

While some project activities were dropped because of lack of funds, the PCR does not mention any revisions or amendments to outcomes and eintended beneficiaries.

d. Principal activities/Components:Provide a clear and concise description of the principal activities/components. In so doing, highlight any revision/amendment.

The principle activities of the project shown in the logical framework are: - Technical Assistance for ZAWA- Training and sensitization of staff- Design/implement Communication strategy/education program- Procure consultants/contractors for rural water supply and sanitation- Capacity building of Community to manage water and sanitation-Procure NGO’s for design/implement school water supply and sanitation program- Procure consultants /contractors for urban water supply and sanitation- Training / sensitize staff/ communities on IWRM- Procure consultants design / implement water resources monitoring system / data base

Budget limitations led to some sub-activities being dropped, Cost of service studies (these are not listed in the PCR) Training support for private sector in the sanitation activities One elevated tank whose construction was cancelled due to cost escalation, Re-establishment of the groundwater monitoring system through construction of

monitoring boreholes, was only partially completedAlso, the rainwater-harvesting component, which is under the IWRM (Integrated Water Resources Management) component, is yet to be implemented, because for many women it is a cultural taboo to use rainwater for their personal hygiene.

3. PROJECT PERFORMANCE ASSESSMENT

RELEVANCE

a. Relevance of the project development objective: Evaluation of the relevance ex-ante and ex-post (including during the implementation phase). The relevance of the project objective (duringthe evaluation ex-ante and the post-evaluation) in terms of alignment with country’s development priorities and strategies, the beneficiary needs (including any changes that may have occurred during the implementation), applicable Bank sector strategies, the Bank country/ regional strategy, and general strategic priorities of the Bank. This criterion equally assesses the extent to which the project’s development objective was clearly stated and focused on outcomes and the realism of the intended outcomes in the project setting.

The appraisal report noted that the project was consistent with the country’s overall development framework and long-term socio-economic development goals as laid out in: (i) Tanzania’s Vision 2025 and (ii) Zanzibar’s Vision 2020, which aim to reduce poverty and improve living standards through sustainable development, including investment in basic economic and social infrastructure, such as water and sanitation facilities.

The PCR noted that during the implementation period, the project’s purpose remained fully aligned with the water sector and GoZ development strategies. The beneficiary needs focused on improved health outcomes through behavioural change and through improved and sustained access to clean potable water and improved sanitation facilities for the safe disposal of human waste

Rating = 4.

b. Relevance of project design (from approval to completion):The evaluator should provide an assessment of the relevance of the project design regardless of the one provided in the PCR. The evaluator will also comment on the PCR conclusion for this section, and will provide an evaluation of the relevance of the project design. The latter assesses the soundness and the timing of eventual adjustments, or technical solutions to ensure the achievement of the intended results (outcomes and outputs), the adequacy of the risk assessment, environmental and social protection measures, as well as the implementation arrangements. For Programme Based Operations (PBO), an assessment will be made on the relevance of the prior actions, the policy dialogue and the extent to which the operation could have been more pro-poor in its design.

This section is based on the PCR and the appraisal report, since no supervision reports were available to the PCREN.

The design of the project was based on assessment studies and on consultations with all stakeholders including the project beneficiaries.

With regards to sanitation the design of the toilets took into consideration the accessibility of the persons with disabilities – both for access to the latrines and for the utilization of the toilet seats.

The section of the PCR on Environmental sustainability says that for rural areas there was not appropriate provision during the design phase on how sewerage waste disposal would be managed when the septic tanks are full.

Some cost estimates as per original design (at appraisal) were low. The Implementation Progress section of the PCR suggests that the original designs were out-dated. It also notes that this led to protracted negotiations aimed at cutting costs for the Urban Water Supply and Sanitation component because, there were insufficient funds to implement all the planned activities in the original designs.

Unfortunately, the PCR does not compare the components of the project before and after changes were

made to the project. It is therefore not clear in the PCR what changes were actually mad during implementation, their cost, and the potential impact.

The rainwater-harvesting component, which is under the IWRM (Integrated Water Resources Management) is yet to be implemented, because for many women it is a cultural taboo to use rainwater for their personal hygiene. This should have been picked up during community consultations.

Rating = 2

EFFECTIVENESS

c. Effectiveness in delivering outputs :Evaluation of the extent to which the project achieved its stated results (obtained from the logical framework) based on the last Implementation Progress and Results Report (IPR) and by considering accurate reporting of direct or indirect evidence on intended and unanticipated outputs. In the absence of sufficient data (as direct evidence), indirect evidence (such as project outcomes and other pertinent processes/elements of the causal chain) should be used particularly in the evaluation of the extent to which the project is expected to achieve its stated results/ objectives. The absence of sufficient data to assess the effectiveness should be indicated (and clearly detailed in the PCR quality evaluation section). The PCR score should equally be indicated in this section.

The information on outputs in this table is based on the appraisal report. Other information from the PCROutputs Progress towards target

ZAWA’s staff trained 127 staff trained for 100% of target: Comments

Sustainable sanitation and hygiene promotion

Target :Gender sensitive national strategy on S & H was adopted/ implemented by 2010. Done

600 teachers, health inspectors trained on gender sensitive S & H promotion . This target was achieved.

300 school health clubs effective, using gender sensitive educational aid by 2013 This Target was achieved

Training documents were developed and used.

Rural water supply systems rehabilitated and newly constructed

9 schemes planned and 9 completed

. Due to higher cost quotations for the works, the water supply network to some villages was limited. Training of Private Operators was not done due to fund limitations.

Community mobilized and continue to sustain the system This was not discussed by the PCR

School water supply and sanitation systems constructed and rehabilitated

This output was fully completed

Urban water supply systems rehabilitated, expanded

3 town schemes planned and 3 completed

Integrated water resources management awareness

Mostly carried out

An assessment of water resources for Zanzibar was done and an institution responsible for the development of water resources was set up. Sensitization campaigns on Rainwater Harvesting & Implementation of Water

Demand Management were carried out in pilot areas. However the effect of cultural taboos in using harvested water is not assessed quantitatively

Monitoring and evaluation system put in place linked with MKUZA national monitoring system

The PCR says that most outputs were completed except for One elevated tank whose construction was cancelled due to cost escalation, The training of private operators that was not done and The partial construction of the monitoring boreholes.

The works of the urban water schemes ended up being more costly than at appraisal thereby leading to the re-scoping of the some components of the project – mainly leaving out the soft components to cater for the physical works. However, the details of what was not done are not explained in the PCR.

The PCR also says that failure of the construction of one elevated water tank has left 6,000 people living in high elevation areas without adequate water. Also those living in areas that have inadequate distribution pipelines are not getting the full benefits of their water scheme. However, the reasons for this is not discussed in the PCR.

Rating = 3

d. Effectiveness in delivering outcomes:Evaluation of the extent to which the project achieved its intended set of outcomes (including for Program Based Operations (PBOs) where complementary measures are necessary for their implementation, namely public awareness, policy dialogue and institutional arrangements for instance). The evaluator should make an assessment based on the results of the last project Implementation Progress and Results (IPR). The evaluator shall indicate the degree to which project outcomes (intended and unanticipated) as well as reasons for any eventual gap werediscussed in the PCR.

Outcome from Logical Framework Target Comment1. Urban and rural water supply and sanitation facilities capable to satisfy demand

URBAN: The target was to increase the percent of population served from 75% to 9o%.Actual achievement at closing = 77%

RURAL: The target was to increase the percent of population served from 59 % to65 %

Actual achievement at closing was a decrease to 45%

The PCR used an incorrect target of 85% instead of 90%

Outcome 1 was missed by a considerable margin.

2. Urban and rural water supply and sanitation facilities satisfactory operating

The indicator for this outcome is an increase from 25% of population practicing good hygiene to 50% practicing good hygiene. Monitoring data indicates that the percentage of people after the project was only 16%

. Outcome 2 was missed, in fact the monitoring data indicated that it got worse

3. Enhanced sanitation and hygiene practices The targets were: Incidence of water born

diseases reduced from 315/1000 to The PCR does not report on actual reduction in disease, even though this is a

164/1000 by 2015.

Also 100 % rural schools with adequate water, sanitary& hand washing facilities, considering gender &disabled needs,

10 % reduction of girls absenteeism

key indicator for one of 4 outcomes.

The PCR says that all rural schools now have adequate water, sanitary& hand washing facilities, considering gender & disabled needs

The PCR has no data on the reduction in absenteeism by girls.

There is no data supporting two of the three indicators of whether Outcome 3 wasmet

4. Applied Integrated Water Resources Management

The Integrated Water Resources Management study was completed and according to the PCR was operational.

It should be noted that the PCR lists outcome indicators in place of the actual outcomes. There is a difference between the outcomes and the outcome indicators. The PCR should make an attempt to determine progress made in achieving the actual outcomes using the indicators as necessary, but recognizing the difference.

The PCR does not adequately explain why targets were not met. It could be that targets were unrealistic, or that they did not take sufficient account of population growth, so that even with considerable increase in facilities the percentage of the population served did not increase by very much. It could also be that actual household connections in urban areas were behind expectations.

In summary, Outcome 1 was missed by a considerable margin. Outcome 2 was missed. In fact the monitoring data indicated that it got worse There is no data supporting two of the three indicators of whether Outcome 3 was met Outcome 4 was met.

Rating = 2

e. Project development outcome: The ratings derived for outcomes and output are combined to assess the progress the project has made towards realizing its development objectives, based on the rating methodology recommended in the Staff Guidance Note on project completion reporting and rating (see IPR Guidance Note for further instruction on development objective rating).

Effectiveness in delivering outcomes = combined ratings of outcomes and outputs. The average is 2.5. However, since ratings are only whole numbers from 1 to 4, and since outcomes are more important than outputs, Rating = 2

f. Beneficiaries:Using evidence, the evaluator should provide an assessment of the relevance of the total number of beneficiaries by categories and disaggregated by sex.

The PCR shows the following table for beneficiaries.

Actual (A) Planned (B) Progress towardstarget

(% realized) (A/B)

% ofwomen

Category (e.g. farmers, students)

82658 71637 >153% 51 Rural students, teachers, farmers

70595 74747 94.4% 51% Urban dwellers, students, teachers, farmers

89196 Not available N/A-50.3%

Students, teachers in rural and urban areas benefiting from toilets

It is not clear where the planned target for rural beneficiaries was obtained. The logical framework in the appraisal report gives a progress target as “Rural water supply improved and increased for 95,214 people and coverage from 59 % in 2008 to 65 % in. This would mean that the target was only 87% achieved instead of the 153% shown in the PCR.

The planned figure for urban residents shown in the PCR is also different from the appraisal report, The appraisal report shows an expectation of population benefiting of 70, 832, rather than 74,747 shown in thePCR.

The PCR says that those living in areas that have inadequate distribution pipelines are not getting the full benefits of their water scheme. However, this is not discussed further in the PCR and not reflected in the discussion of beneficiaries.

Outcome 2 anticipated an increase from 25% of population practicing good hygiene to 50% practicing good hygiene. However, monitoring data indicates that the percentage of people after the project was actually lower at only 16%. This is not discussed in the above table.

In light of the discrepancies listed above, rating = 2

g. Unanticipated additional outcomes (positive or negative, not taken into consideration in the project logical framework) : This includes gender, climate change, as well as social and socio-economic- related issues. Provide an assessment of the extent to which intended or unanticipated additional and important outcomes have been taken into consideration by the PCR. The assessment should also look at the manner the PCR accounted for these outcomes.

Unanticipated consequences PCREN comment

Benefits to students attending kindergarten areone of the unforeseen benefits. The trauma of a child who imagines he/she is going to fall through the hole of a pit latrine is unimaginableand the project has brought a lot of relief to these youngsters. The toddlers also used to be assisted by elder students and this was a distraction to their study routines. This is also an unforeseen benefit to the elder students. Toilets with wash rooms facilitate increased attendance of girls – providing opportunities to keep clean and keep diginity and more so promote hygiene.

The benefits of improved convenience and time saved are not unanticipated. These benefits are discussed in the cost benefit analysis at appraisal.

Apart from long distances, a number of beneficiaries drew water from valleys – some ofwhich may have been contaminated from washed down human and other wastes during rainy times - and have to carry buckets through steep slopes. It was laborious and limited the levels of water consumed and personal hygiene was adversely affected.

It is not clear whether the PCR is discussing the conditions before the project started or after completion.

Also the PCR does not quantify the number of “beneficiaries>

Contrary to popular thinking that only women and children are involved in fetching water a number of communities in Pemba have men also involved in fetching water. Now with more time on their hands they are able to do other activities which bring in more income to their families. One of the teachers at a school WASH within actually said the men are now able to bring home a special cloth called ‘kanga’ on a more regular basis than before.

The PCR does not indicate the percentage of people who fetch water who are men.

Apart from using the time saved for income generation and leisure most of the communities expressed that they find time to take part in community activites and also participate more in their religious activities.

The PCR does not indicate how much more time is available for the new activity.

Most beneficiaries at the beginning of the project, previously benefited from free water. With the requirement for payments and with the increased tariff, they have moved from paying 2000 shillings per month, for those not metered to paying 4000 shillings per month, and for those having water meters, the payment depends on the consumption. There is some grumbling about the level of tariffs and those that are metred are against it – they would rather pay a fixed monthly fee. Sustainability of the water supply systems, let alone expansion of the systems to supply other communities will greatly depend on the appreciation of the service given and the ownership of the infrastructure by the communities 1st and foremost and also on the constant sensitization of these communities to appreciate the benefits of the water supplied and the need tosustain the supply.

Some grumbling about tariffs is normal. The PCR does not discuss the level of discontent.

EFFICIENCY

h. Timeliness:The timeliness of project implementation is based on a comparison between the planned and actual period of implementation from the date of effectiveness for first disbursement. For Programme Based Operations (PBOs), the timely release of the tranche(s) are assessed through this same criterion.

The following table is from the PCR

Planned project duration – years(A) (as per PAR)

Actual implementation time –years (B) (from effectiveness for 1st

disb.)

Ratio of planned and actualimplementation time (A/B)

Rating*

4.75 years from March 2009 toDecember 2013

6 years from 2009 to 2015 0.79 2

The PCREN agrees with the PCR.Rating = 2

i. Resource use efficiency:Provide and assessment of physical implementation (based on outputs delivered) against resources used (based on cumulative commitments) at completion for all contributors to the project (the Bank, Government, and others). This criterion would normally not applyto PBOs, as there is often no direct link between the outputs and the amount of contribution (in which case the rater would indicate N/A).

The following table is from the PCR

Median % physicalimplementation of RLF outputsfinanced by all financiers (A) (see

II.B.3)

Commitment rate (%) (B) (See table 1.C – Total commitment rate of all

financiers)

Ratio of the median percentagephysical implementation and

commitment rate (A/B)

Rating*

90% 100% 0.9 2

The PCREN agrees with the rating of the PCR

Rating = 2

j. Cost-benefit analysis:Provide an assessment of the timeliness of the development outputs, and the extent to which costs of the costs have been effective and have been provided in the most efficient manner. The PCR rating should be discussed. The evaluator should verify whether the benefits of the project (achieved or expected) exceed its actual costs. To achieve this, evidences will mainly be based on a comparison between Economic Rates of Return (ERR) calculated at appraisal, the mid-term review and completion. When commenting PCR ratings, the degree of utilization of valid sources for evidence justifying the rating assigned should be taken into consideration. The evaluator should ensure of thevalidity of assumptions and that the same model was used for the calculation of others ERRs. For PBOs for which this calculation model does not apply, an assessment could be done with regards to the contribution of policy reforms to economic growth. In the absence of sufficient evidence, an appropriate rating should be assigned.

The EIRR at appraisal is reported in the appraisal report as 16%; The EIRR reported at completion is in the PCR 14%.

The appraisal report says that details of the specific benefits and costs that were included in the cost benefit analysis were included in an annex. However, this annex was not included in the appraisal report provided to the PCREN team.

The PCR discusses in qualitative terms several benefits of the project, but does not describe anything about how benefits wee estimated and quantified for the cost benefit analysis. There is no attachment to the PCR with this information.

There is no information provided on which to judge the validity of the cost benefit analysis.

Rating =1

k. Implementation progress:The assessment of the Implementation Progress (IP) on the PCR is derived from the updated IPR and takes into account the all applicable IP criteria assessed under the three categories : i) Compliance with covenants (project covenants, environmental and social safeguards and audit compliance), ii) project systems and procedures (procurement, financial management and monitoring and evaluation), and iii) project execution and financing (disbursement, budget commitments, counterpart funding and co-financing).

The PCR says that On the whole implementation went smoothly after initial setbacks. It says that during implementation there were neither procurement nor disbursement issues after the initial teething problems.

Rating = 3SUSTAINABILITY

l. Financial sustainability:Provide an assessment of the extent to which funding mechanisms and modalities (eg. Tariffs, user fees, maintenance fees, budgetary

allocations, other stakeholder contributions, aid flows, etc.) have been put in place to ensure the continued flow of benefits after completion, with particular emphasis on financial sustainability. For PBOs, the assessment should focus on financial sustainability of reforms, as well as the Bank’s policy dialogue to promote financial sustainability of the reforms.

The narrative in The PCR on financial sustainability does not provide data on actual revenue collected. There is no comparison of revenue connected vs revenue targets that would maintain financial sustainability. The narrative discusses the fact that until just prior to the start of the project, the government policy was to provided water for free. This policy was changed before the project, Ancedotal evidence is given that some in the community are not happy about paying for water, and not happy about meters. The PCR says that any policy reversal on tariffs would likely be disastrous.

Because no evidence is provided to support financial sustainability, rating = 2.

m. Institutional sustainability and strengthening of capacities:Provide an assessment of the extent to which the project has contributed to the strengthening of institutional capacities – including for instance through the use of country systems – that will continue to facilitate the continued flow of benefits associated with the project. An appreciation should be made with regards to whether or not improved governance practices or improved skills, procedures, incentives, structures, or institutional mechanisms came into effect as a result of the operation. For PBOs, this should include an assessment on the contributions made to building the capacity to lead and manage the policy reform process; the extent to which the political economy of decision making was conducive to reform; the Government’s commitment to reform; and how the design reinforced national ownership.

The narrative on institutional sustainability discusses the technical assistance provided by the project and discusses the importance of this TA to institutional sustainability. However, the narrative does not discuss whether the institution is being satisfactorily managed at the time of project completion.

In the lessons learned section the PCR says that there is need to move from a project organizational structure to a regular water and sanitation authority.

The lessons learned section also says the construction of the toilets will require waste management and the capacity for such is not available. There is therefore need to either build private sector capacity for waste management or the authorities will need to provide that service.

The PCR also notes that a number of beneficiary communities either have no community committees or have committees, which are non-functional. In view of the widespread complaints about water tariffs ZAWA with the support of the RGoZ will need to strengthen partnership with the communities by strengthening the committees or where they are non-existent by ensuring that such community committees are set up and are functioning

Because there is no evidence of satisfactory institutional management, rating = 2

n. Ownership and sustainability of partnerships:Provide an assessment of whether the project has effectively involved relevant stakeholders, promoted a sense of ownership amongst the beneficiaries (both men and women) and put in place effective partnerships with relevant stakeholders (eg. local authorities, civil society organizations, private sector, donors) as required for the continued maintenance of the project outputs. For PBOs, the assessment should measure the extent to which the Government’s capacity to conduct consultations during policy dialogue and the extent to which the Bank supported the Government in deepening the consultation processes.

The PCR notes that during preparation/appraisal, field visits and interviews were conducted with key stakeholders, focusing on the project objectives, the expected impact and benefits. AfDB and UN Habitat jointly did the project preparation. Officials of the Government of Tanzania (GOT), Ministry of Finance (Mainland), Revolutionary Government of Zanzibar (RGOZ), Ministry of Water, Construction, Energy

and Land (MWCEL), Ministry of Finance & Economic Affairs (MOFEA) and several other relevant ministries took part in the preparation. The Technical Assistance Teams in Zanzibar from other Development organizations were i.e.; FINIDA, GTZ and JICA were also consulted. The cooperation between ZAWA and the AfDB is progressing well even after project completion. Communication betweenZAWA and the UN HABITAT also went / is going on well.

As noted above, the PCR notes that a number of beneficiary communities either have no community committees or have committees which are non-functional. RGoZ will need to strengthen partnership with the communities by strengthening the committees or where they are non-existent by ensuring that such community committees are set up and are functioning.

Rating = 3

o. Environmental and social sustainability:Provide an assessment of the objectivity of the PCR rating on the project’s implementation of environmental and social mitigation/enhancement measures with regard to the Environmental and Social Management Plan (ESMP), the capacity of country institutions and systems, as well as the availability of funding to ensure the environmental and social sustainability of the operation. This criterion would normally only apply to Environmental Category I and II projects.

The PCR notes that before the project most if not all beneficiaries were using either pit latrines or water based latrines with water drawn from wells which, considering the distance or time taken to draw water, posed a major challenge. The availability of water provided through the project has afforded schools and communities to construct modern toilets, many of which have septic tanks which has reduced ground water contamination by human waste. In addition the high student toilet ratio before the project increasedthe likelihood of human pollution as students in some schools resorted to using open defecation.

The PCr also notes that the project followed all the Environmental concerns that were raised at Appraisal and mitigated the situation before and during the implementation. In relation to sanitation however, as noted above, for rural areas there has not been appropriate provision on how sewerage waste disposal will be managed when the septic tanks are full.

Rating = 3

4. PERFORMANCE OF STAKEHOLDERS

a. Bank performance:(Preparation/approval, ensure of Quality at Entry (QAE) : quality of the supervision, completion) : Provide observations on the objectivity of the PCR ratings and feedback provided by the Borrower, and if necessary, re-assess the Bank’s performance throughout the project cycle (design, implementation, completion) by focusing on evidence from the PCR in relation to 7 criteria defined in the PCR Guidance Note.

The PCR narrative, says that many adjustments were made by the Bank to facilitate the implementation ofthe Project.

The cost estimates were off by a considerable amount, due to the use of out-dated designs. The PCR doesnot indicate the amount and does not provide a full picture of the project restructuring caused by shortage of project funds.

The PCR says that at Appraisal, the Bank should have consulted the communities much better – especiallyin relating to the Religious and Cultural concerns of Women not using Rainwater - this would have minimized the constraints met by the implementation of the Rainwater harvesting component. In Zanzibarit is taboo for females to use rainwater for bathing.

The PCR says that the Bank needs to engage more with Government in relation to Preconditions before the implementation of the project so that Government fulfills them early enough. For instance, there was a requirement to engage a qualified procurement officer. The project should have made a provision for engaging a qualified procurement officer even before project effectiveness (emphasis in text of PCR)

Rating = 3

b. Borrower performance: Provide observations on the objectivity of the PCR ratings, and if necessary, re-assess the Borrower’s performance throughout the project cycle (design, implementation, completion) by focusing on evidence from the PCR in relation to questions defined in the PCR Guidance Note.

The PCR discussion of borrower performance is very limited.

When the project was being designed, most staff of ZAWA were just being hired, and they were unable to participate significantly.

An important factor in project implementation has been the willingness of the borrower to allow ZAWA tocharge for water. The effort to educate the population of the necessity to charge for water is important and needs to continue.

Rating = 3

c. Performance of other stakeholders: Provide observations on the objectivity of the PCR ratings, and if necessary, re-assess the other shareholders’ performance throughout the project cycle (design, implementation, completion) by focusing on evidence from the PCR in relation to relevant questions specific to each stakeholder (co-financiers, NGO, contractors and service providers).

Two of the three contractors performed well. The other contractor had problems with delays as well as quality.

The consultants especially the Integrated Water Resources expert performed very well in community sensitization on rainwater harvesting and the implementation of Water Demand management in pilot areas in Pemba, Nungwi and Unguja.

Due to good communications UN Habitat enabled Zawa disbursements went smoothly and work on the report for Water demand management was accomplished in good time. Everything that was planned including the signing of the Agreement for rainwater harvesting was done in time.

Rating = 3

5. SUMMARY OF OVERALL PROJECT PERFORMANCE

a. Overall assessment: Provide a summary of the project/programme’s overall performance based on the PCR 4 key components (Relevance, Effectiveness, Efficiency and Sustainability). Any difference with the PCR and the reasons that have resulted in them should be mentioned. For cases with insufficient evidence (from the PCR and other documents) available, the evaluator should assign a partly satisfactory rating (to be revised) until a post project performance evaluation (e.g. PPER, PER or PRA) is complete.

The project provided significant benefits to the population, but was handicapped by out-dated cost estimates,

b. Design, implementation and utilization of the M&E (appreciation of the evaluator):Provide an assessment of planned and actual cost of the design, implementation and utilization of the M&E system. Design : To which extent the project M&E system was explicit, adequate and realistic to generate and analyse relevant data ; Implementation : To which extentrelevant data was collected – Elements of M&E implementation and effectiveness in the PCR ; Utilization : degree of utilization of data generated for decision-making and resource allocation – elements of M&E utilization in the PCR.

The appraisal report says that an agreement was reached with Google through UN Habitat to build a monitoring database, that would involve collection of geo-referenced ground-level data; a system of benchmarking service providers to enhance not only coverage efficiency and quality of service, but also accountability to customers. The appraisal report says that the “Google” M&E Database should be operating by 06/2009. However this M&E system is not mentioned in the PCR.

The appraisal report also says that “Information from various sources will also be routinely gathered, as part of the monitoring and evaluation framework for the project. The experience and lessons learned will be duly documented and the information will be shared among stakeholders through joint technical and sector reviews and other appropriate forums. The Bank through its supervision missions, will share this knowledge and experiences with other development partners.”

However, the PCR does not mention the M&E framework or its design, adequacy or performance at all.

Rating = 16. EVALUATION OF KEY LESSONS LEARNED AND RECOMMENDATIONS

a. Lessons learned: Provide a brief description of any agreement/disagreement with all or part of the lessons learned from the PCR after analysis of the project performance with regards to each of the key components of the evaluation (Relevance, Effectiveness, Efficiency, and Sustainability). List the PCR main new and/or reformulated pertinent (and generic) lessons learned for each of these components here. It is recommended that no more than five lessons learned are discussed. Key questions and targeted audience must also be specified for each lesson learned.

Lessons learned as reported in PCR Comments of PCRENThe procurement process should be enhanced and followed up. Bank procedures need to be better promoted and only when National Procedures have been harmonized with Bank Procedures should they be used. Constant Trainings of Bank procedures to staff of the PIU

This is a consistent concern expressed in PCRs

Vandalism and resistance to pay for water will continue as the population was conditioned on free water services. It is important for sustainability and reduction of defaults on payment to intensify on sensitization and the strengthening of WASH Community Committees and where there is none establish them – should be promoted. Ownership of the Project by the Communities needs to be enhanced.

This is project specific, but agreed

Staff stability at ZAWA is crucial for ensuring that capacityis in place for successor projects for purposes of smooth implementation of projects.

This is project specific, but agreed

It is clear that the sequencing of conditions precedent had an impact on speedy implementation of the project. The Bank can change its rules and use some of the Trust funds to pre-finance the set-up of ZAWA and the recruitment of

Agreed that this would be useful in many projects

key members of staff with the agreement of financiers of the Trust Funds.The Bank should always remain flexible when the need for project rescoping when resources are inadequate is paramount. This obviates project implementation delays

This statement is too general. The PCR should be more precise in defining what it means by flexibility.

b. Recommendations: Provide a brief description of any agreement/ disagreement with all or part of the recommendations from the PCR. List the PCR main new and/or reformulated recommendations (requiring more actions by the Borrower and/or the Bank) here.

Recommendations as reported in PCR Comments of PCRENThe country just came out of a system that provided free water services for household use. The change to full cost recovery for water services is likely to be met by resistance and to have the vigorous complaints that the mission noted. As ZAWA moves towards full cost recovery and ownership of the systems, it should enhance introduction and installation of new water meters and it should step up the sensitization campaign among the communites to ensure full payment compliance. Otherwise ZAWA’s sustainability would be at stake.

Agree with this project specific recommendation

Much of the staffing of ZAWA and its organizational structure is a result of the project, which raises concerns about its stability as an organization. Hence the new HR development policy and strategy needs to be earnestly implemented and entrenched within ZAWA for future organizational stability which in turn will ensure that there is smooth implementation of the current and of new projects as the capacity for project implementation will have been entrenched.

Agree with this project specific recommendation, but much more will be needed in addition to an HR strategy

Strengthening WASH Committees will be very important for purposes of compliance by communities with the tariff policy and minimization of vandalism. Water conservation advocacy can effectively be carried out through such committees.


The construction of the toilets will require waste management and the capacity for such is not available. There is therefore need to either build either private sector capacity for waste management or the authorities will need to provide that service.


The stability of ZAWA is crucial to having capacity in placefor successor projects for purposes of smooth implementation of projects.


7. COMMENTS ON PCR QUALITY AND TIMELINESSThe overall PCR rating is based on all or part of the criteria presented in the annexe and other: The quality of the PCR is rated as highly satisfactory (4), satisfactory (3), unsatisfactory (2), and highly unsatisfactory (1). The timeliness of the PCR is rated as on time (4) or late (1). The participation of the Borrower, co-financier, and the bank’s external office(s) are rated as follows: Very Good(4), Good (3), Fair (2), Poor (1).

1. Quality of the PCR: Rated 2 Because of deficiencies in discussing the target numbers of

beneficiaries between PCR and appraisal report, lack of explanation of computation of Cost benefit analysis, and the inconsistency between PCR and appraisal report in the lists of outcomes and outputs,

2. Timeliness of PCR: rated 4

3. The participation of the Borrower, co-financier, and the Bank’s external office: Unable to rate. There is no information in the PCR about the involvement of the Borrower, co-financier, and the bank’s external office.

8. SUMMARY OF THE EVALUATIONThis is a summary of both the PCR and IDEV ratings with justification for deviations/comments. Appropriate section of the PCR Evaluation should be indicated in the last column in order to avoid detailed comments. The evaluator must provide a reasonable explanation for each criterion the PCR rating is not validated by IDEV. Consequently, the overall rating of the project could be “equally satisfactory”.

Criteria PCR PCREN Reason for disagreement/ Comments

RELEVANCERelevance of project development objective 4 4

Relevance of project design 3 2 Significant deficiencies in design

EFFECTIVENESSDevelopment objective (DO) 3 2.5

EFFICIENCYTimeliness 2 2

Resource use efficiency 2 2

Cost-benefit analysis 2 1 There is no information provided on which to judge the validity of the cost benefit analysis.

Implementation progress (IP) 3 3

SUSTAINABILITYFinancial sustainability 3 2 There is no evidence of financial

sustainabilityInstitutional sustainability and strengthening of capacities

3 2 There is no evidence of institutional sustainability

Environmental and social sustainability 3 3

OVERALL PROJECT COMPLETION RATINGBank performance: 3 3

Borrower performance: 3 3

Performance of other shareholders: 3 3

Overall PCR quality: 2.94

9. PRIORITY FOR FUTURE EVALUATIVE WORK: PROJECT FOR PERFORMANCE EVALUTION REPORT, IMPACT EVALUTION, COUNTRY/SECTOR REVIEWS OR THEMATIC EVALUATION STUDIES:

- Project is part of a series and suitable for cluster evaluation

- Project is a success story

- High priority for impact evaluation

- Performance evaluation is required to sector/country review

- High priority for thematic or special evaluation studies (Country)

- PPER is required because of incomplete validation rating

Major areas of focus for future evaluation work:

a) Performance evaluation is required for sector/ country review

b) Cluster evaluation (institutional support)

c) Sector evaluation (budgetary support or public finance management reforms)

Follow up action by IDEV: Identify same cluster or sector operations; organize appropriate work or consultation mission to facilitate a), b) and/or c).

Division Manager clearance Director signing off

Data source for validation: Task Manager/ Responsible bank staff interviewed/contacted (in person, by telephone or

email) Documents/ Database reports

Attachment:

PCR evaluation note validation sheet of performance ratings

List of references

Appendice 1

PROJECT COMPLETION REPORT EVALUATION NOTE Validation of PCR performance ratings

PCR rating scale:

Score Description4 Very Good – Fully achieved with no shortcomings3 Good – Mostly achieved despite a few shortcomings2 Fair – Partially achieved. Shortcomings and achievements are roughly balanced1 Poor – very limited achievement with extensive shortcomings

UTS Unable to score/rateNA Non Applicable

Criteria Sub-criteriaPCRworkscore

IDEVreview

Reasons for deviation/comments

RELEVANCE Relevance of the projectdevelopment objective (DO) during implementation

4 4

Relevance of project design (from approval tocompletion)

3 2Significant deficiencies in design

OVERALL RELEVANCE SCORE

EFFECTIVENESS* Effectiveness in delivering outcomes

Outcome1 UTS UTS

Outcome2 UTS UTS

Effectiveness in delivering output

Output1UTS UTS

Output2UTS UTS

Development objective (DO)

Development objective rating

3 2.5

Beneficiaries

Beneficiary1UTS UTS

Beneficiary2 UTS UTS

Criteria Sub-criteriaPCRworkscore

IDEVreview


Unanticipated outcomes (positive or negative not considered in the project logical framework) and their level of impact on the project (high, moderate, low)Institutional development

Gender

Environment & climate change

Poverty reduction

Private sector development

Regional integration

Other (specify)

EFFECTIVENESS OVERALL SCORE

EFFICIENCY Timeliness (based on theinitial closing date)

2 2

Resource used efficiency

2 2

Cost-benefit analysis 3 1 There is no information provided on which to judge the validity of the cost benefit analysis.

Implementation progress (from the IPR)

3 3

Other (specify)

OVERALL EFFICIENCY SCORE

SUSTAINABILITYFinancial sustainability

3 2 No evidence of financial sustainability

Institutional sustainability and strengthening of capacities

3 2 No evidence of institutional sustainability

Ownership and sustainability of partnerships

3 3

Environmental and social sustainability

3 3

*The rating of the effectiveness component is obtained from the development objective (DO) rating in the latest IPR of the project (see Guidance Note on the IPR). The ratings for outputs and outcomes are determined based on the project’s progress towards realizing its targets, and the overall development objective of the project (DO) is obtained by combining the ratings obtained for outputs and outcomes following the method defined in the IPR Guidance Note. The following method is applied: Highly satisfactory (4), Satisfactory (3), Unsatisfactory (2) and Highly unsatisfactory (1).

Criteria Sub-criteriaPCRWorkscore

IDEVreview


BANK PERFORMANCE

Proactive identification and resolution of problems at differentstage of the project cycle

UTS UTS

Use of previous lessons learned from previous operations during design and implementation

UTS UTS

Promotion of stakeholder participation to strengthen ownership

UTS UTS

Enforcement of safeguard and fiduciary requirements

UTS UTS

Design and implementation of Monitoring & Evaluation system

UTS UTS

Quality of Bank supervision (mixof skills in supervisory teams, etc)

UTS UTS

Timeliness of responses to requests

UTS UTS

OVERALL BANK PERFORMANCE SCORE

BORROWER PERFORMANCE

Quality of preparation and implementation

UTS UTS

Compliance with covenants, agreements and safeguards

UTS UTS

Provision of timely counterpart funding

UTS UTS

Responsiveness to supervision recommendations

UTS UTS

Measures taken to establish basis for project sustainability

UTS UTS

Timeliness of preparing requests UTS UTS

OVERALL BORROWER PERFORMANCE SCORE

PERFORMANCE OF OTHER STAKEHOLDERS

Timeliness of disbursements by co-financiers

UTS UTS

Functioning of collaborative agreements

UTS UTS

Quality of policy dialogue with co-financiers (for PBOs only)

UTS UTS

Quality of work by service providers

UTS UTS

Responsiveness to client demands UTS UTS

OVERALL PERFORMANCE OF OTHER STAKEHOLDERSThe overall rating is given: Very Good, Good, Fair and Poor.

(i) Very Good (HS) : 4(ii) Good (H) : 3(iii) Fair (US) : 2(iv) Poor (HUS): 1

DESIGN, IMPLEMENTAION AND UTILIZATION OF MONITIRING ANDEVALUATION (M&E)

Criteria Sub-criteriaIDEVScore

Comments

M&E DESIGN M&E system is in place, clear, appropriate and realistic

1 No M&E data provided in PCR

Monitoring indicators and monitoring plan were duly approved


Existence of disaggregated gender indicator


Baseline data were available or collected during the design


Other, specify

OVERALL M&E DESIGN SCOREM&E IMPLEMENTA-TION

The M&E function is adequately equipped and staffed


OVERALL M&E IMPLEMENTATION SCOREM&E UTILIZATION

The borrower used the tracking information for decision


OVERALL M&E UTILIZATION SCOREOVERALL M&E PERFORMANCE SCORE


PCR QUALITY EVALUATION

Criteria PCR-EVN Comments

(1-4)

QUALITY OF PCR

1. Extent of quality and completeness of the PCR evidence and analysis to substantiate the ratings of the various sections

2

2. Extent of objectivity of PCR assessment score 3

3. Extent of internal consistency of PCR assessment ratings; inaccuracies; inconsistencies; (in various sections; between text and ratings; consistency of overall rating with individual component ratings)

2

4. Extent of identification and assessment of key factors (internal and exogenous) and unintended effects (positive or negative) affecting design and implementation

3

5. Adequacy of treatment of safeguards, fiduciary issues, and alignment and harmonization

3

6. Extent of soundness of data generating and analysis process (including rates of returns) in support of PCR assessment

2

7. Overall adequacy of the accessible evidence (fromPCR including annexure and other data provided)

2

8. Extent to which lessons learned (and recommendations) are clear and based on the PCR assessment (evidence & analysis)

3

9. Extent of overall clarity and completeness of the PCR

3

Other (specify)

PCR QUALITY SCORE 2

PCR compliance with guidelines (PCR/OM ; IDEV)

1. PCR Timeliness (On time = 4; Late= 1) 4

2. Extent of participation of borrower, Co-financiers & field offices in PCR preparation

UTS

3. Other aspect(s) (specify)

PCR COMPLIANCE SCORE 4

*** rated as Very Good (4), or Good (3), or Fair (2), or Poor (1)

References