factors affecting project performance in the agriculture and social sectors: a review of...

8/3/2019 Factors Affecting Project Performance in the Agriculture and Social Sectors: A Review of Postevaluation Reports Bet

1/28

ASIAN DEVELOPMENT BANK SST: 98010

SPECIAL EVALUATION STUDY OF

FACTORS AFFECTING PROJECT PERFORMANCE

IN THE AGRICULTURE AND SOCIAL SECTORS:

A REVIEW OF POSTEVALUATION REPORTS

BETWEEN 1991 AND 1997

December 1998


2/28


3/28

CONTENTS

EXECUTIVE SUMMARY ii

I. INTRODUCTION 1

A. Background and Rationale 1B. Objectives and Scope of Activities 6C. Limitations of the Study 7

D. Finalizing the Report 8

II. MAJOR FINDINGS OF THE STUDY 8

A. Performance Measure 8B. Typically Monitored Indicators of Project Performance 10

1. Delays 102. Cost Underrun/Overrun 103. Bank Supervision 11

C. Indicators of Project Performance Specific to AGSO Sectors 12

1. The Banks Performance 122. Performance of the Consultants During Implementation 133. Performance of the Executing Agency 154. Performance of the Borrower 17

5. External Factors 19

III. CONCLUSIONS AND IMPLICATIONS 20


4/28

EXECUTIVE SUMMARY

Historical trends of overall project performance, by year of approval and year ofpostevaluation, revealed that the performance of projects in the agriculture and socialinfrastructure (AGSO) sectors was consistently lower than that in the energy and infrastructure(ENIN) sectors (Figures 1-4). The study attempts to clarify the major causes of project successand failure that are specific to AGSO and ENIN sector groups, and draw some implications and

suggestions useful in addressing issues pertaining to the performance of projects in the AGSOsectors. This is a pilot attempt to use statistical procedures in assessing the performance ofBank-assisted projects.

The study is based mainly on information readily available in 141 projectperformance audit reports on projects approved between 1981 and 1992 and postevaluatedbetween 1991 and 1997. Projects approved after 1992, when the Bank adopted the secondMedium-Term Strategic Framework, are not yet ready for postevaluation.

The study involved statistical analysis using regression models. Two sets ofmultivariate regression models were built. The models were based on a conceptual frameworkcomprising the major participants in a project. The first set of models focused on identifying thedistinction between the typical factors that affect project performance in AGSO and ENIN sectorgroups. The second set assessed the significance of typical as well as other factors specific toAGSO sectors. The models used two project performance measures: the reestimated economicinternal rates of return (REIRR) at postevaluation, and the deviation of REIRR from the EIRRestimate at appraisal, measured by the percent gap in EIRR.

The study was limited by data and information constraints. In the regressionmodels, only EIRRs could be easily incorporated as dependent variables; consequently, themodels included only 34 projects from AGSO sectors, of which only 3 were social infrastructure.Information constraints made it difficult to analyze the impact of such variables as participatoryapproach and environmentall of which are highly relevant under the Banks current policiesas projects referring to them were limited.

Caution was exercised in applying the results of the statistical tests that confirmand quantify the relationships between the measures of project performance and the causativefactors. The tests were conducted for only a limited number of cases.

The findings summarized here were concurred by operations departments(especially those concerned with AGSO sectors) based on their experience.

The study found that managing time and cost overruns alone was insufficient toimprove the performance of projects in AGSO sectors. Such factors significantly affected projectperformance only in ENIN sectors. The study indicated that, in the case of projects in AGSOsectors, attention should be paid as well to other factorsboth project specific and externalwhose impact on project performance is large. This suggests a weakness in the Bankstraditional project administration mechanism where disproportionate attention had been directedto managing time and cost. The study underscores the importance of the newly introducedproject performance reportwhich comprehensively reviews the various aspects of a project


5/28

iii

compared with the traditional Project Administration Committee notes, especially for socialinfrastructure projects.

The models identified the following significant determinants of projectperformance in AGSO sectors (Figures 5-6).

At the country level, the macroeconomic environment is the dominant factoraffecting the REIRR. The borrowers performance is also a critical determinant of projectsuccess. Good borrower performance implies a sector policy environment favorable to projectoperation, governments commitment to the project, and provision of domestic funds.

Concerning borrower performance, project success in AGSO sectors depends onthe Executing Agencys capability to supervise project implementation and properly utilizefacilities.

Statistical results show that adequate consulting services during implementationalso yield significant rewards, by sharply reducing the gap between the appraisal andpostevaluation EIRRs.

The Banks foremost contribution can come at the project preparatory technicalassistance (PPTA) stage where, in particular, greater realism is expected. The study revealedthat the PPTA was weak for projects in AGSO sectors (Figures 7-8).

The implications of the findings for Bank policy and future operations arehighlighted in the following paragraphs.

At the country level, the Bank should more realistically assess themacroeconomic and policy environment, and choose a set of interventions that best fit, giventhe existing constraints.

At the project level, in countries where the policy framework is distorted and theimplementation environment is weak, necessary advisory and capacity building TA servicesshould in many cases precede lending activities. These services should not be attached toloans.

At the fact-finding stage of the PPTA, and subsequently at project appraisal, theBank should carefully assess the capacity of the Executing Agency for project implementation,utilization of facilities, and commitment to maintenance (adequacy of staff skills and funds). TheBanks continuing focus on encouraging commitment to ownership of project goals byincreasing Executing Agency involvement in design and implementation is well underscored.

Important lessons were learned from the study and valuable comments and

suggestions received from operations departments are as follows:

Timely and relevant cognizance of social issues as critical determinants ofproject success should be analyzed. A more detailed treatment of exogenous and endogenousvariables should be carried out so as to launch remedial measures for improving projectperformance. A closer review is needed to ascertain why most successful AGSO projects were


6/28

iv

prepared using a PPTA, while many AGSO projects other than those rated as generallysuccessful were only partly successful or unsuccessful despite having had a PPTA.


7/28

I. INTRODUCTION

A. Background and Rationale

1. In 1992, the Bank set five strategic development objectives.1 The targeted projectmix for public sector projects was set at 50:50: at least 50 percent of projects should have social

or environmental objectives either as a primary or secondary objective. The Bank's Medium-Term Strategic Framework (1995-1998) confirmed the continued relevance of these objectivesand the project mix, and redefined the mission of the Bank from financing development projectsto providing an integrated package of development services.2

2. In line with such a strategic operational thrust, 194 (or 58 percent) of 333 projectsapproved during the past five years had social or environmental improvement as a primary orsecondary objective. Thirty-two percent of these are agricultural projects, while 51 percent are insocial infrastructure, together reaching 83 percent. In addition, irrespective of their having socialand environmental primary or secondary objectives, projects in the agriculture and socialinfrastructure (AGSO) sectors are meant to bring about more extensive indirect social andenvironmental advancement than projects in other sectors. Therefore, achievement of theBanks strategic development objectives heavily relies on the performance of Bank projects inAGSO sectors.

3. However, a cursory review of the postevaluated projects presents someworrisome features concerning the past performance of projects in AGSO sectors (Box 1). Of

the 257 postevaluated projects3 in these sectors that were approved between 1968 and 1992,only 44 percent were rated as generally successful, while 81 percent of projects in the energyand infrastructure (ENIN) sectors were rated as generally successful. The rating of unsuccessfulwas given to 15 percent of projects in AGSO sectors and to only 4 percent in ENIN sectors(Table 1).

Box 1

The Postevaluation Offices (PEO's) evaluation reports and other studies assess the causes ofproject success and failure, and extract lessons learned as feedback to ongoing Bank operations.Annual reviews of postevaluation findings reveal a number of causes of project failure, some of whichare recurrent. Furthermore, clear differences in the performance of projects in AGSO and ENINsectors suggest that sectoral elements affect performance.

Statistical analysis of a sample of evaluated projects could produce information useful inidentifying possible generic and structural constraints inherent in socioeconomic conditions ofdeveloping member countries (DMCs) and in Bank operations. The findings from this analysis couldenable the Bank to address such constraints through a sector or strategic approach.

1The objectives are (i) promoting economic growth, (ii) reducing poverty, (iii) supporting human development(including population planning), (iv) improving the status of women, and (v) protecting the environment.

2These include support for policy reforms, capacity building, and regional cooperation.

3257 projects (or 50 percent) belong to AGSO sectors, while 185 projects (or 36 percent) fall in ENIN sectors,with the remaining 73 projects belonging to other sectors.


8/28

2


9/28

3

Table 1: Assessment of Overall Performance of Projects

Sectors Ratinga

All ProjectsProjects Approved Between

1981 and 1992 andPostevaluated Between 1991

and 1997Number Percent

bNumber Percent

b

AGSO All 257 50 88 62

Generally Successful 112 44 34 39Partly Successful 106 41 43 49Unsuccessful 38 15 10 11

ENIN All 185 36 39 28Generally Successful 150 81 32 82Partly Successful 27 15 7 18Unsuccessful 8 4 0 0

Others All 73 14 14 10

Total 515 100 141 100

aNo rating was provided for one project in AGSO sector.

bPercentages for All ratings are calculated with respect to the total number of projects, while percentages for individualrating categories (generally successful, partly successful, and unsuccessful) are calculated with respect to the number ofprojects under All ratings in each sector group.

4. A closer look at the projects approved between 1981 and 1992 andpostevaluated between 1991 and 1997 reveals that the gap between the two groups of sectorsis considerable: only 39 percent of the projects in AGSO sectors were rated as generallysuccessful as against 82 percent for projects in ENIN sectors.

5. Tables 2 and 3 contrast the historical trends of overall project performance, byyear of postevaluation and year of approval, respectively. Projects in AGSO sectors consistentlyperformed less satisfactorily than those in ENIN sectors in each evaluation year (Table 2). Table3 confirms the same for each approval year.1 While all of these projects, except one, wereprocessed prior to 1992 and before the Bank adopted Medium-Term Strategic Framework and astrong emphasis on social and environmental objectives, the findings provide a critical hint onthe future achievement of the Banks strategic development objectives as well as the overallquality of the Banks project portfolioparticularly if sufficient measures have not been taken toredress the situation. It is generally acknowledged that the developmental challenge is higher inAGSO projects due to their greater complexity, weaker institutions, and other factors includingthe large number of direct beneficiaries.

1Analysis of the worsening trend in the performance of AGSO projects in relation to ENIN projects is beyondthe scope of the study, but could be taken up in a future study.


10/28

4

Table 2: Overall Performance of Projects, by Evaluation Year

Projects (Number) by Evaluation Year

Rating Sectors 1991 1992 1993 1994 1995 1996 1997

GenerallySuccessful

AGSO 6 3 8 6 3 4 4

ENIN 4 5 4 5 4 4 6

Partly Successful AGSO 5 7 5 6 7 7 6

ENIN 2 2 0 1 1 1 0

Unsuccessfula

AGSO 1 1 2 1 0 1 4

Percentage of projects that were rated as Generally Successfulb

All sectors 52 44 57 60 47 45 45

AGSO 50 27 50 46 30 33 29

ENIN 67 71 100 83 80 80 100

aNo project in ENIN sectors was rated unsuccessful.

bPercentages for All sectors are with respect to the total number of projects in all sectors.Percentages for AGSO and ENIN sector groups are with respect to the total number of projects in Allratings (including No rating).

Table 3: Overall Performance of Projects, by Year of Approval

Projects (Number) by Year of Approval

Rating Sectors 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992

GS AGSO 6 7 6 4 6 2 0 2 0 1 0 0

ENIN 1 2 5 6 3 1 6 6 0 0 1 1

PS AGSO 9 7 7 3 7 4 1 2 2 1 0 0

ENIN 2 0 1 0 0 1 1 1 0 1 0 0

US AGSO 2 1 2 2 0 0 1 1 1 0 0 0

Percentage of projects that were rated as GSa

Allsectors 35 47 52 67 50 50 60 62 29 25 100 100AGSO 35 47 40 44 46 33 0 40 na na na na

ENIN 33 100 83 100 100 50 86 86 na na na na

na = not available, GS = generally successful, PS = partly successful, US = unsuccessful,AGSO = agriculture and social infrastructure, ENIN = energy and infrastructure.


11/28

5

aPercentages for All sectors are with respect to the total number of projects in all sectors. Percentages forAGSO and ENIN sector groups are with respect to the total number of projects in All ratings (including Norating).


12/28

6

B. Objectives and Scope of Activities

6. The ultimate objective of the study is to improve understanding of factorsaffecting the performance of Bank-financed projects.

7. In line with this objective, the study attempts to answer the following questions:

(i) What are the major factors systematically affecting project performance?

(ii) Are there such factors specific to AGSO and ENIN sector groups?

(iii) How significantly does each factor affect the performance/rating of projects inAGSO sectors?

(iv) What are the implications of the study findings for the Bank and the DMCsespecially in terms of Bank policies and future operations?

8. The scope included

(i) review of project performance audit reports (PPARs) on projects approved from1981 to 1992 and postevaluated between 1991 and 1997 (Appendix 1, Table 1),other relevant postevaluation reports, and other reports of the Bank and theWorld Bank (WB) (Appendix 2); and creation of a database of the information,

including relevant additional information from the Postevaluation InformationSystem (PEIS) (Appendix 3);

(ii) statistical analysis (Box 2) of typical factors relating to cost, design,implementation, operation, borrower performance, etc. as judged in the PPARs,to identify sector elements of the factors specific to AGSO and ENIN sectors;and

(iii) implications regarding future actions to address strategically the generic andstructural constraints on projects in the AGSO sectors.


13/28

7

Box 2: Statistical Analysis

After creating a database on all projects, two types of analysis were performed. First,descriptive statisticsfrequencies, percentages, and histogramswere prepared to illustrate thedistribution of projects against each indicator. Second, multivariate regression analysis using theordinary least square method was performed to assess relationships between each indicator and theperformance measures.

Two sets of multivariate regression models were built. The models were based on a conceptualframework comprising the major participants in a project. Factors relating to participation bybeneficiaries and the environment could not be included (para. 9). The first set of models focused onidentifying the distinction between the typical factors that affect project performance in AGSO and ENINsectors. The second set assessed the significance of typical as well as other factors specific to AGSOsectors.

The first set of models included explanatory variables relating to the

?? Banks performance at project preparation: presence of project preparatory technicalassistance (PPTA) for design;

?? Banks performance during implementation: supervision in person-days;?? executing agencys (EAs) performance during implementation: time and cost

overruns/underruns; and

?? borrowers performance: long-term country economic performance and mean growthrate of gross domestic product.

The second set of models included explanatory variables relating to the

?? Banks performance at project preparation: rating on quality of design;

?? Banks performance during implementation: rating on adequacy of supervision;?? performance of consultants during implementation: rating on adequacy of consulting

services;

?? EAs performance during implementation: rating on adequacy of EA capability tosupervise implementation;

?? EAs performance during operation: rating on adequacy of utilization of facilities, andrating on adequacy of repair and maintenance;

?? borrowers performance: composite indicator representing a satisfactory policyenvironment for the project, commitment to the project, and provision of domestic funds;and

?? external factors: composite indicator on the presence of recession, political unrest, andinclement weather during project implementation, and qualitative index on change inprices between appraisal and postevaluation.

The equations and the results of the regression analyses are in Appendix 3.

C. Limitations of the Study

9. The study is a pilot attempt to employ statistical methods in analyzing theperformance of projects. The analysis relies on information from PPARs. In the models, onlyeconomic internal rates of return (EIRRs) could be easily incorporated as dependent variables.


14/28

8

Consequently, although the study included 141 projects88 projects in AGSO sectors and 39in ENIN sectorsthe models included only 34 cases from AGSO sectors and 31 cases fromENIN sectors. Of the 34 cases from AGSO sectors, only 3 social infrastructure projects werefeatured. However, the database on the typically monitored indicators included all projects inboth AGSO and ENIN sector groups, while that on other indicators specific to AGSO sectorscovered more than 75 percent of the projects in these sectors.1 Information constraints made itdifficult to analyze the impact of such variables as the participatory approach and theenvironment, as the number of PPARs referring to them is still limited. Therefore, the Missionexercised caution in interpreting the results of the regression models and drawing conclusions.In the early stages, the Economic Analysis and Research Division (EDAN) provided commentsand expertise on the technical aspects of the regression analysis used in the study, whichensured that the analysis was correctly conducted and the inferences drawn followedaccordingly. In summary, the findings of the study as a pilot attempt to statistically analyze theperformance of Bank-assisted projects produced meaningful (Appendix 3, Tables 6 to 9) andunderstandable results.

D. Finalizing the Report

10. As this study is a pilot attempt, the Mission organized an inception workshopwhere the methodology and approach for the study were clarified.

11. The final report duly reflects comments received at another workshop, and thosefrom operations departments/offices on the interdepartmental draft.

II. MAJOR FINDINGS OF THE STUDY

A. Performance Measure

12. The regression models used two dependent variables:

(i) the REIRR as reported in the PPAR (in cases when there was a range of valuesfor REIRR, the lowest value was used); and

(ii) the change in EIRRs between appraisal and postevaluation, represented by thepercent gap in EIRRs (PGIR).2

1The data constraints precluded logit-modeling using ordinal dependent variables such as performance ratingand achievement of objectives. Limitations of time and resources also disallowed quantile-regression foranalyzing distinct characteristics of outstanding and poor performing projects.

2PGIR = (EIRR at postevaluation EIRR at appraisal) x 100.

EIRR at appraisal


15/28

9

13. EIRRs were estimated in 65 cases (46 percent of all cases) at both appraisal andpostevaluation. Nearly all projects in ENIN sectors and about 50 percent in AGSO sectors hadREIRRs.1

14. The REIRR represents the efficiency of the investment and is a major yardstickof project performance/rating at postevaluation. Generally, when the REIRR is above 10percentwhich is the usual EIRR benchmark at appraisal as wella project is rated asgenerally successful. A project is rated as partly successful when the REIRR is between 4 and10 percent, and unsuccessful when REIRR is below 4 percent.

15. The model focused on identifying the direction and magnitude of eachexplanatory factors potential impact. Typically, the question raised was whether the impactwould be large enough to effect a change in the performance rating one or two categories up ordown (e.g., from partly successful to successful).

16. The EIRRs at appraisal varied considerably from the REIRRs (Appendix 1, Table4). Disregarding external factors, when a project is appraised well, is implemented successfully,and operates efficiently, the gap in EIRRs should be zero.

17. A review of the data on the REIRR and the PGIR (Appendix 1, Tables 3 and 4)yielded two findings on project performance in AGSO sectors.

(i) Projects in AGSO sectors are performing less satisfactorily than those in ENINsectors. Projects in AGSO sectors have low REIRRsbelow 10 percent in 20 of35 cases, with an average of 10.5 percent2 (Appendix 1, Table 3). The average

of REIRRs for projects in ENIN sectors, at 18.5 percent, is well above thebenchmark for a rating of generally successful.3

(ii) The EIRRs at appraisal differ considerably from the REIRRs. The data suggestthat appraisal in both sectors was biased. In AGSO sectors, less than half thelevel of the EIRR estimated at appraisal was achieved at postevaluation.4 InENIN sectors,5 where the gap in EIRRs was smaller, about two thirds the level ofthe EIRR estimated at appraisal was achieved at postevaluation.

1Projects in Agriculture and Support Services (AGSS), Water Supply and Sanitation (WSS), Education, andHealth and Population (HAP) sectors were not featured, as there were few or no cases with REIRRs.Economic analysis was not required by the Bank for projects in these sectors during that approval period.

2REIRRs ranged from 0 to 18 percent, except three cases, which had REIRRs above 20 percent.

3In nearly 25 of 34 projects, REIRRs were above 10 percent. In three cases, REIRRs were higher than 35

percent.4In AGSO sectors, PGIRs ranged from minus 88 percent to plus 38 percent. Except in four cases, PGIRswere below zero, indicating that REIRR is below EIRR at appraisal. In ENIN sectors, PGIRs ranged fromminus 100 percent to plus 100 percent; in one case, it was plus 300 percent; PGIRs were positive in 9 of 31cases.

5In the case of projects in the energy sector, such differences between the EIRRs at appraisal and theREIRRs occurred mainly because the assumptions at appraisal relating to increases in tariffs were not fullyachieved due to sociopolitical reasons.


16/28

10

B. Typically Monitored Indicators of Project Performance

1. Delays

18. In the study, time variations were significant in both AGSO and ENIN sectors.The average delay was about 2 years (Appendix 1, Table 5). Projects in Water Supply and

Sanitation (WSS), Health and Population (HAP), and power sectors had long delays of over 40months.1 A trend of increasing delays by evaluation year was observed (Appendix 1, Table 6). Itmay be appropriate to anticipate an average of 2-year delay and reflect it in the sensitivityanalysis in the economic evaluation in Reports and Recommendations of the President.

19. With respect to completion on time or ahead of schedule, the proportion ofprojects was high in ENIN sectors28 percent compared with 15 percent in AGSO sectors.

20. The magnitude of time overrun was generally greater for projects in ENINsectors. The reason could be that estimates of their implementation periods at appraisal wereunrealistic. The average implementation period at appraisal was an ambitious 2.75 years inENIN sectorsmuch below the same for projects in AGSO sectors, which was 4.73 years, andthe average actual implementation period of 4.66 years for projects in ENIN sectors.Understandably, nearly one third of projects in ENIN sectors experienced more than 100percent time overrun (Appendix 1, Table 7).

21. According to the model analysis, other things being equal, every unit of increasein time overrunmeaning 100 percent delay in completion, would result in reduction by 2.2percentage points in the REIRR, and a deviation by minus 11 percent from the EIRR atappraisal. The coefficients were zero for AGSO sectors, implying that managing delays alonewould not improve project performance measured in terms of REIRRs.

2. Cost Underrun/Overrun

22. Cost underruns were more common than overruns in both sector groups. Theunderruns were greater in projects in AGSO sectors.2

23. The cost underrun/overrun affected the REIRR significantly only in ENIN sectors.The REIRR model revealed that, other things being equal, the average impact of 50 percentoverrun is to decrease the REIRR by 1.6 percentage points. The coefficients were zero forAGSO sectors, implying that even substantial cost underruns would not significantly improve

project performance in terms of REIRRs.1

Overall, 14 projects in AGSO sectors (7 in ENIN sectors) had delays of over 40 months. Another 16 inAGSO sectors (4 in ENIN sectors) had delays of over 30 and up to 40 months.

2Some reasons cited in the PPARs for such cost underruns were (i) devaluation of the local currency; (ii) costcutting in the purchase of equipment or materials, especially in the education sector. Only 22 of 88 (25percent) projects in AGSO sectors had cost overruns, while 18 of 37 (48 percent) projects in ENIN sectorshad overruns (Appendix 1, Table 8).


17/28

11

3. Bank Supervision

24. Bank supervision, in person-days, was on average above 100 person-days for aproject in AGSO sectors; however, it was above 100 person-days in only 13 of 48 projects inENIN sectors (Appendix 1, Table 9). The input variable, person-days spent on project review,however, was inadequate to assess whether Bank supervision was effective or not. In themodels, the coefficients for Bank supervision were zero for both AGSO and ENIN sectorgroups.1

1Adequacy of Bank supervision was rated only for projects in AGSO sectors (Appendix 1, Table 10). Theaverage of ratings was between generally adequate and partly adequate. However, according to the models,the marginal impact of adequacy of Bank supervision on EIRRs was not significant.


18/28

12

C. Indicators of Project Performance Specific to AGSO Sectors

1. The Banks Performance

25. The Banks performance was found important for project performance. The

following variables were used in the models:

(i) presence or absence of PPTA, and(ii) qualitative rating on the project design.

a. Presence of PPTA

26. Overall, 93 of 141 projects were preceded by PPTA (Appendix 1, Table 11). Ofthe 93 projects, 45 percent were rated as generally successful, 42 percent as partly successful,and 10 percent as unsuccessful (Appendix 1, Table 14). On the positive side, 85 percent of thegenerally successful projects in AGSO sectors were preceded by a PPTA.1 This partiallysupports the Banks process cycle, which normally involves a PPTA for a loan project. However,among the AGSO projects preceded by a PPTA, a considerable number (64 percent) wereeither partly successful or unsuccessful. It appears that a PPTA is largely necessary for projectsuccess, but is by no means sufficient.2

27. First, PPTAs did not guarantee the outcome of projects in AGSO sectors(Appendix 1, Tables 12 and 13).3 Evidently, the average of REIRRs for projects with PPTA fellsharply, even below those without PPTA. This raises a serious concern on the quality ofeconomic analysis at appraisal, presumably based on the PPTA.4

28. The coefficient for the presence of a PPTA, was minus 5.2 in the REIRR model,meaning that the average REIRR for projects that were preceded by PPTA was 5.2 percentagepoints lower than the average for projects with no PPTA.

29. Second, the PGIR model revealed that, other things being equal, the presence ofa PPTA resulted in huge negative gaps in EIRRs, suggesting an overestimation of economicimpacts at appraisalprobably assuming conditions that could not be actually achieved duringimplementation.5 This is especially applicable to partly successful projects; the average of

1For ENIN sectors, the value was 72 percent.

2The Projects Department suggested that projects without PPTAs may perform better because they aresimpler in design and may be repeats of previous projects.

3

The overall performance of projects with PPTA in AGSO sectors during the approval period 1981-1987exhibited a declining annual trend (Appendix 1, Tables 14 and 15).

4For projects that were preceded by PPTA, the averages of EIRRs were 25.6 percent at appraisal, 18.7percent at project completion, and 10.3 percent at postevaluation, respectively; however, similar values forprojects that were not preceded by PPTA, were 21.3 percent, 15.1, and 14 percent, respectively.

5The coefficient was minus 45 in the PGIR model, meaning that the average negative deviation in REIRRsfrom EIRRs at appraisal for projects with PPTA was 45 percent compared with that for projects withoutPPTA. As a result, the average PGIR for all projects in AGSO sectors was very low at minus 61 percent.


19/28

13

EIRRs at appraisal was even higher for these projects than for those that were rated asgenerally successful.130. As an additional note, the presence of a PPTA was also strongly associated withtime overruns, implying that PPTAs were not effective in ensuring timely implementation either.

b. Quality of Design

31. The quality of design was rated high, partly adequate, and low (Box 3). In about53 percent of the cases, the rating was only partly adequate (Appendix 1, Table 16). Morespecifically, such cases were from Irrigation and Rural Development (IRAD), fisheries, andagricultural and support services (AGSS) sectors. Incidentally, the data showed no statisticalevidence of linkage between high quality of design and the presence of PPTA. For example, forthe total of seven projects in AGSS sector that were preceded by PPTA, the quality of designwas partly adequate for five cases and low for the rest. Of the remaining eight AGSS sectorprojects not preceded by PPTA, four scored high in quality of design.

Box 3

High Quality of Design Low Quality of Design

Design is detailed, well prepared, andflexible for possible modifications; designused appropriate approach and hasappropriate components; and design includeseveral cost savings measures (PEO Nos.352, 358, 457, 486, 491, 501, 337, 396, 429,

484, 346, 430).

Limited analysis of policies and institutionalframework; inappropriate assessment andprovision of working capital; inflexibility andlimited focus; and insufficient or noconsideration of sociocultural aspects (PEONos. 350, 358, 370, 387, 393, 458, 486).

32. The coefficient for quality of design at entry was 2.9 in the REIRR model,meaning that the average impact of a unit improvement in the quality2 was to increase REIRRby 2.9 percentage points. Other things being equal, if the quality of design would improve tohigh for projects in AGSO sectors,3 the average of REIRRs would go up from 10.5 percent to13.5 percent; more projects would qualify for the generally successful rating (Appendix 1, Table16).

2. Performance of the Consultants During Implementation

1

The average EIRR at appraisal was 27.7 percent for partly successful projects with PPTA and 25.5 percentfor generally successful projects with PPTA; the corresponding averages of REIRR were 5.9 percent and14.9 percent, respectively (Appendix 1, Table 12). The pattern is the same for the without PPTA case.

2For example, the rating improves from low (rating = 0) to partly adequate (rating = 1).

3The project-weighted average of ratings was 0.95. The improvement in quality, to high in all projects, willresult in an increase of 1.05 units in the average rating, from 0.95 (current average) to 2 (rating for highquality). The resultant impact on the REIRR is 1.05 times the coefficient (2.9), which is 3.0 percentagepoints.


20/28

14

33. The adequacy of consulting services was rated qualitatively (Box 4). Statisticalanalysis revealed that when the quality of PPTA was high, the performance of consultants wasalso adequate during implementation for all AGSO projects.1

1The analysis identified a positive correlation significant at a probability of rejection of 1 percent. Furtherreview of individual projects in this regard was not undertaken.


21/28

15

Box 4

Adequate Consulting Services Inadequate Consulting Services

Performance of consultants was consistentwith the terms of reference; efficientconsulting services with respect to siteinvestigation, design submission and constructionsupervision; on-schedule accomplishment of

work; acceptable development scheme andsuccessful transfer of new technology (PEONos. 390, 399, 412, 422, 439, 451).

Insufficient professional expertise on sector/country development; slow performance ofconsultants and incapability to adapt tochanges; deficiencies in coordinating andscheduling of project activities; no under-

taking of relevant surveys; poor workingrelationship with EA staff; limited under-standing of beneficiaries sociocultural traits;and overlooking of important qualifyingconsiderations and technical aspects (PEONos. 354, 384, 357, 375, 493, 463, 443).

34. The coefficient for adequacy of consulting services was 2.5 in the REIRR model,i.e., every unit increase in the rating on adequacy of consulting services would have raised theREIRR by 2.5 percentage points. For example, if the rating for adequacy of consulting services,which was only partly successful in several projects in IRAD sector, improved to adequate,1 theaverage of the REIRRs would have been higher by 4.5 percentage points: from 9.1 percent to13.6 percent. There would have been more projects with the rating of generally successful(Appendix 1, Table 18).

3. Performance of the Executing Agency

35. The EAs performance was found to be even more important for project outcome.

36. The following variables were used in the analysis:

(i) qualitative rating on capability to supervise project implementation,(ii) qualitative rating on utilization of project facilities, and(iii) qualitative rating on maintenance and repair of project facilities.

a. Capability to Supervise Project Implementation

37. Nearly 65 percent of the EAs in country group A were rated incapable or partlycapable, compared with only 39 percent in group B countries2 (Appendix 1, Table 19). Box 5

1

The average rating was at 1.2. The improvement would result in a REIRR equivalent to 1.8 times thecoefficient.

2Group A countries are Afghanistan, Bangladesh, Bhutan, Cambodia, Peoples Republic of China, CookIslands, India, Kiribati, Kyrgyz Republic, Lap Peoples Democratic Republic (Lao PDR), Maldives, Republicof Marshall Islands, Federated States of Micronesia, Mongolia, Myanmar, Nepal, Pakistan, Solomon Islands,Sri Lanka, Tonga, Tuvalu, Vanuatu, Viet Nam, and Western Samoa; Group B, Indonesia, Kazakhstan,Republic of Nauru, Papua New Guinea, Philippines, and Thailand; and Group C, Fiji, Hong Kong, Republicof Korea, Malaysia, Singapore, and Taipei, China.


22/28

16

gives some characteristics of EAs with adequate and inadequate capability. The coefficient forcapability of EA was 4 in the REIRR model, so that the average effect of every unit increase inthe rating is to increase the average of REIRRs by 4 percentage points. For example, if EAs forprojects in IRAD sector were to perform as well as those for projects in the urban developmentand housing (URD) sector,1 the average of REIRRs would go up from 9.1 percent to 12.7percent.

Box 5

Adequate Capability of EA Inadequate Capability of EA

Well-qualified, committed, and well-equipped staff for project management;smooth/effective coordination between EAs;sound working relationship withbeneficiaries, consultants, and Bank staff;compliance with project-specific covenants;prompt appointment of project personnel;appropriate organizational structure; strongcapacity to run continuing programs andmarket projects (PEO Nos. 332, 410, 418,419, 434, 468, 478, 348, 461, 480, 494,500).

Reliance on external funding; insufficientinterest among EA staff, frequent staffturnover and untimely appointment of keymanagement staff; deficient compliance withproject-specific covenants; inadequateexpertise in specific project/sector dataanalysis; inability to overcome weakness ofdata or to reassess the fundamentalassumptions of project design; inappropriatebenefit monitoring and evaluation andabsence of specific organizational unit tomonitor; nonfamiliarity with procurementprocedures; weak accounting and reportingsystems (PEO Nos. 330, 333, 388, 452,398, 477, 400).

b. Utilization of Facilities

38. The average of ratings was 1.1, between moderately utilized and highly utilized(Appendix 1, Table 17).2 The analysis revealed that in cases where facilities were rated highlyutilized, the EA was rated capable. It was also found that in such cases, the performance ofconsulting services were also adequate. Box 6 gives some reasons for nonutilization of projectfacilities.

Box 6

Highly Utilized Project Facilities Unutilized Project Facilities

Most facilities were fully/well utilized; andmajor modifications of facilities allowedsatisfactory operation (PEO Nos. 352 and430).

Facilities unutilized due to budgetconstraints; untimely use of facilities;improper use of credit facility due toinsufficient collateral; unutilized facilities dueto poor location and poor physical working

condition (PEO Nos. 458 and 486).

1The average of ratings for capability of EA was 1.7 for IRAD sector and 2.6 for URD sector.

20 = facilities not utilized, 1 = moderately utilized, and 2 = highly utilized.


23/28

17

39. The coefficient for utilization of facilities was 3 in the REIRR model, so that everyunit increase in the rating will increase the average of REIRRs by 3 percentage points. In thePGIR model, the coefficient for utilization of facilities was 11. Other things being equal, forexample, if utilization of facilities in projects in IRAD sector1 improved to highly utilized, then theaverage of REIRRs would improve from 9.1 percent to 11.8 percent.

c. Maintenance and Repair

40. Projects in country group A suffered from poor maintenance, while those in groupB fared better (Appendix 1, Table 20). Information was available for only 60 percent of thecases.2 Maintenance was rated inadequate/partly adequate in most projects.3 It appeared thatmaintenance and repair were not distinctly satisfactory even for outstanding projects in AGSOsectors. Maintenance was rated adequate in only 7 percent of the cases. Consequently, in themodels, the coefficients for this indicator were not significant. Considering the close linkage ofadequate maintenance with sustainable and efficient operations, the Bank should place greateremphasis on ensuring adequate maintenance and repair of projects in AGSO sectors.

4. Performance of the Borrower

a. Qualitative Rating

41. The borrowers performance is represented by a composite index signifying thepresence of supportive policies, adequate domestic funding, and commitment to the project(Box 7). It was found to be the most important indicator of project outcome.4

42. In 86 percent of the cases, the borrowers performance was consideredsatisfactory. Such cases were equally distributed among country groups A and B (Appendix 1,Table 21). The borrowers performance was unsatisfactory in all projects rated as unsuccessful.

Box 7

Satisfactory BorrowersPerformance Unsatisfactory Borrowers PerformanceSuccessful privatization of productdistribution; provision of adequate budget;encouragement of increases in schoolenrollment; promotion of aquaculture andinland fisheries development; priority tomeasures to reduce incidence of disease

Little progress in reforms; governmentpolicies inconsistent with project goals; lackof support for policy changes by governmentbureaucracy; inadequate attention to repairdeteriorated project facilities and funds forregular maintenance; withdrawal of state

1The average rating was 1.1.

2Maintenance and repair were rated 0 when inadequate, 1 when partly adequate, 2 when generallyadequate, and 3 when adequate.

3The averages of ratings were partly adequate (just above 1.0) in IRAD, forestry, and URD sectors. Averageswere below a worrisome 0.5 in education and HAP sectors.

4A qualitative rating on performance was used: zero = unsatisfactory performance, and 1 = satisfactoryperformance.


24/28

18

and to protect the environment (PEO Nos.443, 358, 370, 457, 458, 486, 491, 499, 337,396, 429, 484, 346, 432).

patronage from inland freshwater fisheryactivities; and termination of project activitiesand reduction of project scope (PEO Nos.443, 451, 390, 463, 493, 375).

43. The coefficient for borrowers performance was 7 in the REIRR model, so thatthe average effect of satisfactory performance was to improve REIRR by 7 percentage points. Inother words, unsatisfactory borrowers performance as opposed to general expectations at

appraisal alone would result in a decline in the REIRR by 7 percentage points; the average ofthe REIRRs would be 3.5 percent rather than 10.5 percent: most projects would be rated asunsuccessful.

b. Country Economic Performance Index

44. The models used an index of country economic performance developed by WB1(Appendix 1, Table 22). On a scale of 1 to 5, the index represents the average countryperformance rating for the period 1979 to 1992. This covers the approval periods (1981 to 1992)for cases in this study.

45. The averages of the REIRRs for country groups A and B were 14.7 percent and15.4 percent, respectively. The average of PGIRs for both groups was minus 27 percent.

46. In the REIRR model, other things being equal, if Nepal, the country with thepoorest performance rating (2.73) mimicked Thailand, the country with a high performancerating (4.13), the average of REIRRs for Nepals portfolio would significantly improve by 15percentage points. This suggests that the Bank needs different policies to help countries likeNepal, which face substantial challenges that impede development.

47. The specific significance of the index in AGSO sectors could not be perceivedbecause of constraints, which concentrated data in only some countries.2

c. Country Policy Environment and Institutional Quality

48. The study supported previous findings that the quality of a countrysmacroeconomic policy environment is an important factor in differing performance within itsproject portfolio.3 The study used the WBs classification of DMCs in a 2x2 matrix by policy

1The index rates countries based on separate ratings for equally weighted components: short-term economic

management, long-term economic management, poverty reduction, and responsiveness to WB. The indexnot only captures countries medium- and long-term structural conditions but also avoids the collinearityproblems in various macroeconomic indicators (Source: WB. 1995. Annual Review of Evaluation Reports).

2All cases in AGSO sectors were crowded among five countries, whose indices ranged between 3 and 3.8.

3The 1997 World Development Report, The State in a Changing World, finds that sound macroeconomicpolicy and capable state institutions have large impacts on a countrys long-term growth performance and onthe effectiveness of public sector investments. This was confirmed in another WB study (WB. 1997. AnnualReview of Development Effectiveness(ARDE).


25/28

19

performance and institutional quality (in other words, governance).1 The average performance ofthe Banks portfolio was calculated for each country and group of countries.2

49. The average project portfolio performance, for the group of countries with bothhigh policy performance and high institutional quality was 0.84 (as against 1.00 for the bestperformance) compared with 0.72 for the group with low policy performance, but partlyadequate institutional quality (because there are no low countries among Bank DMCs) (Table4).

Table 4: Country Classification by Policy Performance and InstitutionalQuality, and Average Project Portfolio Performance

a

Policy PerformanceInstitutional High Medium Low

Quality Country APR Country APR Country APR

High ThailandMalaysiaGroup Average

1.00 (8)0.73 (11)0.84

PartlyAdequate

PakistanSri LankaGroup Average

0.75 (14)0.68 (11)0.72

Low IndonesiaPhilippinesGroup Average

0.7 (25)0.6 (15)0.66

Bangladesh 0.7 (15)

aAPR = average project portfolio performance. The number of projects in each country is shown inparentheses. The group average is the project-weighted average for all countries in a group.

50. The interrelationship between best policy and effective institutions also suggeststhat getting institutions right brings significant gains not only in portfolio performance but alsoin the future stability of the policy environment.

5. External Factors

51. The effect of external factors on project performance was also observed (Box 8).During project implementation, prices decreased in several cases (Appendix 1, Table 23). Thestudy was limited to analyzing the statistical significance of the impact of this important factor onthe REIRR or the PGIR, as more than half the PPARs (56 percent) did not report on this factorexplicitly. Projects in WSS, IRAD, and AGSS sectors suffered from other external effects suchas political unrest, inclement weather, and recession (Appendix 1, Table 24). Due to dataconstraints, the models did not reveal any statistically significant impact of price decreases onthe REIRR or the PGIR.

1The measure of policy performance is a project-weighted index based on three principal components:inflation, fiscal balance, and openness. Institutional quality, which is also project-weighted, is a composite ofmeasures of quality of government: the extent of red tape involved in any transaction, the regulatoryenvironment, and the degree of autonomy from political pressure (Source: WB. 1997. ARDE).

2The average project portfolio performance is the project-weighted average of performance rating (in whichgenerally successful = 1, partly successful = 0.5, and unsuccessful = 0) of a countrys project portfolio.


26/28

20

Box 8

Significant External Factors

Change of government in 1986 created uncertainty; political attentionfocused on the possible privatization of the units complexes; intermittent civildisturbances; catastrophic floods in 1987-1988 affected loan recovery rate;occurrence of a hazardous fire; economic recession; and major politicalupheaval and peace and order problems (PEO Nos. 337, 346, 350, 354, 384,

412, 430, 435, 443, 463).

III. CONCLUSIONS AND IMPLICATIONS

52. The study faced various data constraints and other limitations (para. 9), includinga restricted number from social infrastructure projects. It yielded indicative results and reinforcedlessons already known but not fully addressed in the Bank.1

53. The study confirmed that monitoring the project implementation schedule anddisbursement levelswhich in effect means, managing time and cost overrunswasinsufficient to improve the performance of projects in AGSO sectors. Such factors appearedmore important in ENIN sectors.

54. For AGSO projects studied, equal attention should have been paid to other

factorsboth project specific and externalwhose impact on project performance wassignificant. This supports the recent effort in the Bank to introduce a broader, more impact-oriented approach to project implementation monitoring and management based on the projectperformance report.

55. Regression models identified such factors as macroeconomic performance andpolicy and governance to be critical determinants of performance in AGSO sectors. At thecountry level, the Bank needed to assess more realistically the macroeconomic and policyenvironment, and choose a set of interventions that addressed existing constraints. At theproject level, in countries where the policy framework was distorted and the implementationenvironment was weak, necessary advisory and capacity building services were frequentlyneeded to precede lending activities.

56. The study confirmed that the borrowers performance was the most critical

determinant of project success. Good performance? which implies a policy environmentfavorable to project operation, government commitment to the project, and provision of domesticfunds? could improve project performance significantly.

57. After borrower performance, project success in AGSO sectors depended on theEAs capability to supervise project implementation and properly utilize facilities, and itscommitment to maintenance and repair. The Banks continuing focus on encouraging

1The 1998 Annual Performance Evaluation Programincluded similar findings.


27/28

21

commitment to ownership of project goals by increasing EA involvement in design andimplementation is well underscored.

58. Statistical results show that adequate consulting services during implementationyielded significant rewards by sharply reducing the gap in EIRRs between appraisal andreevaluation.

59. The Banks foremost contribution can come at the PPTA stage where, inparticular, greater realism could be required. The PPTAs included in this study did not generallyensure satisfactory project design or adequately guard against overestimating the EIRRs. Thestudy suggests a possible need to undertake a critical review of Bank policies and practicesrelating to PPTAs with a view to enhancing their positive contribution to project performance.


28/28

22

APPENDIXES

Number Title Page Cited on(page, para.)

1 Statistical Tables 18 4,8

2 Bibliography 33 4,8

3 Database on Indicators and Regression Analysis 34 4,8

factors affecting project performance in the agriculture and social sectors: a review of...

Documents