memorandum to the executive directors subject: fy01 annual...

MEMORANDUM TO THE EXECUTIVE DIRECTORS

Subject: FY01 Annual Report on Portfolio Performance (ARPP) This year’s ARPP finds that the health of the portfolio has improved substantially for the fifth consecutive year and now meets key targets set by Management and the Board. The share of the portfolio at risk is now less than half what it was in FY96. The improvement has been broad-based covering almost all Regions and Networks. This encouraging result is reinforced by the upward trend in OED evaluations and correlates with significant improvements in the quality of the Bank’s work in project preparation and appraisal, supervision, and Economic and Sector Work. The global economic slowdown and events of September 11 may adversely impact the portfolio in the short-run. The Bank is clearly well positioned, however, to deliver successful project outcomes in the 80-85 percent range. Going forward, the challenge is to sustain the gains made so far while giving heightened attention to a few areas with significant potential for further benefits. The agenda for the near term includes refining our portfolio monitoring system to better identify and manage risks. We will further strengthen the management of Economic and Sector Work, an important component of our non- lending services, with the aim of progressively integrating other aspects of non- lending into our portfolio management system. We will also improve our measurement and monitoring of the poverty orientation of the portfolio to better align it with key corporate and Millennium Development Goals. James D. Wolfensohn President

by Shengman Zhang

AANNNNUUAALL RREEPPOORRTT OONN PPOORRTTFFOOLLIIOO PPEERRFFOORRMMAANNCCEE

FFIISSCCAALL YYEEAARR 22000011

DECEMBER 17, 2001

ACRONYMS AND ABBREVIATIONS

AAA Analytical and Advisory Services ACS Activity Completion Summary AFR Africa Region ARPP Annual Report on Portfolio Performance ARDE Annual Report on Development Effectiveness APL Adaptable Program Loan BW Business Warehouse DO Development Objective CAS Country Assistance Strategy CDF Comprehensive Development Framework CFAA Country Financial Accountability Assessment CODE Committee on Development Effectiveness CPAR Country Procurement Assessment Review CPIA Country Policy and Institutional Assessment CPPR Country Portfolio Performance Review CRM Corporate Resource Management Group DEC Development Economics and Chief Economists EAP East Asia and Pacific Region ECA Europe and Central Asia Region ESSD Environmentally and Socially Sustainable

Development Network ESW Economic and Sector Work FIL Financial Intermediary Loan FSE Finance Network GEF Global Environment Facility HDN Human Development Network IAD Internal Auditing Department ICR Implémentation Complétion Report IP Implementation Progress IRIS Integrated Records and Infrastructure System LACI Loan Administration Change Initiative LCR Latin America and the Caribbean Region

LIL Learning and Innovation Loan M&E Monitoring and Evaluation MDG Millennium Development Goals MNA Middle East and North Africa Region MOP Memorandum of President MP Montreal Protocol OED Operations Evaluation Department OPCS Operations Policy and Core Services PAD Project Appraisal Document PER Public Expenditure Review PIP Portfolio Improvement Program PRSC Poverty Reduction Support Credit PRSP Poverty Reduction Strategy Paper PREM Poverty Reduction and Economic

Management Network PSR Project Status Report QEA Quality at Entry Assessment QAG Quality Assurance Group QSA Quality of Supervision Assessment QER Quality Enhancement Review QESW Quality of Economic and Sector Work RDW Rural Development Department RVP Regional Vice President SAR South Asia Region SIL Specific Investment Loan SSP Sector Strategy Paper TF Trust Fund TAL Technical Assistance Loan TTL Task Team Leader WBI World Bank Institute WSS Water Supply & Sanitation

ACKNOWLEDGEMENTS

This report is the product of a Bank-wide team effort, with inputs from almost all VPUs across the Bank. It was prepared by a team co-managed by Tom Zearley and David Hughart, and included Praveen Kumar, Giovanna Prennushi, Kene Ezemenari, Chitra Bhanu, Ronald Ridker, John Hayward, Y.T. Shetty, Jane Loos, Jason Mayfield, Bing Deng, Leila Cruz, Shashi Gupta, Susan Crisostomo, and Ann Walters. Useful suggestions and contributions were received from a large number of colleagues across the Bank, including Jane Armitage, Suman Babbar, Judith Baker, Shawki Barghouti, Antonella Bassani, Harold Bedoya, Connie Bernard, Jaime Biderman, Marc Blanc, Jerry Caprio, Charles Chandler, Alain Colliou, Anis Dani, Cornelis De HaanRobert Drysdale, Victoria Elliot, Wendy Fleit, Lucia Fort, Fritz Ford, Ruth Kagia, Christine Kessides, Odin Knudsen, Stefan Koeberle, Jody Zall Kusek, Laszlo Lovei, Alastair Mckechnie, Jean Roger Mercier, Bronagh Murphy, Sarah Nedolast, Hoveida Nobakht, Felicity Proctor, John Roome, Rene Ruivivar, Jamal Saghir, Anand Seth, Narendra Sharma, John Sinclair, Inder Sud, John Underwood, Keshav Varma, Samuel Wedderburn, Ai Chin Wee, Richard Westin, Ve ra Wilhelm, Elizabeth White, Ulrich Zachau, and Jaime Zaldivar. Prem Garg, Director, Quality Assurance Group, guided the overall effort.

ANNUAL REPORT ON PORTFOLIO PERFORMANCE FISCAL YEAR 2001

CONTENTS

PAGE NO .

EXECUTIVE SUMMARY...................................................................................................................................................... i REPORT

I. Introduction.............................................................................................................................................................. 1 II. Portfolio Size and Composition ........................................................................................................................... 2 III. Portfolio Performance ............................................................................................................................................ 11 IV. Managing Quality and Risks for Results ............................................................................................................ 21 V. Improving ESW Quality and Management........................................................................................................ 28 VI. Measuring the Poverty Orientation of the Portfolio.......................................................................................... 37 VII. Recommendations................................................................................................................................................... 41

TEXT TABLES Table 2.1 Bank Portfolio, FY96-FY00 ....................................................................................................................... 2 Table 2.2 FY01 Portfolio by Lending Instrument..................................................................................................... 5 Table 3.1 Portfolio Improvement by Source of Financing...................................................................................... 12 Table 3.2 Portfolio Improvement by Region ............................................................................................................. 12 Table 3.3 Portfolio Improvement by Network........................................................................................................... 13 Table 3.4 Quality Assessment Results and Portfolio Management Indices ......................................................... 16 Table 3.5 Net Disconnect.............................................................................................................................................. 19

FIGURES Figure 2.1 Portfolio Distribution by Region FY96-01............................................................................................. 7 Figure 2.2 Portfolio Distribution by Network FY96-01……………………………………………………. 8 Figure 3.1 Projects at Risk by Region and Network................................................................................................ 12 Figure 5.1 Direct Administrative Spending on Analytical and Advisory Activities .......................................... 29 Figure 5.2 ESW Spending by Country Lending Volume, FY00-01 ..................................................................... 30 Figure 5.3 ESW Spending by CPIA and Population................................................................................................ 31

BOXES Box 2.1 Impact of September 11 Events on Portfolio ......................................................................................... 4 Box 2.2 Increasing Education Lending .................................................................................................................... 8 Box 3.1 Turning Portfolios Around: The SAR Experience................................................................................. 13 Box 3.2 Credibility of Portfolio Risk Assessment ................................................................................................. 14 Box 3.3 Net Disconnect: Causes and Cures ............................................................................................................ 20 Box 4.1 Status of FY00 ARPP Recommendations................................................................................................ 21 Box 4.2 ECA’s Program for Improving Project DOs ............................................................................................ 27

ANNEX Annex 1: Guarantees......................................................................................................................................................... 41

STATISTICAL APPENDIX

Annual Report on Portfolio Performance, FY01 i

EXECUTIVE SUMMARY

1. The Annual Report on Portfolio Perfomance (ARPP) provides a strategic overview to the Board and senior management of the status of Bank operations, the impact of the actions taken to improve them and the likely future evolution of the portfolio. It also identifies priority actions needed to sustain the gains in portfolio quality in recent years. In addition, this ARPP examines two special topics: the quality and management of Economic and Sector Work (ESW), and an approach to assessing the poverty orientation of the portfolio.

2. Portfolio size and composition. The Bank’s portfolio at the end of FY01 was about 20 percent smaller in terms of commitments than it was five years earlier, and most of the decline was in the IBRD portfolio. Half of the decline in commitments occurred in FY01. Closing of adjustment operations from the last global financial crisis accounts for much of the FY01 drop, but the most important factor over the five-year period was a decrease in the average size of IBRD investment operations (from $106 million in FY96 to $87 million in FY01). Current lending programs suggest some further shrinkage of the portfolio over the medium term, but forecasting is made difficult by uncertainties created by the events of September 11.

3. The sectoral composition has shifted away from PSI projects and towards emerging Bank priorities such as Health, Social Protection and the Environment. Other high priority sectoral and thematic initiatives receiving support in FY01 included AIDS/HIV, programmatic lending for poverty reduction, post-conflict reconstruction, governance and anti corruption, and gender equality. The size of the Education portfolio has been shrinking since FY96. But new Education lending increased in FY01, and actions are being taken to sustain demand. The Education portfolio warrants close monitoring given its critical role in poverty reduction.

4. The current sectoral classification system is not well adapte d to monitoring or reporting on alignment with Millenium Development Goals or other corporate objectives. Nor does it correspond to the classification systems being used by the Bank’s development partners. The new system of thematic and sectoral coding, now under implementation, is an important step toward addressing both of these issues.

5. The rules and procedures governing the Bank’s menu of lending instruments needs updating and clarifying. Work is underway on a revised Bank operational policy for adjustment lending. OPCS also plans to launch systematic reviews of investment lending instruments.

6. Portfolio Performance . The share of the portfolio at risk is now less than half what it was five years ago. Both commitments at risk and projects at risk fell to 12 percent in FY01. This encouraging result is reinforced by the upward trend in OED evaluations and correlates with significant improvements in the quality of the Bank’s work in project preparation and appraisal, supervision, and economic and sector work. The improvement has been broad-based covering practically all Regions and Networks.

7. There is no doubt that the overall improvement has been substantial. Nevertheless, the figures generated by the current at risk system may somewhat overstate the real decline in riskiness. Also as the number of problem and at risk projects has declined, the portfolio indicators at the regional and network levels are becoming ratios of smaller numbers and hence

Annual Report on Portfolio Performance, FY01 ii

more volatile. While QAG plans to review the at risk system, more candid supervision reporting and ratings at the project level is also needed.

8. An analysis of future prospects, using historical correlations between outcomes and key drivers like quality at-entry and quality of supervision, indicate that the Bank portfolio is well positioned to deliver successful outcomes in the 80-85 percent range. At the same time, OED evaluations continue to show that the Bank needs to pay greater attention to sustainability .

9. Managing Quality and Risks for Results. There was significant progress in FY01 in strengthening portfolio management tools and systems. New operational guidelines for project supervision were issued, and management information systems further enhanced. The budget for core operations has been increased in FY02 in order to sustain improvements in portfolio management.

10. Two additional areas of strategic importance in managing for results have been identified: raising the profile of sustainability in portfolio management, and sharpening the statement of project-level development objectives (DOs). The presentation of DOs in project documents needs to be improved.

11. Improving ESW Quality and Management. Analytical and Advisory Services (AAA) are an important part of the Bank's work and should be integrated into the "portfolio" with monitoring, managing and evaluation tools analogous to those for lending. This ARPP is starting the process with ESW.

12. The direct cost of ESW to the Bank over the past five years has averaged $60 million. QAG assessments indicate that the quality of ESW has improved. Despite increased attention by senior management in the past two years, ESW remains under-managed compared with the lending program. This is reflected in ESW delivery delays, over-programming, and cost over runs. The Activity Completion Summary (ACS), put in place several years ago to provide feedback on the impact of ESW, is also not being used effectively.

13. Measuring the poverty orientation of the portfolio. The Bank needs to improve its measurement of the poverty orientation of the portfolio. The Program of Targeted Intervention (PTI), which has been the only indicator of the poverty focus of the Bank’s portfolio is overly simplistic. It should be replaced with a broader conceptual framework and set of indicators focused on the poverty orientation of (i) country programs, (ii) sector support strategies, and (iii) individual operations.

KEY RECOMMENDATIONS

• Review and refine the portfolio monitoring indicators and the at risk flag system.

• Give greater emphasis to sustainability throughout the project cycle. Sharpen statements of Development Objectives in project documents especially in the Project Appraisal Document.

• Develop and mainstream mechanisms to systematically monitor the Networks’ contribution to enhancing operational quality.

Annual Report on Portfolio Performance, FY01 iii

• Continue strengthening ESW management to bring it up to par with lending. Progressively integrate AAA into the "portfolio" for the purpose of monitoring and managing.

• Replace the PTI as the primary indicator of the poverty orientation of the Bank’s portfolio with a broader conceptual framework and set of indicators focused on (i) country programs, (ii) sector strategies, and (iii) individual operations.

Annual Report on Portfolio Performance, FY01 1

I. INTRODUCTION

OBJECTIVES AND APPROACH

1.1 The Annual Report on Portfolio Performa nce (ARPP) provides an opportunity for the Board and Senior Management to focus on the Bank’s portfolio—the primary vehicle for delivering results to our clients. The main objectives, as defined in the Approach Paper reviewed by CODE,1 are as follows:

• Assess the current status of the Bank’s portfolio, and review portfolio management efforts and results in relation to the FY01 ARPP recommendations and the key messages in the Strategic Directions paper.

• Assess likely future trends and challenges to the portfolio over the medium-term.

• Examine two special topics: the quality and management of ESW, and the poverty orientation of the portfolio.

• Identify priority actions needed to sustain and build upon the gains in portfolio quality during the past few years.

1.2 The FY01 ARPP draws primarily from materials that are prepared as part of regular portfolio monitoring functions carried out by the Regions and Networks, supplemented by project/portfolio data in the Bank’s management information systems. Consistent with past ARPPs, the report uses a five-year time frame (FY96-01) to examine medium term trends in the portfolio. In preparing the ARPP, extensive consultations were held with staff from around the Bank to ensure a broad institutional perspective on portfolio management. An array of self-evaluative processes developed in the Bank over the past few years has also helped make the ARPP process more analytical and forward-looking. STRUCTURE AND COVERAGE 1.3 The report is organized into five subsequent Chapters. Chapter II reviews the recent trends in portfolio size and composition. Chapter III assesses portfolio performance across Regions and Sectors. It also reviews the outcome of the Portfolio Improvement Program (PIP) for FY01 and outlines portfolio priorities and targets for the future. Chapter IV covers progress made on recommendations from the FY00 ARPP, and portfolio management more generally. Chapter V brings Economic and Sector Work (ESW) into the ARPP “portfolio” for the first time. Chapter VI presents an initial conceptual framework and indicators for measuring the poverty orientation of the portfolio. Recommendations are summarized in Chapter VII. The Appendix contains a detailed set of supporting statistical material.

1 Approach Paper, FY01 Annual Report on Portfolio Performance (CODE2001-64), June 21, 2001.


II. PORTFOLIO SIZE AND COMPOSITION

The size of the IBRD/IDA portfolio has decreased significantly since FY96, with most of the reduction occurring in the IBRD component. Portfolio composition has remained relatively unchanged across regions over the past five years, with EAP and LCR continuing to have the largest share of commitments. AFR still accounts for the largest number of projects in the portfolio. In terms of sectoral composition, there has been a marked shift away from PSI projects towards emerging Bank priorities such as Health, Social Protection and Environment. The Education portfolio warrants close monitoring. The implementation of a new sectoral classification system should improve the monitoring of Bank activities and outputs.

PORTFOLIO SIZE

2.1 The Bank’s lending portfolio (IBRD/IDA/TF) at the end of FY01 consisted of 1,561 operations representing commitments of $105.9 billion. 2 The latter figure represents a 10 percent drop in nominal terms —12 percent in real terms —from the year before (Table 2.1). The sharp reduction resulted mainly from a steep increase in loan closings as many of the adjustment loans approved in FY99, an unusually high volume, left the portfolio.

2.2 The cumulative decline in the portfolio has been 19 percent in real terms from the peak in FY96. Over the five-year period, the dominant factor has been a drop of 20 percent in the average size of IBRD investment projects. The use of single -tranche adjustment operations—not usually captured in the ARPP portfolio because they generally open and close within a fiscal year—has also contributed. Eight such operations were approved in FY01 totaling $1.8 billion.

TABLE 2.1: BANK PORTFOLIO , FY96-FY01

2.3 IBRD net commitments have declined by nearly $20 billion since FY96, reaching $67 billion in FY01, the lowest level in more than a decade. The reduction was driven by an increase in portfolio exits much more than a decline in lending volume. Exits rose to $18.6 billion in FY01 from $13.3 billion in FY00, and from an average of $16.1 billion during FY96-99. The volume of new IBRD lending in FY01 fell only slightly to $10.5 billion from $10.9 billion in FY00. This level of lending represents a return to the experiences before the 1997-98 global financial crisis when IBRD lending surged to record heights of over $21 billion. Demand for IBRD lending slackened following the crisis with the improved global economic environment

2 The portfolio also includes 18 Guarantees provided under the Guarantee Program. Their performance is

reviewed separately in Annex 1.

Sources FY96 FY00 FY01 FY96 FY00 FY01

IBRD 779 768 717 86.5 82.8 67.0 IDA 746 737 739 39.2 33.3 37.3

TF 82 85 105 1.1 1.1 1.6

TOTAL: Nominal 1,607 1,590 1,561 126.8 117.3 105.9

Real /a 130.5 119.7 105.9

/a Based on the Planning Assumptions Committee Commitment Deflator for IBRD/IDA lending.

Number of Projects Net Commitments ($B)


and increased availability of private capital of the late 1990s. The shift away from large infrastructure projects and FSE lending also contributed to the decline, and is reflected in the portfolio in the reduction in the average size of IBRD investment projects from $106 million in the FY96 portfolio to $87 million in FY01.

2.4 IDA net commitments have declined more modestly over the last five years (to $37.3 billion in FY01 from $39.2 billion in FY96). New IDA lending actually increased in FY01 to $6.8 billion from $4.4 billion in FY00. Contributing to this increase was $1.8 billion in new adjustment lending. The FY01 increase was also due to a scale -up of activities by AFR in response to the HIV/AIDS crisis (through the first Multi-country HIV/AIDS Program for $500 million), post conflict situations (resumption of lending in Ethiopia and Eritrea), and oil price shocks (seven supplemental operations for US$157 million). 2.5 The portfolio of trust funds and other operations including GEF, Montreal Protocol (MP), and Special Financing, grew both in number and volume since FY96. There were 105 operations in the grant portion of the portfolio with commitments of $1.6 billion in FY01 (compared with 82 operations and $1.1 billion in FY96). The biggest increase in commitments occurred in the freestanding GEF portfolio, which rose to above $700 million in FY01 from about $500 million in FY96. 3 This rise was due to growing demand for GEF funding, especially from countries in AFR. The MP component, on the other hand, remained relatively stable at about $500 million. Special Financing programs, which are funded out of the Bank’s net income, grew in terms of number of operations (from 8 in FY96 to 37 in FY01) with new initiatives approved for Kosovo, East Timor and West Bank and Gaza, but overall outstanding commitments amounted to less than $400 million in FY01. 2.6 Currently planned levels of new lending suggest that some further shrinkage in the portfolio is likely. An annual lending program of $15-20 billion made up of $9-12 billion of IBRD loans and $6-8 billion of IDA credits are consistent with a portfolio size in the $90-100 billion range. But new lending could go beyond the upper ranges mentioned above if borrowers request additional funding to cope with the consequences of the September 11 events. (Box 2.1). Demand for adjustment lending in particular could rise substantially over the medium term.

3 The total active GEF portfolio amounted to $900 million at end-FY01, and was comprised of 49 free-

standing projects and 35 blended operations (with an IBRD loan or IDA credit).


INSTRUMENTS

2.7 The Bank’s lending portfolio is classified as either investment or adjustment. Investment operations account for 97 percent of projects and 91 percent of commitments. Specific investment loans (SILs) are the most widely used instrument, representing more than 75 percent of total commitments and 70 percent of operations in the FY01 portfolio ( Table 2.2). SILs appear to be the favored choice for a number of the Bank’s biggest borrowers. The recent QEA4 results show that the quality of SILs is high, and SIL portfolio performance in terms of riskiness is in line with the portfolio as a whole.

2.8 Adjustment lending, a category including seven different types of operations, has accounted for almost 40 percent of total IDA/IBRD lending volume over the last five years. However, because adjustment operations are relatively short lived, they comprise just over 3 percent of operations and 9 percent of commitments at end FY01. The sharp reduction of adjustment loan commitments to $6.4 billion at end FY01 (from $13.4 billion) reflects the closing of several large IBRD-funded operations particularly in Argentina, Russia, Brazil, and Mexico. Over the same period, IDA-backed adjustment loan commitments rose from $2.3 billion to $2.9 billion.

BOX 2.1: IMPACT OF SEPTEMBER 11 EVENTS ON PORTFOLIO

The full impact of the events of September 11 and their aftermath on the Bank’s portfolio remains uncertain. Portfolio management is expected to be more difficult in the foreseeable future. The recent events have delayed recovery from the 2001 global economic slowdown, and have increased risk perceptions in large emerging markets. Some deterioration in the riskiness and performance of the Bank's portfolio is likely in FY02 and beyond, as borrowers are adversely affected by one or more factors such as refugee inflows, reduced tourism, declining commodity prices and export demand, rising costs of trade transactions, and/or reduced access to international capital markets. The portfolio in some countries is likely to be directly affected by changed public expenditure priorities or by reduced availability of counterpart funds for Bank-financed projects. New or supplemental adjustment lending may be an option for affected countries with sound macroeconomic policies and reform programs. The Bank may also, within existing policies, advance the preparation or increase the scope of selected new investment projects, such as social funds, or provide supplemental investment operations where the impact of the crisis has led to project cost overruns. Emergency recovery operations may be provided in countries where the September 11 events have led to serious short-term economic dislocations, such as those caused by a sharp increase in the number of refugees. The Bank will also work with borrowers, if necessary, to restructure the existing portfolios—for example, by reallocating loan proceeds to mitigate the social impact of the crisis —or to increase or accelerate disbursements under existing operations, as appropriate.


TABLE 2.2 FY01 PORTFOLIO BY LENDING INSTRUMENT

2.9 QEA results reveal that adjustment operations are consistently well prepared. Strong features of SALs include close harmonization of objectives with the CAS, and its underpinning by extensive ESW; prudent use of the single tranche approach to implement difficult reforms before Board approval; rapid preparation by the borrower and Bank staff; and quick action by borrowers to meet conditions of effectiveness. Aspects with scope for improvements include failure to align the financing with the depth and complexity of the reforms; insufficient analysis of the agencies’ capacity to implement the reforms; shortcomings in the identification of risks combined with inadequate arrangements for evaluating impact and measuring outcomes; and need for more in depth analysis of the impact on stakeholders.

2.10 Emergency Recovery Loans (ERL) are designed to assist countries emerging from post-conflict, or to address emergencies. The volume and number of ERL operations in the portfolio is modest (37 projects with $2.9 billion of commitments in FY01) and performance somewhat better than average in terms of riskiness. A hallmark of these operations is the speed with which they can be prepared. In the QEA4 sample reviewed, an operation considered best practice in terms of responsiveness to client needs, took just eight days from concept review to Board presentation. Having taken into account the need for speed, QEA4 panelists nevertheless faulted several ERL operations for failing to explicitly consider strategic options and risks, and to link the operations to longer-term development goals.

2.11 Although the volume of free-standing Technical Assistance Loans (TALs) has declined since FY96, TALs remain important, especially in the areas of public sector reform and economic management. More than one third of the Bank’s free-standing technical assistance (121 operations, and $2.5 billion of commitments in FY01) is managed by the PREM network, and close to a third is directed to public sector reform. Although the performance of TALs has improved significantly since the early 1990s, the supervision quality still lags behind that of the rest of the portfolio. The PREM network made a number of recommendations to improve outcomes. Key proposals include allowing for a flexible approach to handle the complexities and risks of TA operations; adopting new, more streamlined procurement approaches; and facilitating monitoring and Bank-wide learning for TA by developing performance indicators, and providing cross-regional knowledge sharing. Turning these recommendations into concrete actions will require outreach to country directors and improved incentives for staff working on TA projects

2.12 Use of Adaptable Program Loans (APLs) continued to grow in both number and volume. At end FY01, there were 98 APLs totaling $3.9 billion compared with 69 totaling $2.9 billion one year earlier. A review of APLs through March 2000 found that they were being

INVESTMENT No. $B ADJUSTMENT No. $BSIL Specific Investment Loan 1,080 79.4 SAL Structural Adjustment Lending 30 5.0TAL Technical Assistance Loan 121 2.5 SAD Sector Adjustment 18 3.9APL Adaptable Program Loan 98 3.9 PRSC Poverty Reduction Support Credit 2 0.4LIL Learning and Innovation Loan 84 0.4 PSL Programatic Structural Adj Loan 2 0.2SIM Sectoral Investment and Maintenance 64 5.5 ECO Expanded Cofinancing Operation 1 0.0ERL Emergency Recovery Loan 37 2.9 DRL Debt Reduction Loan 0 0.0FIL Financial Intermediary Loan 24 1.9 RIL Rehabilitation Loan 0 0.0

1,508 96.5 53 9.5

TOTAL INVESTMENT + ADJUSTMENT 1,561 106.0


used as intended (i.e., supporting a long-term development program on a phased basis) but that costs and preparation time exceeded expectations, and quality was below average in several areas such as fiduciary, safeguards, monitoring and evaluation, triggers and development impact. These findings were supported by recent QEA4 results which showed that the quality of APLs trailed the overall results. Key issues in the design and implementation of APLs have to do with the appropriateness of their phasing, triggers for evaluating progress and exit strategies. OPCS is preparing the third retrospective of APLs which will provide the basis for a review of guidance to staff.

2.13 While the number of Learning and Innovation Loans (LILs) in the portfolio grew from 71 in FY00 to 84 in FY01, new approvals dropped from 30 in FY00 to 20 in FY01. Project preparation costs for LILs have been higher than expected (average preparation expenditure of $190,000). The elapsed time from concept to Board of 13 months, was also double the 6-month Bank target, suggesting that LILs are not the quick, low cost operations they were intended to be. During QEA4, questions also arose about operations processed as LILs that did not meet the original learning and piloting criteria. New guidelines issued in May 2001 reinforced the original concept of LILs as agile, cost effective, instruments for testing and piloting promising development solutions. A new monitoring system is ensuring that the LILs currently under preparation meet the 6-month preparation target and $100,000 preparation costs ceiling.

2.14 Several types of lending have been at least partially displaced as newer forms have been introduced. For example, use of Sector Investment and Maintenance (SIM) loans in the portfolio has steadily declined in recent years in favor of the APL, which is similar but more flexible. The Expanded Co-financing Operation (ECO) has not been used in the last five years, and there have only been three Debt Reduction loans (DRL) and four Rehabilitation loans (RIL) in that period. The mix of investment and adjustment lending for a country is largely determined through the CAS process but the choice of lending type within the two categories has been less well managed, with staff not having adequate guidance on the appropriate use of the different instruments. The rules and procedures governing the menu of instruments needs updating and clarifying. Work is underway on a revised Bank operational policy for adjustment lending. Similarly, OPCS plans to launch systematic reviews of Bank investment lending instruments.

REGIONS AND COUNTRIES

2.15 AFR has the largest number of projects in the portfolio among the six regions, accounting for 24 percent of the active portfolio in FY01(Figure 2.1). After several years of contraction, the AFR portfolio has begun to stabilize and the outlook for future growth is promising. LCR remains second in both number of projects and volume, although the lending volume has shrunk slightly faster than the Bank as a whole since FY96 due to a reduction in average investment loan size. The number of projects in ECA’s portfolio continued to grow as a result of a build-up of programs in several countries (Azerbaijan, Bosnia -Herzegovina, Georgia, Kosovo, Macedonia), but the region’s portfolio size fell 20 percent in dollar terms in FY01 due largely to a sharp fall in lending to Russia. EAP has the largest share of commitments in the active portfolio, but both the number of projects and the volume of the region’s portfolio shrank as several country portfolios were further streamlined (Indonesia, Philippines), and lending to China was constrained by portfolio exposure limits. SAR has the largest average project size, making it third in terms of commitments but fifth in number of projects. The primary factor in the decline in SAR’s portfolio since FY96 has been reduced lending to


Pakistan due to performance concerns and economic sanctions. With new lending to Pakistan in the pipeline, however, the region’s portfolio may grow in FY02. MNA’s portfolio remains the smallest among the six regions, and also experienced a decline in portfolio size and number in FY01. The lending outlook for MNA remains clouded following the September 11 events, but is not expected to show much growth as many countries in the region continue to have access to alternative sources of capital.

FIGURE 2.1 PORTFOLIO DISTRIBUTION BY REGION FY96-01

2.16 Portfolio composition has shifted over the last five years away from countries with weak policy and institutional environments4; in particular the share of the portfolio in low CPIA countries declined from 22% in FY96 to 3% in FY01. Much of this decline reflects improvement in policies and strengthening of institutions in several important borrowers (Brazil, Russia, Mozambique, Tanzania, and Yemen). Greater selectivity in lending programs, in light of the lessons of aid effectiveness studies, has also been a contributing factor in the shift. The share of the portfolio in low-income countries5 declined to 40 percent in FY01 from 48 percent in FY96. But this is entirely due to country reclassifications, with several borrowers moving into higher income groups, including China, Sri Lanka and Honduras.

2.17 The FY01 portfolio includes operations in 127 countries, with a heavy concentration in seven countries (China, India, Mexico, Argentina, Turkey, and Indonesia) which account for almost half the total portfolio in terms of net commitments. In comparison, 70 of the smallest borrowers account for only 5% of the commitments in the portfolio. Despite the large disparity in the share of net commitments, each of the two groups (the seven largest and the 70 smallest) account for about a quarter of the portfolio in terms of number of projects. This illustrates the adaptability of the Bank lending program to the diverse needs, interests and capabilities of different borrowers.

4 As measured by the Country Policy and Institutional Assessment (CPIA). Low CPIA countries are those

with a rating of 3 or less on a scale of 1 to 6. 5 Country groupings as per WDR definitions.

14 14

26 27

1715

2322

6 6

17 17

0

10

20

30

40

50

% O

F T

OT

AL

AFR EAP ECA LCR MNA SAR

By Commitments (%)

2824

181814

20 20 20

8 9 11 9

0

10

20

30

40

50

% O

F T

OT

AL

AFR EAP ECA LCR MNA SAR

By No. of Projects (%)

FY96

FY01


NETWORKS AND SECTORS

2.18 The Private Sector and Infrastructure (PSI) portfolio (551 projects, $47 billion) is the largest among the five networks, but it has shrunk disproportionately—25 percent in commitments—over the past five years. Part of the reason for the reduction was the improved availability of private funds for projects in the Electric Power, Oil and Gas, and Telecommunication sectors. The Bank’s safeguards policies may also play a role. As noted in the recent Cost of Doing Business study, while borrowers generally felt that the direct and indirect value of the Bank’s policies exceed their costs, they may be discouraging clients – concerned about possible delays in project design, launch and implementation – from borrowing for PSI investments.

2.19 The Human Development Network’s (HDN) share of the portfolio has expanded, and now accounts for 24 percent of total commitments (Figure 2.2). The growth was entirely in the Social Protection and Health sectors. The Education portfolio has been shrinking since FY96. However, new lending increased in FY01 suggesting an upward trend. HDN is taking action to build the demand for education operations (by increasing analytical work for example) as well as expanding the Bank’s role in mobilizing local and donor financing (Box 2.2). The effectiveness of these measures bears close monitoring.

FIGURE 2.2 PORTFOLIO DISTRIBUTION BY NETWORK FY96-01

19 21

74

1924

6 8

5044

0

10

20

30

40

50

% O

F T

OT

AL

ESSD FSE HDN PREM PSI

By Commitments (%)

2425

5 4

24 27

8 9

4035

0

10

20

30

40

50

60

ESSD FSE HDN PREM PSI

By No. of Projects FY96

FY01

BOX 2.2 INCREASING EDUCATION LENDING

To build the demand for Bank education operations, HDN is pursuing three complementary avenues:(i) Increasing analytical work to strengthen the capacity of sector units in the Regions to conduct

effective policy and technical dialogue with our client countries. A significant share of resources of the Education Anchor unit will be used in partnership with regional resources to restore the knowledge base of country education sectors.

(ii) Engaging the Country Units in a search for realistic strategies to accelerate progress toward meeting the Education for All (EFA) program targets, which form part of the Millennium Development Goals, in a more effective manner. HND is in the process of carrying out a country-by-country analysis of financing requirements as well as an assessment of needed policy reforms. In non-EFA countries, we are looking systematically at the potential for increased demand for education operations in the light of country needs and constraints.

(iii) Participating more actively in CASs and PRSPs to ensure that education becomes a more central dimension of the Bank’s programmatic lending.


2.20 The relative stability of the ESSD network portfolio share masks a modest increase in the Environmental portfolio in FY01 and a continued decline in the Rural Development portfolio.6 As the majority of the poor still live in rural areas, questions have been raised as to how the Bank’s investments are tracked spatially. A meaningful estimate of the Bank’s lending to rural and urban areas is not possible given the Bank’s sectoral classification. Many operations, for example adjustment loans, are national in scope in that they impact both rural and urban beneficiaries. Also the physical location of a project may in any case not be an adequate indicator of urban/rural bias. Investments in urban areas could have positive impacts on the rural economy and vice-versa. Both the rural and urban sector boards are discussing the issue of measurement and have updated their sector strategies to better reflect the synergies between urban and rural development.

2.21 The PREM portfolio has grown to 132 projects and $8.1 billion since FY96, but still represents only 8 percent of the portfolio. 7 The FSE portfolio is the smallest and has shrunk by almost half in terms of commitments since FY96, from $8.6 billion to $4.4 billion in FY01. This decline was due to the closing of several large adjustment operations made during the global financial crisis. The uneven performance of FILs, traditionally the main instrument used for FSE lending, has also been a factor.

2.22 A deeper understanding of the changes in the portfolio over time might be gained by examining shifts in the distribution of the portfolio by sub-sector. Unfortunately, this level of analysis is made difficult by weaknesses in the Bank's sector classification system. For instance, some of the fastest growing groups of projects in the portfolio are the "other" sub-sector categories within each sector's portfolio. At end-FY01, these "other" projects as a group accounted for 18 percent of the projects in the portfolio and 14 percent of commitments. In the ESSD portfolio, almost one-quarter of all projects are assigned to “other agriculture” and “other environment”. Bank management has recently launched an effort to develop a new system of sector and thematic codes. Once in place, the new classification system should lead to improved internal monitoring of Bank activities and output in relation to key corporate and Mille nnium Development goals, facilitate managerial decision-making about strategic choices and budget allocations, and enhance external reporting.

2.23 The Bank is targeting resources to selected high priority sectoral and thematic initiatives. For example:

• Fighting HIV/AIDS. The Bank has committed itself to combating the spread of HIV/AIDS around the world. In FY01, the Bank approved 9 new IDA credits, totaling $399 million, in support of the fight against HIV/AIDS in Africa (Burkina Faso, Cameroon, Eritrea, Ethiopia, Ghana, The Gambia, Kenya, Nigeria and Uganda). This funding is part of the Bank’s $500 million Multi-country HIV/AIDS Program for AFR. In SAR, a $40 million IDA credit was approved to help prevent the outbreak of an HIV/AIDS epidemic in Bangladesh. Similarly, the Bank committed $155 million of

6 To some extent, the decline in the rural development lending is a result of a reclassification of assistance

under different sectoral headings, e.g. lending for rural roads which has increased, is now in the PSI portfolio.

7 Including the fast-disbursing operations that both entered and exited the portfolio in FY01 would raise PREM’s share but only to 9 percent.


IBRD funding for the Caribbean Region Multi-country HIV/AIDS Prevention Project in FY01.

• Programmatic Lending for Poverty Reduction. To strengthen the programmatic approach to country development, the Bank introduced a new lending instrument in FY01, the Poverty Reduction Support Credit (PRSC). PRSCs are being offered to IDA-eligible countries that are seeking support in implementing policy and institutional reforms identified in the country’s Poverty Reduction Strategy Paper (PRSP). It is expected that the PRSP/PRSC framework will allow donors to combine their efforts behind a single program, with harmonized monitoring and evaluation, focusing on results at the project, program and country levels. The Board in FY01 approved two PRSCs—to Uganda and Vietnam— and 14 more are under preparation.

• Post-Conflict Reconstruction. The Bank has expanded its post-conflict lending and advice. New operations were approved in FY01 for Kosovo, Yugoslavia, West Bank and Gaza, East Timor, Ethiopia and Eritrea. The Bank is also working closely with the donor community to organize the reconstruction efforts for Afghanistan.

• Community Driven Development. The Bank intensified support for CDD in FY01, providing new loans totaling $1.4 billion. In AFR, 17 countries are receiving Bank assistance to increase funding of community groups and elected local governments to enable their greater participation in policy and institutional reform. Similar efforts are underway in East Asia (Cambodia, East Timor, Indonesia, Vietnam), Eastern Europe (Albania, Armenia, Romania), and Central Asia. The CDD portfolio also continued to grow in South Asia and Latin America.

• Improving Governance and Combating Corruption. The Bank is supporting a multi-pronged strategy for combating corruption, combining economic policy reform, public sector administration reform, legal and judicial reform, and strengthening oversight, transparency and accountability in the use of public resources. Core diagnostic work—including Public Expenditure Reviews, Country Financial Accountability Assessments, and Country Procurement Assessment Reports—is being stepped up. Community driven development projects help to hold governments accountable. Projects approved in FY01 aimed at addressing corruption include support for judicial reform in Armenia, for civil service strengthening in Bangladesh, and for tax revenue collection and expenditure management reforms in Colombia.

• Promoting Gender Equality. In FY01, the Bank completed a major Research Report on gender and development that places gender at the center of the Bank’s work on poverty reduction. Drawing on strong empirical evidence presented in the report that gender inequalities hinder development effectiveness, the Bank also prepared a new strategy to better integrate gender into lending and non-lending assistance. The strategy was endorsed by the Board in September 2001.

2.24 To meet the increasing demands for results, accountability and transparency from Bank shareholders and stakeholders, a priority agenda for the coming years is to put in place a framework for tracking and analyzing the portfolio vis -à-vis its focus on the underlying development objectives, especially its poverty orientation. Chapter VI of this report offers proposals on an initial conceptual framework and a preliminary set of indicators that might be used to assess the poverty orientation of the Bank’s portfolio.


III. PORTFOLIO PERFORMANCE

The portfolio riskiness decreased in FY01 for the fifth consecutive year and is now less than half the FY96 level. This encouraging result is reinforced by the upward trend in OED evaluations and correlates with significant improvements in the quality of the Bank’s work in project preparation and appraisal, supervision, and economic and sector work. The improvements in FY01 were concentrated in SAR and ECA, but looking back over five years they have been nearly across-the-board. At the same time, analysis suggests that the decline in riskiness may be somewhat overstated. This issue will be addressed by QAG in FY02 along with other measures undertaken to sustain improved portfolio performance.

PORTFOLIO RISKINESS

3.1 Portfolio performance has improved every year since FY96:

• Projects at risk fell to 12 percent in FY01 from 15 percent in FY00, while the percentage of commitments at risk dropped from 16 percent to 12 percent. These figures are less than half the FY96 levels of 29 percent and 31 percent, respectively, for projects and commitments at risk.

• OED outcome evaluations are up to 82 percent satisfactory for projects closed in FY01 and evaluated to date compared with 76 percent for FY00 closings, 72 percent for FY99, and 69 percent for FY96 closings.

3.2 Lending Type and Financing. Adjustment and investment lending were essentially equally risky at the end of FY01 in terms of number of operations, but adjustment was considerably less risky in volume of commitments. This pattern reflects the short life span of adjustment operations; most are still wit hin a few years of approval. It also reflects strong improvements in borrower performance in a number of countries, especially upper middle -income ones; and increased attention to borrower ownership. The IBRD and IDA portfolios are about equal in terms of riskiness. Also the riskiness of the trust fund portfolio is about the same as that of the overall Bank portfolio.

3.3 An indicative analysis of the factors behind the improvement in the portfolio since FY96 suggests that about two-thirds of the decline in pr ojects at risk has come from the reduction in the project–specific risks and about one-third due to improvement in the country context. Changes in the country and sector mix had minor effects overall. Project-specific actions by borrowers and the Bank were more important in the IBRD portfolio than in IDA’s. (Table 3.1)


TABLE 3.1 PORTFOLIO IMPROVEMENT BY SOURCE OF FINANCING /a

3.4 Regions and Countries. The reduction in portfolio risk in FY01 was concentrated in the SAR and ECA portfolios, specifically in India and Russia. The Bangladesh, Nepal and Ukraine portfolios also improved. SAR’s projects at risk dropped to 10 percent in FY01 from 23 percent the year before and commitments at risk fell even further from 26 to 9 percent (Box 3.1). Commitments at risk rose in AFR to 17 percent in FY01 from 12 percent in FY00, driven by increasing portfolio problems in Cote d’Ivoire, Senegal, Zambia and Zimbabwe. Portfolio risk also rose in MNA, which now has the highest percentage of at risk projects among the six regions. The improvement in the portfolio since FY96 has been widespread, with risk in the other five regions achieving or approaching the level only EAP had in FY96 (Figure 3.1).

FIGURE 3.1. PROJECTS AT RISK BY REGION AND NETWORK

3.5 The analysis of portfolio improvement over the last five years by region (Table 3.2) suggests that they fall into two groups: (a) ECA, MNA and SAR, where major reductions in project-specific risks account for the bulk of the overall improvement, and (b) EAP, LCR, and AFR, where improvements in project-specific risks are more modest. In AFR and LCR, changes in country factors are more important than in other regions of the Bank.

TABLE 3.2 PORTFOLIO IMPROVEMENT BY REGION *

AFR EAP ECA LCR MNA SAR BANKFY96 Portfolio at Risk (%) 35 12 35 29 36 25 29FY01 Portfolio at Risk (%) 15 9 12 12 17 10 12Net Improvement (%) 20 3 23 18 18 15 16Attributable to: Changes in Country Factors (%) 12 -1 2 8 5 1 5 Changes in Country/Sector Mix (%) 3 -1 3 3 -4 -3 2 Changes in Project Specific Risks (%) 6 5 19 6 17 17 10* Figures may not add up due to rounding.

IBRD IDA BANK

FY96 Portfolio at Risk (%) 30 28 29FY01 Portfolio at Risk (%) 13 13 12Net Improvement (%) 17 16 16Attributable to: Changes in Country Factors (%) 5 6 5 Changes in Country/Sector Mix (%) 0 3 2 Changes in Project Specific Risks (%) 12 7 10/a Figures may not add up due to rounding.

35

15 12 9

35

12

29

12

36

17

25

10

29

12

0

10

20

30

40

50

% P

roje

cts

at R

isk

AFR EAP ECA LCR MNA SAR Bank

By RegionFY96

FY01

27

10

27

2

26

13

30

14

31

14

29

12

0

10

20

30

40

50

ESSD FSE HDN PREM PSI Bank

By Network


3.6 Networks and Sectors. Compared to a year ago, portfolio risk in terms of number of projects came down in all but one network (PREM) in FY01, with the sharpest reduction in Finance (FSE). This was partly due to the close collaboration between the Networks and Regions. FSE for example asked each region’s sector manager to present a plan of action early in the year and then followed up with regular monitoring of progress. Since FY96, every network has improved its commitments at risk ratio by at least 10 percentage points.

3.7 At the sector level, a similar pattern emerged, with major sectors showing a reduction in portfolio riskiness over the past five years. Notable among the sectors were Agriculture, Oil & Gas, and Water Supply and Sanitation (WSS), which each reduced the proportions of projects and commitments at risk by more than 20 percentage points between FY96 and FY01.

3.8 The analysis of portfolio improvement over the last five years by Network suggests that they also fall into two groups: (a) FSE and PREM where major reductions in project-specific risks accounted for essentially all of their overall improvement, and (b) ESSD, HDN and PSI, where improvements in project-specific risks were more modest, and changes in country factors also contributed significantly (Table 3.3).

TABLE 3.3 PORTFOLIO IMPROVEMENT BY NETWORK

3.9 The Bank portfolio is substantially more healthy than it was five years ago and is now well positioned to deliver satisfactory outcomes in the 80-85 percent range on a sustained basis. However it is important to ensure that the methodology of risk assessment remain robust and credible so that the improvement is not overstated. (Box 3.2). QAG plans to undertake a review

BOX 3.1: TURNING PORTFOLIOS AROUND: THE SAR EXPERIENCE

In response to low ratings in the previous year for both project quality-at-entry and quality of supervision, the South Asia Region took action to reduce portfolio risks in FY01. SAR’s efforts also targeted quality at entry.

• SAR strengthened regional and country level reviews of the portfolio.

• Risk ratings and flags were shared and discussed with borrowers.

• Supervision strategies were mandated for all at risk projects.

SAR also gave more support to project teams. The number of Quality Enhancement Reviews (QERs) went up. Regular videoconferences on project and portfolio quality were introduced between HQ and regional staff in the field. Training of Bank and borrower staff in workshops and clinics held in Washington, India, Sri Lanka and Pakistan was enhanced.

ESSD FSE HDN PREM PSI BANKFY96 Portfolio at Risk (%) 27 27 26 30 31 29FY01 Portfolio at Risk (%) 10 2 13 14 14 12Net Improvement (%) 17 26 13 16 17 16Attributable to: Changes in Country Factors (%) 7 3 4 2 6 5 Changes in Country/Sector Mix (%) 1 -1 3 -5 1 2 Changes in Project Specific Risks (%) 10 24 7 19 9 10


of the portfolio monitoring system in FY02 with the aim of refining the portfolio monitoring indicators and the at risk flag system to more accurately reflect riskiness in the portfolio. At the same time, it is important that line managers review project risk ratings more closely and provide staff with incentives to accurately reflect the performance of their projects.

3.10 FY02 Portfolio Improvement Program (PIP). Launched in 1996, the PIPs have motivated managers and staff to focus on specific problems in the portfolio. Updated at least annually, the PIP includes countries, sectors and large projects that contribute disproportionately to the overall risk of the Bank’s portfolio. 8 The strong improvement in portfolio indicators makes it possible to draw up PIP lists for FY02 that are shorter than those of recent years. Six countries—Algeria, Ecuador, India, Nepal, Paraguay, and Ukraine—were removed from the list in recognition of their markedly improved portfolios while three new countries—Bolivia, Venezuela and Zambia —have been added. The carryover PIP countries are Cote d Ivoire, Kenya, Lao PDR, Lebanon, Russia, Uzbekistan, and Zimbabwe. The sectoral portfolios are all healthy enough that no sectoral PIP list is warranted for FY02, although the Education portfolio bears watching with $2.1 billion or 20 percent of its portfolio in the at risk category. Twelve large projects have been identified for the FY02 PIP, compared to 18 in FY01. 9 The targeted 8 PIP portfolios: PIP countries are those where more than 50 percent of projects and/or more than 33 percent

of commitments are at risk , with more than eight active projects and/or $250 million in commitments. PIP sectors are those with more than 50 percent of projects and 33 percent of commitments at risk with more than 20 active projects and/or two billion dollars in net commitments.

9 Argentina Buenos Aires Urban Transport, Argentina SEGBA V / Yacyreta, Brazil Federal Highway Decentralization, Brazil Mato Grosso Natural Resources, Cote d’Ivoire Transportation Sector Adjustment, India Orissa Power Sector, Lebanon Emergency Recovery, México FOVI Restructuring, Mexico Technical Education/Training, Pakistan Social Action Program II, Turkey Basic Education I, Turkey Privatization Social Support.

BOX 3.2 CREDIBILITY OF PORTFOLIO RISK ASSESSMENT

As senior management has strengthened the incentives to improve portfolio statistics over the last several years, staff and managers have worked to minimize the number of projects in the at risk category. Where they have done this by substantively addressing problems, this is all to the good. But to the extent that they may have done this by working around the risk flag system, the magnitude of the improvement in the portfolio could be exaggerated.

Results of several assessments highlight areas of concern:

• QAG assessments of supervision quality gave less than satisfactory ratings to about 30 percent of projects on quality and realism of project reporting, and note that in many Bank units, PSR reporting is seen as a formality by task managers.

• Persistence of a substantial disconnect between PSR and OED ratings. The net disconnect between PSR and OED ratings of development objectives has been 8-9 percent over the past several years. Combined with the proportion of projects rated unsatisfactory on development objectives (averaging 6-7 percent over FY00-01), this implies a portfolio risk level of about 15 percent.

• Analysis of a sample of projects with two flags showed that in a substantial proportion of cases, TTLs and their managers did not give additional risk flags to projects (thereby avoiding the at-risk category) despite evidence in the PSR strongly suggesting that the additional risk flags were warranted. If even one-fourth of the projects with two flags were put into the at risk category, the proportion of the portfolio reported at riskwould be about 16 percent instead of 12 percent.

• Review of PSRs of projects with “golden flags” (regional management override of an “at risk” designation) found that they were inconsistent with comments in the PSRs themselves about half of the time.


countries and projects constitute only about 10 percent of the total portfolio but account for about one-quarter of the total risky projects and about one-half of the total commitments at risk.

PORTFOLIO MANAGEMENT

3.11 Quality at entry is a key contributor to eventual project success, and results from the QEA4 assessment (based on a sample of 100 operations approved between January 2000 and June 2001) indicate a continued improvement in quality, with 94 percent of operations rated satisfactory or better (89 percent in QEA3). Improvements in quality were broad-based, with most regions and networks getting higher ratings than during the previous assessment. (Table 3.4). For the first time, three regions (LCR, MNA and SAR) achieved scores of 100 percent satisfactory. Noteworthy gains also occurred in FY01 in FSE and HDN, while ESSD’s performance deteriorated somewhat.

3.12 Of concern was the finding that the quality ratings were substantially worse for IDA operations than for IBRD. Also some erosion was noted in the traditional quality areas of strength (technical, financial and economic analysis). Evidence from QEA4 suggests that both the Bank and borrowers are showing risk aversion to the identification and preparation of projects that are likely to trigger resettlement policies.

3.13 Quality of supervision is increasingly important as the balance of the portfolio shifts from “blueprint” projects to those with more flexible, adaptive designs. Recognizing this, the Bank has put more resources into supervision over time and raised the profile of portfolio management in the last several years. The result has been improved ratings for assessments of the quality of supervision. In FY97, the first year this type of assessment was carried out Bank-wide, only 63 percent of the sampled projects were rated satisfactory or better on supervision. By FY00 it was 92 percent with gains across all regions and networks. Management has continued to actively support and monitor recommendations from QSA4 to minimize any backsliding on supervision quality. Given the recent high supervision quality, QAG has skipped the QSA exercise for FY01 and instead is undertaking a review of the supervision performance of a sample of at risk or borderline at risk (two flags) projects. The aim is to evaluate the effectiveness of Bank actions in managing the riskiest portion of its portfolio. Preliminary results are expected in early 2002.


TABLE 3.4 QUALITY ASSESSMENT RESULTS AND PORTFOLIO MANAGEMENT INDICES

QAE (% satisfactory)

QSA (% satisfactory)

Realism (%)

Proactivity (%)

CY97 FY01 FY97 FY00 FY97 FY01 FY97 FY01 REGION AFR 58 89 54 86 69 83 62 88 EAP 92 82 65 96 72 77 74 92 ECA 89 97 78 96 54 65 78 90 LCR 100 100 64 96 58 81 71 83 MNA 57 100 57 100 75 88 71 89 SAR 88 100 73 77 69 93 75 100

NETWORK ESSD 82 90 57 90 65 93 63 92 FSE 100 100 53 100 67 100 65 92 HDN 68 98 61 92 63 80 75 90 PREM 81 95 57 65 66 68 85 86 PSI 86 92 70 97 68 75 69 88

BANK 82 94 63 92 66 80 70 90

QAE = Quality at Entry. QSA = Quality of Supervision Assessment

3.14 The Realism Index is the ratio of the number of actual problem projects over the number at risk , which is defined in turn as the sum of the number of actual problem projects and the number of potential problem projects. Actual problem projects are those identified as such by task teams in Project Status Reports, while potential problems are those with three or more indicators (“flags”) associated with unsatisfactory outcomes but not classified as problem projects by the task teams. Bank-wide, Realism has increased in each of the last four years, from 66 percent in FY96 to 80 percent in FY01, meeting the target for the year. SAR, FSE, and ESSD are well above the average while ECA and PREM are well below.

3.15 The Proactivity Index measures the extent to which identified problem projects are addressed in a timely fashion. Syste matic tracking of this index over the past four years has helped to bring about substantial progress in reducing the incidence of projects remaining in problem status for extended periods. Proactivity rose in FY01, reaching 90 percent. SAR had the largest increase in Proactivity among the Regions between FY00 and FY01 with a jump from 76 to 100 percent, and ECA’s increase from 75 to 90 percent was also impressive. Among the Sectors, Urban, WSS, and Agriculture achieved 100 percent proactivity. PSM, PSD and ENV, on the other hand, all showed declines and ended FY01 with ratios falling short of 80 percent.

3.16 As the number of problem and at risk projects has declined, Realism and Proactivity at the regional or network level are becoming ratios of smalle r numbers and hence are more volatile. As mentioned earlier, QAG plans to review and reconfigure the at risk system in FY02.

3.17 Performance with respect to long-standing problem projects has improved considerably over the last several years. In FY01, only 14 projects had been in problem status for 24 months or more, compared with about 22 projects at the end of FY96.

3.18 Effectiveness delays. Long delays in effectiveness are associated with disproportionate implementation difficulties and poorer pr oject outcomes. Sometimes, complicated legislative approval processes of some borrowers contribute to delays. For the past several years, about 40 percent of all loan approvals have needed more than the standard four-month period to become effective. While the Bank cannot control issues pertaining to legislative approvals for example, it


often exacerbates delays by adding project conditions late in the preparation cycle, and by premature project approvals.

3.19 The more extreme cases of operations with effectiveness delays of 18 months 10 or more included 12 projects at the end of FY01, the bulk of which were in AFR and LCR (6 and 3 respectively). In spite of these delays, implementation progress is rated satisfactory in the PSRs for 8 of the 12 projects raising questions about the realism of the PSR ratings. A review of these projects showed that almost all deserved the at risk classification. It is recommended that projects with effectiveness delays beyond 18 months be rated “unsatisfactory” on project implementation by default, subject to override by regional management in cases where the task team provides adequate justification.

3.20 Closing date management. Only about a third of projects with FY01 closing dates closed on time, and half required extens ions beyond 12 months. This has been the pattern over the last five years. About 65 percent of IBRD and 70 percent of IDA projects have been extended. The number of retroactive extensions has dropped sharply since the requirement for the Managing Director’s approval was introduced in FY01. Management and staff need to be more realistic at appraisal about the time needed for project completion thereby reducing the need for extensions. It is particularly important that with a problem project, the extension be linked to concrete actions by the borrower to resolve the underlying problems.

3.21 Cancellations in FY01 amounted to $5.5 billion, a higher figure than in any of the previous five years. Essentially all of the incremental cancellations above and beyond the average levels of recent years were accounted for by ECA and EAP, which together cancelled $3.6 billion in FY01. In ECA’s case, the principal factor was the cancellation of a single large adjustment operation in Russia. In the case of EAP, the higher level of cancellations resulted from proactive portfolio management including encouraging borrowers to ask for cancellation of loans for projects that they were no longer fully committed to implement. Cancellation of loans for non-performing projects, or projects that are no longer responsive to borrower priorities, is clearly an important instrument in the Bank’s toolkit. Over the last 5-6 years, cancellation volumes have averaged about 10 percent for IDA credits and about 20 percent for IBRD loans. The IBRD figure is high enough to raise questions about the initial appraisals of borrower commitments and capacity while the difference between IDA and IBRD cancellations suggests that IDA borrowers may feel less financial incentive to cut off marginal projects.

3.22 Dropped projects while not part of the ARPP portfolio, do have a bearing on the Bank’s cost of doing business. Tighter management of the pipeline has resulted in a reduction in the number of dropped projects from 289 in FY00 to 226 in FY01, with a slight improvement in the costs of dropped projects from $22.1 million to $21.6 million. There were 94 dropped projects with more than $50,000 of accumulated costs and these accounted for 95 percent of total dropped projects costs, similar to FY00. The remaining 132 projects had average costs of about $10,000 indicating that staff and management are generally acting early to drop unviable projects. Many of the dropped projects with high costs are caused by external factors (about 50 percent) such as changes in government priorities, conflict situations in countries, and requirements that were not met at the country level. Some dropped projects are ultimately incorporated into other Bank projects or the government's own work program (about 25 percent). A new monitoring tool has been introduced by CRM in FY02 to better track and report on the reasons for dropped projects and to enable a better assessment of possible benefits from the expenditures incurred on such 10 Six months for adjustment operations and LILs.


projects. But, as the total amount being spent on dropped projects equals about 20 percent of what was spent on projects presented to the Board, QAG plans to review a sample of projects dropped over the past couple of years to strengthen accountability for pipeline management and to draw lessons for greater cost effectiveness.

PORTFOLIO OUTCOMES

3.23 The encouraging results on portfolio risk are reaffirmed by OED’s assessment of projects exiting the portfolio. OED has almost completed its evaluation of projects that exited in FY00. About 76 percent of the projects evaluated had satisfactory outcomes and the net disconnect is 9 percent. Results for FY01 are only about half complete and therefore preliminary, but they point to a further increase in the proportion of satisfactory outcomes to about 82 percent for the FY01 cohort11 with a similar net disconnect. If achieved, this would mark the highest level of satisfactory outcome since FY79.

3.24 The improvement in project outcomes over the last five years has taken place in all six regions. SAR (with the largest improvement but a relatively small portfolio) and AFR (with a strong improvement and the largest of the six regional portfolios) contributed the most to overall improvement. Adjustment outcomes account for only ten percent of OED evaluations but their outcome ratings were significantly better than those of investment projects for each year since FY96 with the exception of FY01 (FY01 data preliminary).

3.25 Projects included in the FY00 and FY01 cohorts were for the most part approved by the Board in the early to mid-1990s. Consequently, these results do not fully capture the more recent improvements in portfolio quality. Over time, the rising trend in portfolio quality is likely to translate into further improvements in project outcomes. An analysis of the portfolio prospects, using historic correlations between outcomes and key drivers (e.g., quality at entry and quality of supervision) suggests that the Bank portfolio is now well positioned to deliver successful outcomes in the 80-85 percent range on a sustained basis. As an institution dealing with difficult development challenges, the Bank must take calculated risks. A higher success rate may be attainable but only at the cost of seriously undermining the Bank’s role in promoting innovative approaches to development. Following last year’s ARPP, the Board endorsed a Bank-wide success rate of 85 percent, with scope for adjustment among regions and sectors. This ARPP proposes that this target is still appropriate.

3.26 Net Disconnect, the difference between OED project outcome ratings and staff ratings in final PSRs, is another check on realism. The preliminary FY01 Bank-wide net disconnect is about 9 percent (Table 3.5). With unsatisfactory outcomes in the present 15-20 percent range, a projects at risk ratio of 12 percent suggests that managers may be unaware of a substantial proportion of projects in trouble. Opportunities for timely remedial action may be missed. Ambiguous statements of development objectives in the PADs and overly optimistic ratings in the PSRs are behind much of the net disconnect (Box. 3.3). Among the regions, the greatest cause for concern is AFR. Specifically, its sector units –Environment and Human Development - are the outliers. AFR fully recognizes the problem and has:

• Introduced reviews of project rating upgrades to ensure that they are warranted;

11 Among the projects evaluated to date, 82 percent were rated as having had satisfactory outcomes.

Projecting on the assumption that FY00’s net disconnect of 9 percent persisted into FY01 yields an estimate of 82 percent.


• Held workshops on ICR preparation; • Planned workshops on PSR ratings focusing on realism and transparency; • Strengthened the accountability of sector managers for project status reporting and

ratings.

TABLE 3.5 NET DISCONNECT (PERCENT)

PORTFOLIO MANAGEMENT TARGETS FOR FY02 AND BEYOND

3.27 Portfolio Management Targets. The Strategic Compact set out the ambitious goal of zero defects in Bank performance. The experience of the past five years suggests that while several Regions and Sectors are already reaching this goal for some indicators, setting targets this high may entail sharply escalating costs in terms of quality assurance and ex-ante clearances. Considering that quality assessments are not an exact science and necessarily involve subjective judgments, accountability for 100 percent targets can also generate excessively defensive staff/managerial behavior. Accordingly, the recommendation is to continue to leave the targets for QEA, QSA and ESW at 90 percent and those for Realism and Proactivity at 80 and 90 percent, respectively, with the understanding that these figures are not target points but instead represent the minimum thresholds for managerial accountability. These targets would continue to be applied uniformly across regions and sectors.

FY96-01 FY00-01

FY96 14 AFR 17 16FY97 11 EAP 9 4FY98 12 ECA 8 9FY99 7 LCR 6 7FY00 9 MNA 5 8FY01* 9 SAR 8 -2*First half of the year.

Bankwide by FY


BOX 3.3: NET DISCONNECT: CAUSES AND CURES

What explains the disconnect? Forty projects that closed in FY00-01, and had a disconnect were reviewed. In 15 cases (or 38 percent), the PSR and OED agreed that achievement of DOs was marginal, but they differed as to whether the projects were marginally satisfactory or marginally unsatisfactory. In roughly half of the remaining 25 cases, the disconnect occurred between the PSR and ICR rating. That is, the task team changed its rating in the process of preparing the ICR. The main reason for this type of disconnect is the different focus of the PSR (which tracks mainly implementation progress) and the ICR (which looks at outcomes and other factors such as sustainability and institutional development, and the Bank and Borrower performance). The remaining cases involved disconnects between the ICR and OED’s evaluation. Here the most common cause of disconnect is that the ICRs allowed for circumstances external to the project in judging outcomes and sometimes rated projects on the basis of scaled-down objectives and expectation of what would happen after project closing while OED focused on DOs as originally defined or formally changed and gave less allowance for external circumstances.

How to reduce the number of disconnects? While there are bound to be some disconnects because of legitimate differences in opinion in marginal cases, several steps can be taken to reduce their numbers in the future. They include:

• Defining project objectives in PADs with clear criteria for judging satisfactory achievement and weights as needed for different objectives and components.

• Reinforcing capacities in regional quality units to assist staff in formulating realistic, measurable statements of DOs and ensuring that project designs include generation and collection of any data that may be needed for the eventual evaluation of project outcomes

• Clarifying guidelines for TMs as to the circumstances under which they should use revised project DOs as the basis for PSR ratings, when the original project DOs need to be used in judging project outcomes, and when revised DOs may be used instead.

• Unifying ICR and OED rating scales and criteria, especially in reference to the relevant DOs and sustainability.

• Reinforcing incentives for managerial review of project ratings and ensuring more consistent attention in the PSRs on prospects for outcomes, sustainability and institutional. development.


IV. Managing Quality and Risks for Results

Much was done in FY01 to strengthen portfolio management in spite of a constrained budget environment. The continuation, deepening and widening of many of these efforts will be important in FY02 and beyond. The ARPP has identified two additional areas of strategic importance in managing for results: increasing the focus on sustainability in portfolio management and clarifying and simplifying the statement of project-level development objectives. A. STATUS OF FY00 ARPP RECOMMENDATIONS 4.1 Substantial progress has been made in implementing the key recommendations of the FY00 ARPP (Box 4.1).

BOX 4.1: STATUS OF FY00 ARPP RECOMMENDATIONS

Recommendations Actions and Results Ensure adequate task budgets for LEN, ESW and SPN.

Board approved a budget increase of $55 million for FY02. Most of the increase was allocated to operational activities.

Reinforce incentives and accountability for quality.

Senior management has given quality performance appropriate weighting in managerial and staff evaluations and appointments. Senior management has also provided visible support, through periodic communications to staff emphasizing the importance of the quality agenda.

Focus attention on areas of lagging portfolio performance.

Both SAR and FSE have turned around their portfolios, and were among the higher performers in FY01. Management of project effectiveness and closing dates has been tightened. New supervision guidelines have been issued, and Management Information Systems are more user-friendly.

Strengthen Fiduciary and Safeguard aspects of supervision.

Procurement Network has helped regions review their procurement staffing needs. Together with the Procurement and FM Boards, QAG assessed the quality of the Fiduciary ESW done during Jan. 2000 to June 2001. Recommendations of this review will be implemented beginning in FY02. ESSD Network has published standards and indicators for safeguard compliance and, with the regions, is closely monitoring compliance, especially for corporate risk projects.

Refocus efforts on poverty reduction and broaden assessment instruments.

FY01 ARPP recommends a framework and indicators for assessing the poverty orientation of Bank’s portfolio. In addition, the Bank is developing a new sector classification system that should allow better tracking of key portfolio objectives, including alignment with the MDGs.

Stabilize portfolio outcome and management targets.

Bank-wide portfolio outcome target has been retained at 80 percent success rate, with some variation by region and sector. The target for realism has been stabilized at 80 percent and those for proactivity, QEA, QSA and QESW at 90 percent, leaving room for calculated risk taking.

Take a more integrated approach to the portfolio within a country programming context

Regions are enhancing their CPPR processes to give them a more strategic focus and make them more continuous with involvement of clients and other donors on priority and cross-cutting issues. Internally, the Bank is revising the OP/BP on CPPRs.


B. PORTFOLIO MANAGEMENT IN FY01 4.2 Numerous actions for strengthening the portfolio were implemented last year covering areas such as a holistic approach to portfolio management, roles of the networks and sector boards, location of work, and resources for portfolio management. Also, further improvements were made to operational guidelines, M&E, safeguard and fiduciary policies, and internal documents and reporting systems. These are discussed below. 4.3 Holistic Approach to Portfolio Management. The FY00 ARPP recommended that an enhanced approach to the present CPPR could play a major role in fostering better integration of the portfolio as a collection of complementary activities (lending and non-lending) in support of a country strategy. OPCS has recently issued a draft updated OP/BP and guidelines for Country Portfolio Reviews (the CPR, which would replace the CPPR). The CPR is viewed as a dynamic and integrative process that links the CAS with Bank-supported projects as well as other activities including grants, guarantees, and selected advisory and analytical services. In the CPR, the Bank and borrower would identify and address systemic issues, learn from implementation experience to improve both the implementation of the existing portfolio and the quality of projects entering the portfolio, reinforce borrower ownership of Bank-financed projects, and ensure the continued relevance of proje cts in the portfolio for sector and country strategies. The Bank would encourage the borrower to take responsibility for the CPR process, with the assistance of the Bank together with key ministries and implementing agencies, people affected by Bank projects and other interested stakeholders and donor agencies in the country. The draft OP/BP/guidelines are being circulated internally for comments, and will be finalized and issued in early 2002. They will reflect emerging best practice with respect to matrix management, decentralization, quality assurance, and the CDF experience.

4.4 Roles of the Networks and Sector Boards. Networks and Sector Boards are key elements of the Bank’s matrix management structure. One of their primary responsibilities is to improve the quality of sector products and processes. Since their creation in 1997, Networks and Sector Boards have evolved rapidly, developing distinct approaches and processes. In part these reflect distinctions between mainline, fiduciary and thematic boards. They also mirror historic differences between sectors in programming styles, and differences in the ways they have adjusted to the renewed demand for strategic cohesion around corporate goals. Network and Sector Board evolution also embodied tensions around functions and accountabilities relative to the country management, some of which were exacerbated by the tight budget situation of FY01. Those on the sectoral side of the matrix often felt themselves over-extended, but without the clear authority to ‘protect’ their professional accountabilities. ‘Rebalancing’ the networks’ mandates was an important part of the answer, along with the easing of some of the most pressing resource constraints. Focus and selectivity was another solution. Following an institutional stock-taking, completed in October 2000, it was agreed that Sector Boards should give special attention in terms of quality and knowledge support on a jointly defined set of high risk/high payoff tasks, ranging from specific country PRSPs, to projects, to discrete ESW tasks.

4.5 Expectations vis-à-vis the Networks/Sector Boards remain high. The message from the FY02-04 Strategic Directions exercise is that while there are many positive lessons from this still relatively new organizational approach, progress is uneven. Management is pressing for more learning and further clarification of performance and accountabilities. Sector Boards are now engaged in a more focused learning process as well as the definition of a set of mutually agreed upon indicators on core expectations. Over the coming months, QAG will pilot an assessment of the Networks contribution to enhancing operational quality including both the


governance structure of Sector Boards and the assistance that Sector Anchors may provide. This will be built around the agreed core expectations, and involve support from the learning and knowledge functional groups in the Bank. Exploring the outcomes from last year’s initiative on high-risk/high reward tasks will be a starting point for this exercise.

4.6 Location of Work. A recent task force review of decentralization concluded that the impact of locating country directors in the field has been positive. Soft evidence (client surveys, Cost of Doing Business feedback, Board discussions, and feedback from task managers) indicates that decentralization is supporting the Bank’s goals of greater responsiveness to clients and closer interaction. With respect to Bank products, a QAG assessment of FY01 ESW quality found that the performance of field-managed tasks was strong; they were superior to HQ-managed ESW in terms of dialogue and dissemination, and the likely impact. However, QEA and QSA assessments do not find a discernible difference in quality between field-managed and HQ-managed operations. Current ly, about 30 percent of Bank staff are located in field offices in 104 countries. Harder evidence will be useful when deciding on more staff movements in either direction and in line with changes in business lines/products. Cost benefit is uncertain at present. Further evaluation is needed on the impact and cost effectiveness of decentralization.

4.7 Resources for Portfolio Management. The tight budget situation in FY01 put considerable stress on management and staff responsible for core operational activities (lending, supervision, and ESW). Supervision spending declined to $163 million in FY01, compared with $190 million in FY00, a 14 percent decrease. Recognizing that this situation was not sustainable, and could undermine the recent improvements in portfolio management, the Board approved a $55 million increase in the overall budget. This increase, combined with a 4.6 percent carryover of savings from under spending in FY01, has eased budgetary pressures on operational activities in FY02. Management is closely monitoring operational budgets in FY02 to ensure that they are efficiently used and that unit budgets are not dispersed over an excessive number of new tasks (over programming).

4.8 New Operational Guidelines for Project Supervision. In July 2001, OPCS issued revised guidelines for project supervision which complement an updated operational policy statement and Bank procedures for conducting project supervision. The policy/procedure/guidelines package is intended to strengthen ongoing reforms wit hin the Bank, emphasizing delegation of authority to task managers, effective interaction of country directors, sector managers, and their staff, the role of country office staff during the project cycle, and project quality and the evolving roles of Sector Boards, FM specialists, procurement specialists, and disbursement officers. Overall the documentation package provides comprehensive and clear guidance on the Bank’s role in supporting and overseeing project implementation. One area that merits further review and possible refinement relates to the policies and practices followed in modifying or restructuring a project. At present, the rules governing restructuring are unclear. Staff do not know when they can legitimately change a project’s development objective without being inconsistent with OED’s evaluation methodology. OED and OPCS should provide guidance.


4.9 Monitoring and evaluation (M&E) has been a long-standing concern in portfolio management. Weakness in M&E makes it harder to manage risks, identify and localize problems promptly when they occur, direct resources to where they can do the most good, and transmit the lessons of experience – both positive and negative - from one project or country to another. QSA4 noted that M&E was one of the weaker areas of Bank supervision performance with about 30 percent of the QSA4 sample scoring less than satisfactory. But, M&E performance on new operations is encouraging, according to QEA4. This assessment found that the appropriateness of arrangements to monitor implementation progress - specification of clear and relevant indicators and clear assignment of responsibilities for M&E - improved to 92% satisfactory after only 77% satisfactory under QEA3. Despite this improvement, deficiencies continue to persist in arrangements for evaluating impact and measuring outcomes. In tackling this issue it is important to keep in mind that outcome evaluations are expensive. They are also often more relevant at the sector rather than the project level. Consequently, consideration should be given to combining the outcome and impact assessments of different operations in countries and across sectors where possible. In FY00, Bank management launched an M&E improvement program. As a result, the tools and lessons learned through seven pilot programs are being refined and used to support Bank staff and clients. Among the regions, AFR and ECA have developed training programs to improve M&E across sectors and operations. But progress needs to be accelerated through continued attention and incentives.

4.10 Safeguard and Fiduciary Policies. QAG assessments of implementation of safeguard and fiduciary policies indicate that compliance has increased over time. Nevertheless, the quality of Bank oversight during appraisal and supervision is less than satisfactory in about 20 percent of cases. The Cost of Doing Business task force recommended a shift in emphasis from a project-by-project focus toward a long-term, country-based capacity-building approach to accountability for safeguards and financial management. This would entail increased attention to capacity building and discussion of harmonization of safeguard and fiduciary policies among clients and donors. The Bank has strengthened its management systems for safeguard policies, with a view to improving central coordination while maintaining decentralized responsibility. Improvements in the safeguard management system include: the establishment of an integrated safeguard information system on the application of all te n safeguard policies as confirmed by a recent IAD review; increased training and clarification on the application of safeguard policies; the establishment of a Bank wide coordination body for safeguard policies; and the implementation of safeguard risk assessments conducted by ESSD and regions. Changes are also being made to the Loan Administrative Change Initiative (LACI). These changes (effective January 1, 2002) relate to the financial management assessment, periodic reporting, and the roles and responsibilities of financial management staff.

4.11 The Project Status Report (PSR) is the main instrument for portfolio review beyond the task team and has become increasingly important in recent years with the greater decentralization of staff and managers to country offices and adoption of matrix management. The PSR was enhanced in FY01 based on findings of QSA4 and an OPCS survey of country directors and sector managers. Improvements included the introduction of a new user-friendly, web-based input and display system with automatic linking and archiving of managers’ comments. This has facilitated more timely and meaningful management input into PSR preparation. OPCS plans to further fine-tune the PSR to better meet the needs of different user groups including task teams, portfolio managers, sector managers, fiduciary and safeguard compliance monitoring units.


4.12 Project Appraisal Document (PAD). QEA4 panelists rated the quality of project documentation (primarily PADs and MOPs) relatively low, at 75 percent satisfactory. Major weaknesses included failure to provide a coherent link between the main sector issues and the operations’ strategic context. They cited internal inconsistencies in the PAD, scant coverage of key issues such as institutional aspects, unit costs, implementation program, and financial and economic analyses. The erosion in the traditional areas of Bank strength in project design appraisal—technical, economic and financial—are related to the increased attention to safeguards and fiduciary concerns and reflected in their low profile in the PAD. Several task teams expressed concern about the format of the PAD. A review and improvement of the PAD format would be desirable.

4.13 Management Information Systems . Improvements continue to be made to key management information systems used to regularly monitor the portfolio. Adding several new user reports has further enhanced the Business Warehouse (BW), introduced in FY00 to give managers and staff direct access to portfolio data on a real-time basis from a single-source. Another important development in FY01 was the rollout of the Dashboard reports which provides a set of easy-to-use management reports to improve work program planning and budgeting at the unit level. The Integrated Records and Information System (IRIS), introduced in FY00 as the institutional repository for electronic documents and correspondence, has now been implemented by all six regions. This has enhanced significantly the quality and completeness of internal record keeping, but there is still room for further improvements. Among the shortcomings are the absence of a format or structure for the filing of documents in IRIS according to major steps in the project cycle (instead all documents are filed in chronological order); variability in the completeness of the project files, as filing is left to the discretion of individual staff in the regions; the inclusion of many repetitive draft documents in IRIS along with the final version; and the listing of many non-English documents without translation of even the title into English. C. PORTFOLIO MANAGEMENT IN FY02 AND BEYOND 4.14 Two areas of portfolio management are of strategic importance to future portfolio improvements: (a) managing for sustainability of project benefits, and (b) sharpening statements of project development objectives.

4.15 Managing for sustainability of project benefits. There are concerns inside and outside the Bank that gains made during project implementation are not being sustained after project completion. Less than 60 percent of the nearly one thousand projects evaluated by OED between 1997 and 2000 were rated as likely to be sustainable, with wide variations across regions and lending types. Only one third of AFR projects were rated sustainable in FY99. Although this improved to 56 percent in FY00, it was still far below the other regions. While recent OED evaluations show that the Bank is paying increasing attention to sustainability there is scope for further improvement. Staff and managers need a better understanding of what it is that OED measures when it rates sustainability.

4.16 No one disputes the importance of sustainability, but it takes time and resources to ensure that projects are designed for sustainability and to supervise projects in ways that enhance prospects for sustainability. The low level of performance appears to correspond to the perceived cost of taking effective action to improve sustainability relative to competing demands. If sustainability of project benefits were as visible a part of a unit’s output as new products and supervision of the portfolio, it would get more attention throughout the project cycle. The PSR


format should be modified to give more prominence to the issue of sustainability (see para. 4.101 for other suggested PSR improvements). 4.17 Where Bank supervision of an on-going program or series of projects continues, the Bank has the means of monitoring and intervening to enhance sustainability of projects. In cases where projects closed without direct follow-up, however, supervision usually ends. It is recommended that the ICR section (Section F) dealing with transition to regular operations be tightened and improved to include a Transition Plan for at least those project elements which are to be carried forward. The Transition Plan would state which agency would be responsible for operation and maintenance of the project elements to be continued and list indicators to be monitored during post-completion review. Regions should examine these Transition Plans and identify any which should be monitored through post-closing supervision or broader review exercises such as the CPPR.

4.18 Sharpening Project Statements of Development Objectives . The statement of development objectives (DOs) is a key element of project design potentially serving one or more purposes:

• to generate support for the project among key stakeholders; • to form the basis for implementation planning and economic analysis; • to serve as a goal toward which progress is monitored during project execution; and • to provide a benchmark against which OED rates project outcome after completion.

4.19 In practice, OED and QAG assessments have concluded that DOs are often too general to be of much analytical use. Frequently the DO statement is simply a summary of the components of projects without any clear specification of the expected or hoped for outcomes. DO statements of this type do not provide a basis for selection of particular components, or for making mid-course corrections if conditions change or implementation does not proceed as expected. Unclear and over-ambitious DO statements are also major contributors to the “disconnect” between PSR and OED ratings.

4.20 Statements of Development Objectives in project documents generally need to be made clearer, more measurable, and more focused on project outcomes. Project documents should distinguish more clearly among (a) the broad objectives motivating the project, (b) the expected value of outcomes used for planning and appraisal purposes, and (c) the minimum threshold level of attainment needed for the project to be considered successful. Many of the tools needed are already in place, including the structure for presentation of DOs in PCD/PADs, but more use could be made of them. The ECA region has recently launched efforts to help staff improve the outcome focus of their projects (Box 4.2). AFR has initiated a similar exercise. This type of training should be expanded to other regions.


BOX 4.2: ECA’S PROGRAM FOR IMPROVING PROJECT DOS

ECA has embarked on a region-wide effort to improve the design of lending operations by placing greater emphasis on incorporating a monitorable set of outcome-focused Development Objectives (DOs) into each project. As part of this initiative, the ECA Quality Team is offering assistance to TTLs to improve the strategic focus of new operations and to provide a clear vision of what constitutes success in terms of outcomes. In most cases, this involves shifting the focus from delivering a service or investment (“output”) to changes in the behavior of economic actors outside the project (‘outcomes”).

An example is the Bosnia-Herzegovina Road Management and Safety project where the original output focused DOs to “(i) improve the conditions of the road network in Bosnia and Herzegovina; (ii) improve the organization, management, efficiency, maintenance practices and financing of the country’s road administration; and (iii) improve road safety conditions” were revised by the ECA Quality Team and the TTL to become outcome focused to “improve traffic flow and reduce accident rates on the country’s road network through rehabilitation of priority road sections associated with identified poor road conditions and accident ‘black spots’. A set of annual traffic survey reports are included to track progress towards achieving the revised DOs. This focus on outcomes rather than outputs during design turns a supply-driven project into a demand-driven project.

The ECA quality team also encourages TTLs to lower their sights from broad CAS-level objectives to outcomes that the project can reasonably be expected to produce on its own. About 10 operations have received support in the first quarter of FY02. New requests for support are increasing. In addition to providing an improved basis for M&E, the region is finding that increased clarity of objectives leads to simpler project designs.


V. IMPROVING ESW QUALITY AND MANAGEMENT

Economic and Sector Work (ESW) is the Bank’s main country-based analytical and advisory business line. It provides much of the basis for country assistance strategies, lending programs, policy dialogue, and capacity-building efforts. The Strategic Directions Paper noted that the Bank has under-invested in ESW over the past few years. An attempt is now being made to rebuild country knowledge through increased ESW spending. Including ESW in this year’s ARPP is a step toward the real-time monitoring of ESW tasks needed to help further improve quality, development impact, and cost effectiveness. This chapter looks at the size and composition of ESW across country groups and Networks, discusses how quality has evolved in the last three years, and focuses on management of ESW. It proposes a ‘flag-based’ system for monitoring of ESW portfolio with a view to provide timely signals to managers on tasks at risk of low development impact.

INTRODUCTION

5.1 The core premise of this new chapter for the ARPP is that AAA is an integral part of the Bank’s programming relationship with its client-partners. It is the bridge between country strategy and the ensuing project portfolio. In the era of the Knowledge Bank it also provides the intellectual underpinning to the development relationship, including the critical policy dialogue. The work of building that partnership, notably through aid coordination activities, and now also via PRSP/CDF, is part of AAA. The message, as in Fixing ESW II, is to see the CAS as the principal vehicle for fostering the programming dialogue which should bring individual ESW activities, including the priority Core Diagnostics, into alignment with country strategic priorities.

SIZE AND COMPOSITION

5.2 ESW is part of the Bank’s Analytical and Advisory Services (AAA), aimed at generating and sharing knowledge with its clients and partners. Other important activities in this group include Non-lending Technical Assistance, Knowledge Management by thematic groups and Network anchors, research by DEC, and client training by WBI. 12 In FY01, the Bank spent $177 million in direct costs on AAA, $51 million of which was on ESW.13 Even in nominal terms, this was the lowest level of ESW spending in the last decade. As evident in Figure 5.1 the volume of AAA in dollar terms has gone down over the last two years; it peaked at $212 million in FY99, largely due to Strategic Compact resources. The composition of AAA has also changed over

12 AAA are also referred to as non-lending services. By its very nature, it is not a well-defined group. The

definition followed here is the one used in Fixing ESW reports. Within AAA, ESW and non-lending TA are the two major activities that are country-based i.e., funded directly from the country budgets. Non-lending TA, includes capacity-building activities, which are not tied with any lending projects or ESW.

13 Besides the administrative budget, Trust Funds (TF) are an important source of funding AAA. In FY01 $142 million were spent on AAA from various TF. A major part of this expenditure was on Technical Assistance; ESW received only $13 million. TF by themselves, and in conjunction with administrative budget, are used for a number of advisory services such as Infrastructure Action Program, Public-Private Infrastructure Advisory Facility (PPIAF), Energy Sector Management Assistance Program (ESMAP), and Water and Sanitation Program WSP). The products of these programs are at times referred to as ‘near-ESW’ and are not uniformly treated as ESW for statistical, accountability and quality assessment purposes.


these years; Knowledge Management activities have increased, and so has Technical Assistance. In contrast, the share of ESW has gone down over the last two years.

5.3 Each of the AAA activity serves a specific pur pose towards the Bank’s effectiveness as a knowledge institution. This chapter focuses mainly on ESW for several reasons. Along with lending, ESW is an integral part of the business plan for each country, and therefore, a direct instrument for delivering results to clients. It is the closest equivalent to lending within AAA. ESW is also the single biggest component of AAA, and has reasonably large products. The systems for ESW monitoring and reporting, though still weak, are the most developed among AAA. In future, other AAA could also be covered in similar reviews. The chapter focuses on FY00 and FY01 since comparable data for prior years is not in the system.

FIGURE 5.1: DIRECT ADMINISTRATIVE SPENDING ON ANALYTICAL AND ADVISORY ACTIVITIES

ESW PRODUCTS

5.4 In the past two years, roughly 80 percent of ESW resources were spent on discrete products and the rest on continuous, ongoing process tasks such as economic monitoring and risk assessment. ESW products are highly diverse. Apart from traditional macroeconomic and sector work in the form of reports, they include policy notes, conferences, seminars, and study tours, all aimed at promoting effective participation by clients and the private sector, and enhancing development impact. The number of ESW products delivered has fluctuated between 300-400 over the past few years. In FY01, a total of 306 ESW products were delivered to the clients. There is a certain softness to the number of delivered tasks because ESW group was not tightly defined.

5.5 The Bank has recently decided to recognize some reports as core diagnostic products essential for the Bank to do business in a country. These include Poverty Assessment, Country Economic Memorandum/Development Policy Report, Public Expenditure Review, Country Financial Accountability Assessment (CFAA), and Country Procurement Assessment Report (CPAR), the latter two being defined as Fiduciary ESW. It is planned that by the end of FY04, all gaps in the availability of these reports (i.e., more than five-years old) will be eliminated. In FY01, 54 core diagnostic reports were produced. The number is expected to grow to 90-100 reports by end FY04.

5.6 There has been a significant increase in the past two years in the number of policy notes and other non-formal products such as conferences and consultations. Policy notes are relatively smaller and narrowly focused think-pieces, which provide in-time advice to borrowers but are not

0

50

100

150

200

250

1998 1999 2000 2001

$M

Client Training (WBI)

PFP/PRSP Ass.

Aid Coord.

Tech. Assist.

Knowledge Mgt

Research (DEC)

ESW


formally released as a report. Often they do not generate new knowledge, but rather draw upon the knowledge and cross-country experience of Bank staff.

ESW BY REGIONS AND COUNTRIES

5.7 ESW spending varies widely across Regions. On average, country units in AFR spent 11 percent of country budgets on ESW in FY00 and FY01 together, compared with 19 percent in SAR, and a Bank average of 14 percent. The Regional variations are largely explained by the characteristics of the countries in the Region, such as the lending program for the country, population size of the country, policy and institutional environment (as measured by CPIA ratings), and IDA/IBRD status of the country.14 Countries with bigger lending programs need more ESW to underpin operations. Figure 5.2 bears this out. In FY00 and FY01 together, the Bank spent an average of $2.6 million on ESW for large borrowers compared with about $900,000 for medium borrowers. For countries with no lending, an average of $300,000 was spent to provide policy advice and to keep the Bank’s knowledge up-to-date.

FIGURE 5.2 ESW SPENDING BY COUNTRY LENDING VOLUMES FY00-01

5.8 Population and country performance (as proxied by CPIA) also correlate with ESW spending. For FY00 and FY01, these relationships are evident in Figure 5.3. IDA countries have lower ESW spending than IBRD even after controlling for population, lending, and country performance. This is reflected in, or perhaps driven by, the lower level of ESW in AFR compared to the rest of the Bank.

14 See Fixing ESW: Phase II

0

500

1000

1500

2000

2500

3000

No Lending Small Medium Large

N=22 N=46 N=45 N=10


ESW BY NETWORKS AND SECTORS

5.9 The PREM network accounted for around 40 percent of all ESW delivered in FY01, both in terms of numbers and costs. PREM was followed by the ESSD network which accounted for around 23 percent by cost. Between FY00 and FY01, the distribution of ESW by sectors changed significantly. The share of ESW in the Financial sector went up, largely because of Financial Sector Assessment Programs which are done jointly with the IMF. In the PREM network, macroeconomic policy work, measured in terms of total cost of delivered products, went down by 46 percent, as did the analytical work on poverty reduction. However the share of Public Sector Management increased in response to the increasing focus on Governance and related issues. Sector work in Health and Education saw a big drop in the last two years, from an already low base. The Bank plans to beef up ESW for the Education sector (para. 2.19). These patterns in ESW activity essentially reflect the patterns in aggregate demand from Country Directors, as they respond to shifts in client demands and the Bank’s strategic priorities.

ESW QUALITY

5.10 Overall quality ratings: The only indicators of ESW quality available on a Bank-wide level come from the evaluation of random samples organized by QAG on an annual basis. These reviews look at the quality of an ESW product from four perspectives: strategic relevance, internal quality, client participation and dissemination efforts, and likely impact of the task. Ratings are given for each of these dimensions and on overall quality. The overall quality of ESW has improved from about 72 percent satisfactory in FY98 to 86 percent in FY00. Preliminary results from the FY01 evaluation indicate that improvement in quality will be sustained. The recent improvement has been fairly broad-based, along all dimensions of quality

LowMid

High

Low

Mid

High

0

500

1000

1500

2000

$'00

0

Population

CPIA

FIGURE 5.3 ESW SPENDING BY COUNTRY PERFORMANCE AND POPULATION


and in all Regions. The most striking improvement has been in AFR’s ESW program, where quality has increased to the Bank average from about 50 percent three years ago.

5.11 The reviews have generated useful lessons. An important one is that Quality at Entry is a good predictor of final quality and development impact of ESW.15 There has been a steady improvement in various measures of Quality at Entry in the past two years. Specifically, the measure of Scope and Strategic Relevance has greatly improved and is now about 95 percent satisfactory. This improvement has substantially been helped by the cleaning up of the ESW pipeline in FY00 as part of the reform effort. Few tasks are now entirely supply-driven. However, there may be a need to guard against re-emergence of ESW of limited strategic relevance. To avoid recurrence of major expenditures on ESW of limited strategic relevance, the Bank may consider producing core diagnostic reports only when they are directly relevant to the Bank’s assistance strategy in the country, rather than based on a mechanistic formula.

5.12 Client participation: There are encouraging indicators that the importance of broad, systemic engagement of the client and other key stakeholders (as opposed to delivery of a polished report to a passive recipient) is being internalized. The quality of these FY01 tasks appears to be better than the rest of the tasks. Increasingly, task teams are engaging governments and other stakeholders closely in all stages of the task. About 30 percent of the sample of ESW reviewed by QAG in FY00, and 40 percent in FY01, consisted of tasks in which dialogue and dissemination efforts were significant. There is also an increasing amount of partnership with other donors, particularly visible in CFAA, where the Bank is collaborating with MDBs and donor agencie s. Along with an increasing use of policy notes, these changes suggest a higher responsiveness to client concerns and a focus on the impact of ESW.

5.13 While in terms of overall quality, ESW appears to have largely caught up with project work, the performance of supporting systems, including managerial guidance, has lagged behind seriously. The large ‘disconnect’ with Overall Quality is understood to reflect the ‘missed opportunity’ to further improve the quality. To some extent this disconnect also reflects lower efficiency in resource use.

5.14 Previous reviews had found the quality of ESW to be poorer in low-income and low-performing countries. Preliminary results for FY01 show that this pattern may not be apparent this year, possibly because of a small sample of tasks from low-CPIA countries. The issue is of enough importance to be kept under watch though. The recently formed Task Force on the Bank’s operational work in low-income and low-performing countries will likely address the issue of quality of ESW in these countries.

5.15 Quality of Fiduciary ESW. Fiduciary ESW is a recent entrant to the ESW fold. CPAR is an enhanced version of similar products done for almost a decade now, but CFAA is a new product. Following a request from the Procurement and Financial Mana gement Sector Boards, QAG recently completed a baseline-setting review of these products, covering all 30 of the CPAR/CFAA delivered between January 2000 and June 2001. The main objective of the assessment was to learn how these products could achieve the standards of mainstream ESW. To that end, the products were rated against established standards of ESW, maybe higher

15 For ESW, Quality at Entry refers to (i) clarity about the objectives and scope of the task, issues to be

addressed, intended audiences; (ii) Relevance to the country strategy; (iii) client’s interest in the task; (iii) timeliness; (iv) adequate budget, appropriate skill mix; and (v) setting up of adequate quality assurance mechanisms.


expectations than those against which these tasks were done. Only about 66 percent of the CPAR/CFAA were found to meet the standards of a good ESW.

5.16 The baseline review provided several generic lessons that can help ‘mainstream’ CPAR and CFAA into regular ESW. First, there is a need to get these products better integrated into the country program. The process has begun with increasing discussion of fiduciary ESW in CASs, but some country teams do not seem to fully appreciate their importance as components of public sector management and reform and include them mainly as prerequisites for policy-based lending. Second, there is much scope for improvement in the internal quality of these reports. A substantial number were found to be weak on assessment of risks to the Bank’s funds and operations, reducing their fiduciary value to the Bank and its shareholders. Third, these tasks should be better integrated with other public sector management work, both in terms of their scope and coverage. Fourth, processes for carrying out these tasks should be strengthened. Preparation of a concept paper and appointment of peer reviewers would help improve their internal quality. Sector Boards are preparing to improve guidance for these products based on lessons learnt from this exercise.

ESW MANAGEMENT

5.17 ESW has often received residual attention from both staff and managers. The need for better management was recognized by Fixing ESW: Phase II report which listed more systematic and timely production of ESW, strengthening monitoring of the ESW program, and improving the planning and costing of ESW tasks, as important next steps towards ‘fixing ESW’. There has been increased attention to ESW management in the last two years and progress is being made as noted below:

• Heightened attention of senior management. ESW is now being discussed at Strategic Forum, Monthly Operational Report to Senior Management, Board Discussions, and Quarterly Business and Management Reviews.

• Developed of ESW Modules in SAP and BW which helped management and staff to track ESW deliveries, cost, country coverage, and quality.

• Dropped products expenditures declined from 10 percent in FY99 to 7 percent in FY01. • Increased support and participation by Sector Boards for Core Diagnostic ESW. • Increased attention to ESW in CAS programming. • Increased accountability for ESW deliveries.

5.18 Despite the considerable progress in managing ESW, much remains to be done. The long-term challenge is to continue the progress. Some indicators of under-management of ESW are:

• Timeliness: A timely task is well synchronized with the activities it feeds into – be it a government reform program, a CAS, a lending project, or a budget document in the country. One measure of timeliness is the average difference between the planned and actual delivery dates. While BW data should be treated with caution, average slippage from plans was eight months for the Bank in FY01. 16 The gap is serious, especially since the planned date is likely the most desirable date for maximum impact. In general, the Bank’s time frame for ESW appears to be longer than most political windows of policy

16 The slippage was highest for ESSD and HDN network reports, and the lowest for PREM network reports.


reform. The time taken by the Bank, in part, is a result of the way staff are used. The TTLs are often not able to hold the teams together for a critical 3-4 weeks of report writing once the fieldwork is over. They themselves get diverted to many other tasks before the draft is ready to show to the government.

• Cost Overrun. Budget management of ESW is inadequate. A comparison of Work Program Agreement (WPA) provisions and actual costs for a sample of 100 tasks completed in FY01 and FY02 shows that in 80 percent of the sample, actual cost was higher than the WPA. As much as 16 percent of the tasks spent more than three times, and another 25 percent spent more than twice the WPA provision.

• Over-programming. Weak timeliness and control over the budget are essentially symptoms of a wider issue of over-programming. Too many tasks are being pursued at the same time. In FY00 and FY01 there were more entries than exits with a net gain of 82 tasks in the ESW pipeline.

• Monitoring and Reporting. Tools for managing ESW are improving but are not yet fully effective. The quality of information in SAP and BW is weak. There is still confusion among staff about the SAP milestones for monitoring ESW progress.17 Also the classification and categorization of ESW products has only recently been clarified; it will take some time to make changes in the system. As housekeeping improves, and reports are used more often for day-to-day management by line managers, the quality of information is expected to improve.

5.19 The following recommendations address some of the above issues:

• Service standards for timeliness should be revisited. 18

• Selectivity of ESW programming should be strengthened, guided by development priorities. A systematic move towards medium-term programming, and explicitly spelling out ESW program in CAS would help in containing over-programming.

• SAP milestones should be defined consistently for different products at the Bank level. The attempt should be to give them meaningful interpretations reflecting key operational/decision-making stages in the life-cycle of a task.

5.20 Focusing on ‘ESW-at-Risk’: QAG assessments over the past three years have found that inadequate managerial attention during execution of the task often contributed to weak quality, and in particular, to missed opportunities for greater impact. Managers were sometimes found to have intervened too late, after a substantial amount of resources had been spent and the task was effectively ‘locked-in’. Experience with management of the lending portfolio holds some useful lessons. Over the past few years, a syste m of flagging individual projects has been developed

17 As an example, LCR defines Activity Initiation as Concept Paper Review; SAR defines it as signing off on

the task by the SM and CD. Similar differences can be seen in the interpretation of Implementation Start. The key delivery milestone of ‘Delivery to Client’ is confusing for Fiduciary ESW, where even after the CD signs off on the report for sending to the country, the report is not considered complete till discussions have been held with the clients and changes incorporated into the final report

18 The existing service standard calls for “delivery to client” within five months of “main mission”. Neither concept (delivery to client; main mission) is clear when ESW is done with the active participation of clients.


(the concept of project at risk) and helped in improving lending portfolio performance. A similar set of task-specific leading indicators of final quality and impact, for the ESW program could:

• Alert managers to specific tasks that may need special attention and help them prioritize among their numerous responsibilities.

• Provide at any given time, a snapshot of how many tasks in the program are at risk of not achieving their development impact.

5.21 The main requirement of such a set of indicators would be that (i) they are easy to measure and apply, (ii) they are objective, and most importantly, (iii) they are empirically known to be good predictors of quality and hence development impact of the tasks. The following flags are proposed as examples, but the system should be developed in a participatory fashion, perhaps by a working group of experienced ESW team leaders and managers. The system could be tested on a pilot basis and mainstreamed if found worthwhile .

• No Concept Paper Review (for tasks expected to cost more than $100,000) before spending 25 percent of the expected cost.

• Delayed beyond 6 months from the planned delivery

• No decision draft in six months after the main mission (or implementation start)

• Low-CPIA country

• First of a type product (for example first CPAR in a country)

• Not delivered within two years of initiation

• Expenditure exceeds more than $500,000

5.22 Evaluation framework for ESW and other AAA: The evaluation framework for ESW is stronger than that of five years ago. 19 Apart from QAG reviews, Task Team Leaders are now expected to carry out self-assessments of ESW tasks that cost more than $50,000, in the form of an Activity Completion Summary (ACS). In addition to these task-level reviews, Regions are expected to carry out an annual retrospective of their entire ESW program, and some Regions have also conducted client surveys as part of this retrospective. Finally, there are ad hoc evaluations, such as the client surveys conducted by the Bank in FY98 which, among other topics, provided feedback on the usefulness of Bank’s analyses and studies to the clients. However the ESW evaluation framework leaves certain activities uncovered, and some components are not effective in practice.

5.23 The ACS was introduced in 1996 for self-assessment of the strategic impact of major ESW tasks. A review of a sample of 100 ACS completed in FY00 and FY01 concluded that ACS has so far failed to attract sufficient attention of task teams and managers. Almost 40 percent of the surveyed ACSs were judged unsatisfactory in that they lacked basic information including a task description, objectives, and a self-assessment of the impact. Although the ACS guidelines require a Client survey and Resident Representative’s comments for tasks above $100,000, these requirements were rarely met.

19 See Evaluation of Economic and Sector Work: A Review (CODE97-55), September 15, 1997.


5.24 There are several ways in which the evaluation framework can be strengthened:

• Unlike lending, there is no systematic impact evaluation of ESW from a borrower’s perspective. Regional retrospectives are sporadic. An impact evaluation of large ESW tasks (costing more than $250,000) by OED is worth considering.20

• QAG carries out task-by-task evaluations but does not assess the effectiveness of overall country programs or whether work done by partners was sufficiently leveraged. A pilot assessment of a country’s entire ESW program over a period in-between successive CASs is to be carried out by QAG.

• A large volume of analytical and advisory activities such as non-lending Technical Assistance, and ‘near-ESW’, consume sizable resources and are delivered to the clients, but not uniformly monitored or subject to quality assurance. Non-lending TA should be better defined and monitored more closely. An evaluation framework should be developed for trust-fund supported “near-ESW” products.

• There is a need to revisit the ACS and find out the reasons for the low compliance rates. Done properly, ACS could provide valuable feedback for improving the impact of future ESW in the country.

20 OED covers ESW in its Country Assessment Evaluations and thematic overviews of products such as

Poverty Assessments and Public Expenditure Reports, but does not evaluate ESW impact or effectiveness in a systematic task-by-task way, as it does for lending.


VI. MEASURING THE POVERTY ORIENTATION OF THE PORTFOLIO

This chapter outlines an initial conceptual framework and indicators for assessing the poverty focus of the Bank’s portfolio. The proposed framework calls for indicators at three levels: country programs, sector strategies and individual projects. Some of the needed indicators already exist. Other indicators will need fuller definition and implementation procedures to be developed before they yield useful data on poverty impact.

LESSONS LEARNED FROM THE 1990S

6.1 Poverty reduction has had an important place in the Bank’s development agenda at least as far back as the McNamara period. In the 1980s, adjustment lending took center stage. Publication of Poverty, the 1990 World Development Report, followed by a related Polic y Paper and Operational Directive in 1991 marked a major milestone. The strategy developed in these documents recognized that some targeted interventions were needed, in addition to broadly based growth, and efforts to expand the provision of basic servic es to disadvantaged groups and backwards areas. A 2000 OED evaluation of the strategy concluded that it had a significant and positive impact on the Bank's operational work on poverty in the 1990s.21 It also found that the Bank did not focus enough on monitoring and evaluating the poverty impact of its activities.

6.2 The indicator of poverty orientation most often used in the 1990s was the share of lending going to projects in the Program of Targeted Interventions (PTI). The PTI marker provides a reasonable measure of the share of targeted interventions, but is not a good proxy for the expected poverty impact of Bank assistance. In some contexts, Bank lending may be most effective when supporting the removal of constraints that are limiting overall growth or hampering an equitable distribution of the benefits of growth. For example, Bank lending for policy and institutional reform in the financial sector or for construction of infrastructure—neither of which is typically classified as PTI—may in some countries be high priority in national poverty reduction strategies. The PTI criteria are also difficult to apply consistently across countries and do not take into account what other partners are doing. Despite these shortcomings, PTI became de facto the only indicator of the poverty focus of the Bank’s portfolio in the absence of any more comprehensive measure. The 2000/2001 WDR Attacking Poverty and Board paper on Operationalizing the WDR establish a new paradigm based on promoting opportunity, facilitating empowerment, and enhancing security. The objective of this chapter is to identify practical indicators for measuring poverty orientation of the portfolio.

CONCEPTUAL FRAMEWORK 6.3 Ideally, it would be possible to quantify the effectiveness of the Bank’s interventions in terms of its contribution to poverty reduction in developing countries and to track the year to year changes in its efforts. There are two major obstacles to developing such an indicator:

• Time scale. Most activities in which the Bank is engaged have long gestation periods. Reducing infant mortality and increasing gender equality in primary school will reduce poverty, but measured over decades rather than months or years.

• Attribution. The Bank plays a supportive role to the government and often works with multiple donor partners. To keep the Bank role in perspective, it is useful to recall that

21 The Effectiveness of the World Bank’s Poverty Reduction Strategy: An Evaluation , Report 20145, OED,

May 12, 2000.


annual disbursements in low and lower-middle income countries in FY01 averaged just $2.30 per capita. Attributing impact to a single organization/donor is virtually impossible. The donor community at large has also grappled with the issue of measuring the poverty impact of their programs.

6.4 Overcoming these obstacles has not been possible within the scope of this ARPP. Instead, the more modest goal was established of proposing a conceptual framework and preliminary set of indicators for gauging the poverty orientation of the Bank’s portfolio. To do this, it is necessary to go well beyond the PTI and take a more comprehensive approach that encompasses country assistance strategies and sectoral strategies in addition to individual projects, and that take into account the multiple dimensions of poverty, including gender. 22 The proposed framework focuses on three levels of analysis: country programs; sector strategies; individual operations.

PRELIMINARY SET OF INDICATORS

6.5 Country Programs. Country programs, typically embodied in CAS documents, are the context within which the poverty orientation of projects, ESW, and the rest of the Bank’s work can best be assessed. The country program perspective is also consistent with the management and decision-making structure of the Bank. CAS reviews are currently carried out under the aegis of the Poverty Reduction Board, and reported every eighteen months in the CAS Retrospective.23 These reviews focus on five aspects: use of available diagnoses of the poverty situation; link to the country's poverty reduction strategy; linkages between Bank strategy and poverty diagnosis as well as country strategy; poverty monitoring and evaluation; and client participation in CAS preparation. There is also an overall rating. The scoring on these indicators involves an element of judgment, as the aspects assessed are inherently qualitative and complex. The questionnaire developed for this purpose has been discussed extensively with the Poverty Reduction Board and regional managers, and tested over the last two years. The 2000-01 CAS Retrospective rated just over half of the CASs satisfactory, suggesting ample scope for improvement. The rating process however did not include discussion with the task teams. A more robust methodology, including feedback from task teams, is recommended.

6.6 The lending portfolio may or may not be well aligned with the poverty reduction strategy advocated in the CAS. It is therefore useful to classify projects according to how they address poverty. The PTI did this to a limited degree. A better alternative has recently been developed and applied by the Rural Development Department (RDV) to sort projects into three categories: (a) focused actions which directly address the needs of the poor (25 percent in a sample of 262 FY99-00 projects in rural space); (b) inclusive actions which are broad-based and improve opportunities and services available to the poor (44 percent); and (c) enabling actions which support policies and the wider context for poverty reduction (31 percent).24 Some projects may 22 The Women in Development (WID) ratings are conducted at the same time as the PTI ratings, using similar

methods. WID ratings have many of the same limitations in measuring the gender focus of projects as the PTI ratings have in measuring the poverty orientation. A process should be set up to reassess the WID ratings.

23 See first CAS Retrospective, and Poverty Reduction Group. 2001. "Pro-Poor Country Assistance Strategies: A Review of FY00/01 CASs." PREM Network. World Bank, Washington, D.C. October. The results are for 54 CASs for FY98-99 and 28 CASs for FY00-01 (for FY01, first-semester CASs only).

24 See Rural Development Department. 2001. "Poverty Focus of the Portfolio." In “Rural Portfolio Review. Working Papers.” World Bank, Washington D.C. June.


not fall into any of the three categories because they have central objectives other than poverty reduction, such as global environmental protection. Classification of the entire portfolio into focused, inclusive and enabling categories would provide a basis for characterizing the Bank’s portfolio, but with the understanding that a priori no particular distribution is preferable to another in the abstract. The appropriate distribution of projects across categories in each country would be a function of the poverty strategy and analysis embodied in the CAS.

6.7 Sector Strategy Papers (SSPs). Bank-wide sector strategies, typically articulated in Sector Strategy Papers (SSPs) are based on the Bank’s global experience and commitment to the Millennium Development Goals (MDGs). They provide country teams with guidance on sector emphases and instrument selection and therefore influence, within CAS objectives and constraints, project identification and design. Evaluation of SSPs from a poverty perspective would provide an indicator of the extent to which the standards and guidelines - within which lending and non-lending activities are selected, sequenced, and designed - reflect a sound analysis of their potential impact on poverty and contribute to meeting the MDGs. Specific standards and criteria for evaluation of SSPs from a poverty perspective would cover: (a) linkages between sector diagnosis and poverty reduction, including the MDGs, (b) linkage between the diagnosis and the strategy, and (c) monitoring and evaluation.

6.8 Individual Operations. There are limitations to the usefulness of looking at individual operations. Nevertheless, it is important to evaluate them with respect to how well they fit into the overall country program and whether they are appropriately designed from a poverty-reduction perspective. Currently, the PTI is the only rating available to gauge the poverty focus of a project. The RDV has developed a methodology to evaluate investment projects. It is based on the following analysis: i) Diagnosis - nature and extent of poverty, participation in diagnosis, ownership by sta keholders, treatment of opportunity, vulnerability, access and use of social and economic institutions; ii) Strategy - extent to which project is derived from poverty diagnosis, appropriate in the context of the country and other sector interventions; linkages with CAS, ESW, other projects, lessons learned; iii) Design - extent to which the design is internally sound, adequate M&E, pro-poor arrangements for institutional development and sustainability. The ratings on diagnosis, strategy, and design should be combined into an overall rating for each project. The RDV analysis, based on a desk review of PADs for 92 approved projects during FY99-00, found that the majority of rural development PADs were not fully satisfactory in terms of poverty diagnosis and strategy. A more robust analysis would need to go beyond the PAD, for example interviews with task teams; QAG intends to pilot this analysis during FY02.

6.9 The RDV review found two recurrent problems with respect to the poverty orientation of the projects reviewed. First, many projects do not specify good measurable indicators that are related to poverty outcomes and do not discuss how the indicators will be monitored and the results used during implementation to make any needed changes. Second, institutional development and sustainability issues that may have a bearing on the impact of projects on the poor are not handled satisfactorily.

NEXT STEPS

6.10 The proposed approach outlines an initial framework for assessing the poverty orientation of the Bank's portfolio across three dimensions: country programs, sector strategies, and individual projects. This should yield a much more meaningful picture of the poverty orientation of the Bank’s portfolio than the simple PTI marker used in the past. Feasibility of early implementation was an important consideration in the choice of indicators. Over time, it should


be possible to expand on this set of preliminary indicators with others which come closer to measuring the impact of Bank activities on poverty reduction.


VII. RECOMMENDATIONS

The improvement in the performance of the Bank’s portfolio has been substantial over the past five years. The share of projects and commitments at risk have fallen significantly and the improvement has been broad-based covering practically all Regions and Networks. Among initiatives currently underway to further improve portfolio management include the development of a new system of sector and thematic coding, the preparation of a strategy of Bank assistance to low-income and low performing countries, and an assessment of recent experiences with APLs with a view towards refining the guidelines. Notwithstanding the impressive performance, further actions are necessary for the gains to be sustained, and for the Bank to better measure the poverty orientation of the portfolio and align it with key corporate and Millennium Development Goals. The key ARPP recommendations for FY02 and beyond are as follows:

• Review and refine the portfolio monitoring indicators and the at risk flag system.

• Give greater emphasis to sustainability by raising its profile as an objective throughout the project cycle. Sharpen statements of Development Objectives in project documents especially in the Project Appraisal Document.

• Develop and mainstream mechanisms to systematically monitor the Networks’ contribution to enhancing operational quality.

• Continue strengthening ESW management to bring it up to par with lending. Progressively integrate AAA into the "portfolio” for the purpose of monitoring and managing.

• Replace the PTI as the primary indicator of the poverty orientation of the Bank’s portfolio with a broader conceptual framework and set of indicators focused on (i) country programs, (ii) sector strategies, and (iii) individual operations.


ANNEX 1

GUARANTEE PROGRAM

1. Since the mainstreaming of guarantees in 1994, 18 guarantees for 17 projects (8 partial credit and 8 partial risk and 2 policy based guarantees) have been approved. Of these, 7 guarantees were issued on a stand-alone basis and 11 guarantees are associated with Bank loans to the project. Eighteen guarantees, with exposure of US$1.6 billion25 equivalent, remain in force. The Bank has not made any payments under any of the guarantees.

2. During FY01, the Bank continued to expand its guarantee program and integrate them into its comprehensive package of development-assistance instruments. New structures expanding the use of guarantees to mobilize private capital e.g. privatization were developed. The Board approved a guarantee facility to support the Coal and Forestry sector in Russia in an amount US$200 million. The facility would issue special non-commercial risk guarantee contracts to attract private loans for commercially viable projects in the coal and forestry sectors in Russia. In December 2000, the Board also approved a partial credit guarantee for US$180 million in support of the construction of the Bolivia/Brazil gas pipeline. The pipeline, now operational, opened up a Bolivia’s gas potential and expanded energy markets in Brazil. In April-May 2001, the second Policy-Based Guarantee for US$158.8 million was issued, to support a Notes issuance of US$1.0 billion by the Republic of Colombia. The guarantee leveraged Bank support by 6.3:1, and allowed the country to return to the US market at a time when this market was essentially closed to Latin American sovereigns. Using the Bank guarantee, Colombia was able substantially complete its financing needs for 2001. In addition, the second IDA partial risk guarantee for US$60.9 million (approved FY00) in support of the Haripur Power project in Bangladesh also reached financial closure in April 2001. This is the second guarantee completed under the IDA guarantee program approved by the Board in December 1998. The guarantee will support the construction of a private power plant by protecting investors against certain types of nonperformance by the government.

PROJECT PERFORMANCE UNDER GUARANTEES

3. The performance of the guarantee portfolio is regarded as satisfactory. Construction of private projects supported by PRGs has been completed on schedule. The financial issues related to the HUB power project in Pakistan were resolved and a financial restructuring of the Uch power projects in Pakistan is underway. The Azito power project in Cote d’Ivoire was commissioned as per schedule, however following the coup d’etat in November 1999 and consequential impact on the economy, the power sector in Cote d’Ivoire is under financial strain. While the payments are being made to the project company, the situation is being monitored particularly with regard to its possible impact on the IDA guarantee.

4. Supervision of guarantees requires different treatment than loans as guarantees need to be supervised until they are released, canceled, or have reached the end of their term (loans are supervised during disbursement period only). Lenders, whose debt the Bank guarantees, play a key role in supervision in private projects. Stand-alone guarantees are monitored regularly based on information contained in progress reports received from the lenders and/or borrowers. In

25 As of September 2001.


addition, specific areas needing Bank attention, such as government performance of its contractual obligations and compliance with stipulated sectoral and financial covenants, are monitored as part of the Bank's regular supervision missions. Wherever guarantees are provided along with loans, loan supervision includes two critical parameters that are monitored to ensure the success of these guarantees: (i) project performance, to ascertain to what extent projects supported by guarantees meet their expected financial and technical objectives; and (ii) monitoring of government performance vis-à-vis their contractual obligations outlined in the project documentation and covered by the guarantee.

memorandum to the executive directors subject: fy01 annual...

Documents