q s fy01/02 (qsa5) public disclosure authorized upervision...

QUALITY OF SUPERVISION IN FY01/02 (QSA5)

ANNEXES

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Administrator

26311 Vol. 2

Administrator

Administrator

Quality of Supervision in FY01/02 (QSA5) 35

ANNEX 1: LIST OF QSA5 OPERATIONS

Bangladesh Dhaka Urban TransportBangladesh Export DiversificationBangladesh Silk Development Pilot ProjectIndia AP DPIPIndia Madras Water Supply IIIndia Malaria ControlNepal Basic & Prim. Ed. IISri Lanka Mahaweli Restructuring

South Asia Region

Benin Labor Force Development China Chongqing Urban EnvironmentBenin Private Sector China Fujian II HighwayCameroon Public/Private Partnership, Growth China Qinba Mountains Poverty ReductionCameroon SAC III Indonesia Indonesia - IIDPCameroon Transport Sector Project Indonesia Kecamatan Dev FundCAR Fiscal Consolidation Credit Indonesia Sulawesi Basic Ed.Cote d'Ivoire Distance Learning - LIL Indonesia Sumatra Basic EducationEritrea Ports Indonesia WATSALGabon Pilot Com. Infra. Lao, PDR Dist Upland DevGhana Community Water II Mongolia Financial Sector Adjustment CreditGuinea Capacity Building for Service Delivery Philippines First National RoadsGuinea Village Community Support Program Philippines LGU Finance & Development ProjectKenya Nairobi Mombasa Road Philippines Mindanao Rural DevLesotho Ag. Pol & Cap Bldg Philippines Trans Grid ReinforceLesotho Highland Water IB Vietnam Banking System ModernizationMadagascar Micro Finance Vietnam Coastal Wetl/Prot DevMozambique Education Sector Strategic Program Vietnam Rural Transport IINigeria Second Primary Education ProjectSenegal Ag. Export PromotionSenegal Agr.Srcves & Prod. Orgs Argentina B.A.Urb.Tsp.Senegal Endemic Diseases Argentina Flood ProtectionSenegal Public Serv. Info-Systems Modernization Argentina Forestry/DVSenegal Urban Mobility Improvement Program Argentina Water Sector ReformSierra Leone Transport Sector Pro Bolivia Hydrocarbon Sector Social & Envir (LIL)Sierra Leone Urban Water Supply Bolivia Integrated Child DevlTanzania Agric Research (TARPII) Colombia Financial Markets DevelopmentTanzania PSAC 1 Colombia SANTAFE I (Water/Supply)Zambia Environmental Support Program Colombia Urban Infrastructure

El Salvador Education ReformGuatemala Land Fund (APL)

Albania Power Trnsm & Dist Guatemala Priv Prtcptn Infr. TAArmenia Elec Trnsm & Distr Honduras GEF Biodiversity ConservationBulgaria Trade & Trans Facil in SE Europe Paraguay Asuncion SewerageGeorgia Judicial Reform St. Lucia Poverty Reduction FundLithuania Health Venezuela Millennium Science InitiativeMoldova General Education Venezuela Pub Sec Leg&Adm ModMoldova PSD 2Romania Petrol Sector RehabRomania Schools Rehabilitation Algeria Privatization Assistance (LIL)Russian Fed. Bureau of Econ Pol Algeria Rural EmploymentRussian Fed. Cap Mrkt Dev Djibouti Intl. Road CorridorRussian Fed. Coal IAP Djibouti Public Works/Soc DevRussian Fed. Fin Inst. Egypt, Arab Rep Prvt Sect.& Ag. Dev.Russian Fed. Sust Forestry Pilot Egypt, Arab Rep Pvt. Sect Tourism Inf & EnvTurkey Commodities Mkt Dev Iran, Islamic Rep 2nd Primary Health Care & NutritionTurkey Marmara Earthquake Emer Recon Lebanon General EducationUzbekistan Enterprise Institution Building Tunisia Greater Tunis SewerageUzbekistan Fin Inst Bldg West Bank and G Elec. Sec. Inv & MgmtUzbekistan Tashkent Solid Waste West Bank and G Housing Fiannce

Middle East & North Africa Region

East Asia & Pacific RegionAfrica Region

Europe & Central Asia Region

Latin America & Caribbean Region

36 A QAG Assessment

ANNEX 2: LIST OF MAIN CONTRIBUTORS

Staff Director Arun BanerjeeModerators Arun Banerjee, P. Hari Prasad, Amnon Golan, Xavier Legrain, Saeed Rana

Logistics & Administration Brónagh Murphy, Ann Walters, Leila Ghosh, Susan Crisostomo, Vinod Ghosh

Agueh, Florent Cortes, Maria Hume, Ian Mukherji, S. Socknat, JamesAiyer, Sri-Ram Cuadra, Ernesto Iyer, P. Nawaz, Tawhid Sood, DaveAli-Khan, Mohsin Dakolia, Maria Iyer, Vijay Nyman, Kari Spencer, RichardAnson, Richard de St. Antoine, Jean Kamkwalala, J. Okidegbe, Nwanze Steeds, DavidBabbar, Suman Deininger, Klaus Knudsen, Odin Pannenbourg, Ok Stottman, WalterBajpai, Jit Del Mar Pernia, Joseph Konigshofer, F. Pellegrini, Anthony Sundberg, MarkBanerji, Bob Dethier, Jean-Jacques Legrain, Xavier Prouty, Robert Swaroop, VinayaBerryman, Sue Douglas, James Lockheed, Marlaine Rana, Saeed Swartzendruber, FredBhargouti, Shawki Drysdale, Robert Loos, Jane Rasmusson, Lars Tharakan, GeorgeBlanc, Marc Fardoust, Shahrokh Ludwig, Peter Rees, Colin Tuck, LauraBond, James Feder, Gershon Lundebye, Stein Ridker, Ron Varma, KeshavBromhead, Marjory-Anne Galli, Fabio Marino, Manuel Ringskog, Klas Wam, PerBurnett, Nick Golan, Amnon Marquez, Patricio Rosenblatt, David Westin, RichardCackler, Mark Goldberg, Joseph McCarthy, Eugene Sara, Jennifer Wilson, MarkCarruthers, Robin Halperin, Ricardo Mehta, V. Shah, Jitu Wu, Kin BingChanne, J. Hammam, Sonia Meyerman, Gerald Shetty, Y.T. Xenakis, IreneChaudhry, Y. Hari Prasad, P. Mitric, Slobodan Shimizu, Masaharu Zampaglione, GuiseppeChopra, Ram Hasan, Parvez Mohadjer, Gisu Sinha, Tray Zearley, TomChowohury, Yusuf Hayward, John Motte, Edouard Skelton, Rama Zjip, WillemChurchill, Anthony Hecht, Robert Mukherjee, Hena Skolnik, Richard

Environment Procurement M&EAbu-Haija, Ayman Graham, Douglas I. Baratz, Bernard Dhingra, Subhash Mosse, RobertoAgossou, Hugues Janeiro, Jose Mercier, Jean Roger Ginnz, Jean-Louis Sartorius, RolfAlbert-Loth, Agnes Morel, Xiomara Rahill, Bilal Larrieu, Jorge White, ElizabethBoyce, Daniel Ndiaye, Moustapha Rees, Colin Sethi, HarbakshChen, Nancy Nyaga, John Singh, Hardyal Social DevelopmentDaoud, Edward Olowo-Okere, Edward Vijayaverl, Vijay Cruz, Maria

Schwarz, Luis Harris, BruceRennie, Keith

Alexieva, Nevena (ECCU5)

Bhatnagar, Bhuvan (EASES)

Mba-Kalu, Roseleen (MNCA3)

Sennhauser, Ethel (SASRD)

Vickers, Piers (SASEI)

Balachander, Jayshree (EASHD)

Broemmel, Michael (AFTQK)

Nobakht, Hoveida (OPCPD) Sta. Ines, Noel (EAPCO)

Villaluz, Maya (EASES)

Bapna, Manish (SASRD)

Dundar, Halil (ECSHD)

Nucifora, Antonio (YPP)

Tarade, Ljiljana (ECCHR)

Wasfy, Mona (OPCIL)

Bellew, Romemary (EASHD)

Haddad, Afef (MNACS)

Pahm, Mirriam (EASES)

Tumaliwan-Belizario, Ma. Bella (EACPF)

Benavides, Livia (LCSHE) Khan, Irshad (SASRD)

Quinones, Marie-Theresa (EASRD)

Umali, Dina (SASRD)

Beschorner, Natasha (EACNQ)

Lantin, Manuel (CGIAR)

Sanchez, Pedro (SASEI)

Upadhyay, Jagdish (EASHD)

Observers

Main Contributors

Specialized ReviewersFinancial Management

Panelists


ANNEX 3. FIFTH QUALITY OF SUPERVISION ASSESSMENT (QSA5)

APPROACH PAPER (updated December 9, 2002)

1. The Quality Assurance Group (QAG) hereby announces the commencement of the Quality of Supervision Assessment (QSA5) beginning on July 31, 2002. The following outlines the objectives, scope and methodology that will be employed to carry out the assessment.

Objectives

2. The objective of the QSA5 is as follows:

• Provide senior management with real-time feedback on the overall efficiency and effectiveness of the Bank’s supervision work.

• Evaluate the Bank’s experience in identifying and resolving problems in the riskier

parts of its portfolio.

• Improve learning in key areas of supervision quality and processes. Special Areas of Focus

3. Based on the experience and conclusions of the previous assessments, QSA5 will highlight some special areas which have particular significance in improving the emphasis on outcomes of supervision and accountability. These are as follows:

• As before, QSA5 will include specialized reviews in four areas – financial management, social development, environment, and procurement. Previous assessments also identified monitoring and evaluation as another area with systemic weaknesses and it has, therefore, been decided to revisit this issue more systematically. QSA5 will include a more in-depth review of the monitoring and evaluation (M&E) aspects of supervision, using the RSA3 framework.

• Both QSA4 and the QSR identified some weaknesses in the contribution of managers

in the supervision process. These include the lack of timely guidance to supervision teams and insufficie nt focus on task team skill mix, task team continuity and mentoring. QSA5 will review the progress made in this area.

• Realism of PSRs has been a perennial issue and panelists will be asked to review the

PSR ratings of the projects in the sample and provide their own ratings and explain when there is a difference of opinion with the task team.

• Borrower ownership and capacity building are major ingredients for a projects

readiness to enter its operational phase in a sustainable fashion and achieve its development objectives. Panelists will be asked to examine the extent to which these factors are receiving adequate attention during supervision.

38 A QAG Assessment

• Borrowers (project agency/Ministry) and other key stakeholders(co-financiers, others with direct role in the project) can provide important feedback to improve the quality of Bank supervision and panels will, on a selective basis, contact them to receive client feedback on the Bank’s performance in supervision. Task teams selected for the QSA5 will be requested to provide the names, phone numbers and e-mail addresses of Borrowers/stakeholders who might be contacted.

Sample Selection and Coverage

4. QSA5 would assess Bank performance in supervision of its portfolio during FY01 and FY02. To avoid potential overlap with OED evaluations, the universe will exclude projects expected to be closed by December 31, 2002. Also, to ensure comparability of results across the sample, projects approved during FY01-02 will not be included. With these exclusions, adjustment lending is unlikely to be a part of the sampling universe. The resultant total universe from which the sample would be drawn would be about 950 projects. QSA5 will sample 100 of these projects. To reduce the burden on task managers of multiple assessments, QAG will excuse task managers who have been assessed during the Quality at Entry Assessment for FY02 (QEA5), the ESW assessment for FY02 and the QSR.

5. In order to be able to draw conclusions about the differences in the quality of supervision efforts between the relatively healthier and riskier parts of the portfolio, the sample of 100 projects will be drawn on a random basis from three strata: 35 projects from the problem projects stratum, 35 projects from those with two or more flags, and 30 from those with one or no flag (as of July 1, 2002). QSA5 results will be statistically robust at the Bank-wide level with less than 5% margin of error at 95% confidence level. The margin of error for the three strata will be about 10% at 90% confidence level. However, due to the limitation in sample size, conclusions at the regional or network level will not be sufficiently robust.

Methodology 6. First Stage Assessments. The assessment methodology remains unchanged. Except for areas of special emphasis, for which the guidance questionnaire is being suitably modified, the major determinants of supervision quality as measured by QSA5 would be:

• Focus on development impact. How well is supervision assessing project impact and what is the relevance, timeliness and effectiveness of Bank responses to issues?

• Attention to fiduciary and safeguard aspects. How adequately is the Bank supervising financial management, environment and social aspects, performance and progress monitoring, and procurement and disbursements?

• Appropriateness of supervision inputs and processes. What is the quality of the Bank’s supervision missions, relations with other partners, and management oversight?

• Realism of project ratings and reporting . How realistic and candid are project ratings, including ratings for achieving development objectives and justifications for upgrading project status?

7. For the selected projects, QAG will extract from the Bank’s information systems copies of the SAR/PAD, legal documents, supervision budgets (planned and actual), supervision TORs, PSRs, BTOR, aide-memoires, follow-up letters and status of covenants. However, the

Quality of Supervision in in FY01/02 (QSA5) 39

task manager/task team will retain the responsibility for ensuring that packages obtained from these systems are complete.

8. Panels for the First Stage Assessments will work in two or three person teams, where the third panelist will normally be from outside the Bank. Specialized experts in financial management, environment, social development, procurement, and monitoring and evaluation will assist the panels.

9. The QAG panelists will review the written documentation and rate the operations using the Guidance Questionnaire as the basic framework. The ratings would be based on a four point scale : highly satisfactory, satisfactory, marginal, and unsatisfactory. In using the questionnaire, panelists would be expected to explain the basis for a rating of either best practice or less than satisfactory. Specialized reviewers will contact the responsible members of the task teams in their areas of interest, if necessary, to seek clarification. Following the desk review, the panelists would prepare a list of questions to be discussed during the task team interview. The immediate supervisors and Country Directors may be interviewed if considered appropriate. If the project is supervised from a country office, interviews by teleconference or videoconference can be arranged. Following the interview, QAG will provide the task team and immediate managers with a copy of the completed questionnaire summarizing the rationale for its rating and requesting their comments on the assessment. If the assessment is satisfactory, QAG will finalize it taking into account the comments received from the task team and/or its manager. If the assessment is less than satisfactory, the region would have the option of requesting a second stage in-depth review, on the basis of which QAG would prepare the final assessment.

10. Second Stage Review. Whenever a region requests a second stage review, a new panel will be established to take a fresh look at the project. While the new panel would have access to the first stage report, it will conduct its own comprehensive review and interview managers and staff with knowledge about the country and the sector. As needed, the panel might ask for information additional to what was furnished for the first stage review and may have more than one meeting with the Task Manager or particular task team member. At the conclusion of the review, panels will revise the guidance questionnaire to reflect their ratings, and will provide a report giving details of their findings. The region would have the option to add a rejoinder.

Reporting

11. A synthesis report summarizing the assessments and providing portfolio level information and analysis of factors underlying supervision efficiency and effectiveness will be prepared by end-November, 2002. From the institutional perspective, the report will synthesize the findings on generic issues, compare them with results of previous assessments, revisit recommendations of the QSA4 and QSR reports, and assess progress made to date to improve supervision quality. The QSA5 draft report will be discussed with the regions and networks before being submitted to senior management by mid-January, 2003.

Costs 12. Estimated cost of QSA5 will be about $750,000. These costs do not include approximately 4 staff days per project for task teams assessed in QSA5, which are being funded at the regional level. The budget also assumes that about half of the time of the specialist reviewers will be funded by their respective Sector Boards.

40 A QAG Assessment

Schedule

13. The schedule of key events of QSA5 is as follows:

July 31 QSA5 Announcement to managers/Country Offices August 1 QSA5 sample selection

August 2 Announcement to task managers/teams included in QSA5 sample

August 15-31 At regional request, QAG will provide orientation sessions for

task managers/teams whose projects are included in the QSA5 sample

August 16 Deadline for submission of materials for the briefing books when TL is at HQ (staff on leave may request additional time)

August 20 Orientation for Specialized Reviewers

August 22 Orientation for panelists

August 23 Deadline for submission of materials for the briefing books when TL in the field (staff on leave may request additional time)

August 26 Specialized Reviewers begin reviews

Sept. 3 – Oct. 31 QAG Panels conduct reviews for QSA5 sample

January 17 First working draft of QSA5 synthesis report

February 14 Draft QSA5 report sent to regions/networks for comment

February 28 Final QSA5 report sent to MDs, regions and networks

ANNEX 4. DETAILED RATING TABLES

RESULTS BY REGION A. Overall Assessment by Region

B. Quality dimension by Region – Highly Satisfactory and Satisfactory (%)

OA = Overall Assessment

R1 = Focus on Development Effectiveness

R2 = Supervision of Fiduciary/Safeguard Aspects

R3 = Adequacy of Supervision Inputs and Processes

R4 = Realism of Project Performance Ratings

Total project (#) HS (%) S (%) M (%) U (%)RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5

AFR 52 30 28 8 13 11 46 70 71 40 17 18 6 0 0EAP 37 24 17 14 17 0 51 79 94 30 4 4 5 0 2ECA 37 25 19 16 12 27 62 84 59 22 4 14 0 0 0LCR 50 26 17 8 12 13 56 85 85 30 4 2 6 0 0MNA 37 23 11 8 17 6 49 83 84 27 0 10 16 0 0SAR 37 22 8 8 18 30 65 59 63 24 23 7 3 0 0Bank-wide 250 150 100 10 14 14 53 78 76 32 8 10 5 0 0.4

Region

OA R1 R2 R3 R4RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5

AFR 54 83 82 58 87 88 79 90 85 63 77 84 50 53 32EAP 65 96 94 68 96 98 86 88 90 57 92 94 62 83 65ECA 78 96 86 78 100 86 86 76 98 81 100 100 62 72 67LCR 64 96 98 68 100 98 86 92 82 62 100 98 68 65 45MNA 57 100 90 65 96 90 62 96 96 56 100 90 73 74 38SAR 73 77 93 68 77 93 81 86 100 76 82 100 68 73 93Bank-wide 63 92 90 66 94 92 82 87 90 65 91 94 61 69 53

Region

HS = Highly Satisfactory M = Marginal

S = Satisfactory U = Unsatisfactory

Quality of Supervision in F

Y01/02

41

RESULTS BY NETWORK

A. Overall Assessment by Network

B. Quality Dimension by Network – Highly Satisfactory and Satisfactory (%)








42

A

QA

G A

ssessment


ESD 61 43 22 8 14 2 45 75 91 42 11 5 5 0 2FSE 12 6 6 24 40 0 39 60 92 20 0 8 17 0 0HDN 59 38 19 11 15 27 49 77 57 36 8 17 3 0 0PREM 15 7 8 7 0 28 53 71 60 32 29 11 7 0 0PSI 103 56 45 8 13 13 61 82 77 25 4 9 5 0 0Bank-wide 250 150 100 10 14 14 53 78 76 32 8 10 5 0 0.4

Network


ESD 53 89 94 56 92 98 78 90 85 54 94 96 52 51 46FSE 63 100 92 58 100 92 71 60 100 70 100 100 45 100 78HDN 60 92 83 61 90 83 87 89 100 71 89 93 76 68 56PREM 60 71 89 75 86 89 80 71 89 58 71 89 60 71 78PSI 70 96 91 72 98 93 82 90 88 67 93 93 59 78 50Bank-wide 63 92 90 66 94 92 82 87 90 65 91 94 61 69 53

Network


RESULTS BY SECTOR

A. Overall Assessment by Sector

B. Quality Dimension by Sector – Highly Satisfactory and Satisfactory (%)







ED 27 17 13 15 12 44 47 69 50 36 19 6 2 0 0EMT 29 13 7 10 30 0 48 70 100 27 0 0 14 0 0ENV 11 17 5 5 19 0 42 69 81 47 12 10 5 0 9EP 1 2 2 0 0 83 100 50 17 0 50 0 0 0 0FSP 12 6 6 24 40 0 39 60 92 20 0 8 17 0 0GIC 5 1 2 22 0 0 29 100 100 49 0 0 0 0 0HE 25 12 4 7 5 0 50 95 49 37 0 51 6 0 0PS 14 5 6 8 0 0 50 80 83 34 20 17 8 0 0PSD 9 10 8 0 24 27 64 71 51 36 6 22 0 0 0RDV 49 25 17 9 11 3 46 79 94 40 11 4 5 0 0SDV 1 1 0 0 0 - 0 100 - 100 0 - 0 0 -SP 7 9 2 10 33 0 55 67 100 35 0 0 0 0 0TR 28 16 12 3 7 21 63 86 67 30 7 12 3 0 0UD 16 10 6 11 0 14 76 100 86 13 0 0 0 0 0WSS 16 6 10 15 0 6 78 90 85 7 10 9 0 0 0Bank-wide 250 150 100 10 14 14 53 78 76 32 8 10 5 0 0.4

Sector


ED 62 81 94 64 81 94 90 81 100 73 73 89 82 77 92EMT 59 100 100 61 100 100 68 87 100 52 83 100 57 65 10ENV 47 88 81 47 94 91 88 88 81 54 94 91 28 56 17EP 100 50 100 100 50 100 100 50 100 100 50 100 100 50 83FSP 63 100 92 58 100 92 71 60 100 70 100 100 45 100 78GIC 51 100 100 51 100 100 51 100 37 78 100 100 78 100 63HE 57 100 49 57 95 49 83 100 100 65 100 100 72 52 0PS 58 80 83 73 100 83 79 80 83 56 80 83 58 80 75PSD 64 94 78 64 94 84 82 100 73 55 88 78 49 94 78RDV 55 89 96 60 89 100 76 92 86 56 95 96 57 45 52SDV 0 100 - 0 100 - 100 100 - 0 100 - 0 100 -SP 65 100 100 65 100 100 86 87 100 86 100 100 65 73 0TR 66 93 88 73 100 88 90 89 95 82 100 90 47 68 73UD 87 100 100 90 100 100 87 100 91 68 100 100 64 82 43WSS 93 90 90 88 90 97 100 60 85 77 90 97 85 100 32Bank-wide 63 92 90 66 94 92 82 87 90 65 91 94 61 69 53

Sector



RESULTS BY “AT RISK” CATEGORY*

A. Overall Assessment by “At Risk” Category

B. Quality Dimension by “At Risk” Category – Highly Satisfactory and Satisfactory (%)









Non-Risky 2/ 179 129 78 13 13 16 57 82 76 26 5 8 4 0 0.5

Risky 3/ 71 21 22 2 23 3 43 49 77 47 29 21 9 0 0Bankwide 250 150 100 10 14 14 53 78 76 32 8 10 5 0 0.4

At Risk Status


Non-Risky 2/ 70 95 92 72 95 94 86 91 91 70 93 95 62 68 56

Risky 3/ 45 71 79 48 83 85 70 63 84 52 83 84 58 71 40Bankwide 63 92 90 66 94 92 82 87 90 65 91 94 61 69 53

At Risk Status

Notes:1/ The PSR risk rating for RSA1 was taken at the end of FY97, for QSA4 at the beginning of FY00,

and for QSA5 at the end of FY01.2/ Non-risky ratings include low risk, 2 flags, and golden flags.3/ Risky group includes actual and potential problem projects.

44

A

QA

G A

ssessment

RESULTS BY PROJECT AGE*

B. Overall Assessment by Project Age

B. Quality Dimension by Project Age – Highly Satisfactory and Satisfactory (%)






* Project Age calculated from Board date as follows: for RSA1 - to end of FY97; for QSA4 - to end of FY00; and, for QSA5 - to end of FY02.


<=3 yrs 121 53 33 13 15 16 55 73 68 28 13 15 4 0 13-5 yrs 72 48 37 5 14 20 54 80 76 36 6 4 5 0 0>5 yrs 57 49 30 11 13 3 46 81 87 32 6 10 10 0 0Bank-wide 250 150 100 10 14 14 53 78 76 32 8 10 5 0 0.4

Project Age


<=3 yrs 68 88 84 69 90 85 87 89 94 68 90 88 63 70 543-5 yrs 59 94 96 61 96 96 77 88 89 67 92 97 62 75 54>5 yrs 58 94 90 64 95 96 78 86 88 56 93 96 56 60 58Bank-wide 63 92 90 66 94 92 82 87 90 65 91 94 61 69 53

Project Age




Y01/Y02

45

RESULTS BY LOAN SIZE

A. Overall Assessment by Loan Size

B. Quality Dimension by Loan Size – Highly Satisfactory and Satisfactory (%)







<= $20 M 63 47 31 7 16 9 53 75 82 33 9 9 7 0 0$20-50 M 65 41 30 9 10 9 47 85 74 41 4 16 3 0 1>$50 M 122 62 39 12 15 22 57 75 73 25 10 6 6 0 0Bank-wide 250 150 100 10 14 14 53 78 76 32 8 10 5 0 0.4

Loan Size


<= $20 M 60 91 91 64 93 93 76 86 95 63 95 91 49 69 43$20-50 M 56 96 83 62 97 86 84 87 77 60 91 93 63 71 50>$50 M 69 90 94 69 92 96 84 89 96 69 88 97 67 67 65Bank-wide 63 92 90 66 94 92 82 87 90 65 91 94 61 69 53

Loan Size



46

A

QA

G A

ssessment

RESULTS BY SOURCE OF FUNDS

A. Overall Assessment by Source of Funds

B. Quality Dimension by Source of Funds – Highly Satisfactory and Satisfactory (%)







IBRD 136 60 51 10 17 16 57 81 76 29 2 8 4 0 0IDA 114 70 46 10 10 13 48 79 73 34 10 13 7 0 1GEF 0 9 1 - 24 0 - 53 100 - 24 0 - 0 0Other 0 11 2 - 18 0 - 64 100 - 18 0 - 0 0Bank-wide 250 150 100 10 14 14 53 78 76 32 8 10 5 0 0.4

Source of Funds


IBRD 68 98 92 71 100 93 81 86 89 65 100 97 68 81 59IDA 58 90 86 60 91 90 82 92 90 65 85 88 54 60 54GEF - 76 100 - 88 100 - 76 100 - 88 100 - 65 0Other - 82 100 - 91 100 - 88 100 - 91 100 - 58 0Bank-wide 63 92 90 66 94 92 82 87 90 65 91 94 61 69 53

Source of Funds




Y01/02

47

RESULTS BY LOCATION OF TASK LEADERS (TL)

A. Overall Assessment by TL Location

B. Quality Dimension by TL Location – Highly Satisfactory and Satisfactory (%)









Field-based 34 31 20 17 16 18 47 76 62 36 8 21 0 0 0HQ-based 216 119 80 9 14 13 54 78 79 31 8 7 6 0 0.5Bank-wide 250 150 100 10 14 14 53 78 76 32 8 10 5 0 0.4

Location of TTL


Field-based 64 92 79 70 92 79 83 96 100 75 98 92 63 73 73HQ-based 63 92 92 65 94 95 81 85 88 63 90 94 60 68 49Bank-wide 63 92 90 66 94 92 82 87 90 65 91 94 61 69 53

Location of TTL

48

A

QA

G A

ssessment


ANNEX 5. GUIDANCE QUESTIONNAIRE RESULTS FOR RSA/QSA

SUMMARY ASSESSMENT SHEET

Supervision Assessment 1. Focus on Development Impact 2. Supervision of Fiduciary/Safeguard Aspects 3. Adequacy of Supervision Inputs and Processes 4. Quality and Realism of Reporting

Overall Assessment List the two or three factors critical to the success of this project: The numbers in the boxes represent the percentage of projects rated satisfactory or better. The overall assessment is not an average of the assessments of the constituent elements of supervision. Instead, the reviewer should use his/her judgment in weighing the relative importance of each given the country and the project context. In making its assessment, the reviewer should consider the importance of each category, and within each category, the various questions, to supervision quality.

Assessment Rating

1=Highly Satisfactory 2=Satisfactory 3=Marginally Satisfactory 4=Unsatisfactory NA=Not Applicable

RSA1 QSA4 QSA5

66 94 92

82 87 90

94 65 91

61 69 53

63 92 90

50 A QAG Assessment

Context

A. With the experience implementing and supervising the project during the past year, as far as can be readily determined:

RSA1 QSA4 QSA5

(a) Was the project design sound? (Yes/No) 68 80 70

(b) Was the project ready for implementation at approval? (Yes/No) 60 71 57

(c) Was overall implementation performance satisfactory prior to FY01 (Yes/No) N/A N/A 53 (d) Could the problems encountered during implementation have been i dentified

at entry (Yes/No) N/A N/A 74

(e) Any major changes prior to FY01 (through restructuring and/or MTR)? (Yes/No)

N/A N/A 22

B. Compliance with safeguard policies?

(a) Applicability and compliance with safeguard issues:

Policy Applicability

Environmental Assessment (OD 4.01) 73 Natural Habitats (OP 4.04) 9 Forestry (OP 4.36) 8 Pest Management (OP 4.09) 8 Cultural Property (OPN 11.03) 5 Indigenous Peoples (OD 4.20) 8 Involuntary Resettlement (OP 4.30) 13 Safety of Dams (OP 4.37) 4 Projects on International Waterways (OP 7.50) 1

Projects in Disputed Areas (OP 7.60) 1

(b) In case of compliance issues, including those occurring prior to FY01, what were there nature

and extent? (Count)

(c) Were appropriate measure taken to mitigate safeguard aspects that could have had adverse

impacts? (Yes/No)

1= Highly Satisfactory 2= Satisfactory 3= Marginally Satisfactory 4= Unsatisfactory

NA=Not Applicable

14

76


1. Assessment – Focus on Development Impact

With the experience of supervising the project during FY01-02, assess the quality of Bank performance for:

RSA1 QSA4 QSA5

1.1 Identification and Assessment of Problems

69 93 92

(a) Timely Identification of Implementation Problems

80 97 92

(b) Regard for Development Impact

64 85 86

1.2 Focus on Sustainability

N/A 89 93

(a) Borrower and stakeholder ownership

N/A 90 91

(b) Technical assistance, training, and capacity building

N/A 85 91

(c) Readiness for operational phase

N/A 84 82

1.3 Actions Taken and Follow-up

64 91 82

(a) Appropriateness of Advice and Proposed Solutions to the Borrower (including Action Plan)

74 94 91

(b) Appropriateness and Speed of Bank follow-up (including cancellations, suspensions, etc)

69 85 73

(c) Impact and Effectiveness of Bank actions (including risk management)

58 83 79

(d) Quality of Mid-term Review (if undertaken during FY01-02)

N/A N/A 84

(e) Quality of Restructuring Plan (if undertaken during FY01-02)

N/A N/A 77

1.4 Effective use of CPPR (or other venues for portfolio review with the borrower) to resolve problems affecting the project?

61 84 84

Assessment Rating

1= Highly Satisfactory 2= Satisfactory 3= Marginally Satisfactory 4= Unsatisfactory NA=Not Applicable

52 A QAG Assessment

2. Assessment – Supervision of Fiduciary Aspects

With the experience of supervising the project during FY01-02, assess the adequacy of supervision of:

RSA1 QSA4 QSA5

2.1 Procurement Compliance and promptness

88 92 91

2.2 Financial Management * Accounting/auditing and financial information, SOE

77 84 78

2.3 Legal Aspects Compliance; relevance/clarity/timeliness of advice

82 90 92

2.4 Environmental Aspects e.g. Category” A” compliance

N/A 85 72

2.5 Social Aspects e.g. Social impact

68 69 82

2.6 Performance and Progress Monitoring use of key indicators, progress reporting, attention to results

N/A 79 65

* Includes Disbursement

Assessment Rating


NA=Not Applicable


3. Assessment – Adequacy of Supervision Inputs and Processes

With the experience of supervising the project during FY01-02, assess the quality of Bank inputs and processes:

RSA1

QSA4

QSA5

3.1 Staffing

81 93 97

(a) Staff Continuity 92 94 93

(b) Supervision Skill Mix 72 80 88

(c) Degree of Country Office Involvement and its Contribution 72 91 93

3.2 Supervision Activities

75 92 96

(a) Mission Preparation/TORs Frequency and Time Spent in the Field 77 95 90

(b) Quality of Interaction with Borrower outside of formal Missions? N/A N/A 97

3.3 Quality of Supervision Documentation and Follow-up N/A N/A 78

(a) Aide-memoire and Follow-up Letters (well organized, focused on key issues and solutions, readable)

73 87 87

(b) Quality of supervision documentation received by the panel N/A N/A 78 3.4 Relationships

85 97 93

(a) Relations with the Borrower 82 95 92

(b) Relations with Donors and other stakeholders 84 98 97

3.5 Management Inputs

62 86 79

(a) Adequacy and Speed of Management Attention and Actions 60 79 79

i) Mission’s strategic focus and problem solving N/A N/A 81

ii) Mission’s debriefing N/A N/A 92

iii) PSR and post-mission follow-up N/A N/A 59 (b) Adequacy of Supervision Budget (Too Little, To Much or About Enough) 67 74 80 (c) Effectiveness of Budget Use N/A 89 92

Assessment Rating

1= Highly Satisfactory 2= Satisfactory 3= Marginally Satisfactory 4= Unsatisfactory NA=Not Applicable

54 A QAG Assessment

4. Assessment - Realism and Quality of Project Performance (FY01-02)

RSA1 QSA4 QSA5

4.1 Accuracy, Timeliness and consistency with Subsidiary Ratings in PSR

61 69 50

4.2

Appropriateness of Risk Ratings in PSR N/A N/A 62

4.3

Adequate Justification for any change of Ratings 62 79 70

4.4

Adequate Explanation of DO and IP Ratings 55 64 52

4.5 Appropriate Use of Golden Flags, if applicable

N/A N/A 56

Assessment Rating


NA=Not Applicable


ANNEX 6. SUPERVISION OF POVERTY AND SOCIAL ASPECTS Overview 1. Social developme nt (SD) issues in the sample of 100 operations in QSA5 were assessed by specialized social reviewers. SD issues were found to be relevant in 86 of the 100 projects. The key areas in which social issues were assessed were:

• Extent to which SD and poverty issues have been monitored during supervision;

• Extent to which supervision has assessed and taken account of views of key stakeholders or attempted to strengthen mechanisms for participation in implementation and/or monitoring;

• Where relevant, compliance with Bank social safeguards policies concerning cultural property, indigenous peoples and resettlement and land acquisition;

• Effectiveness of supervision of SD and poverty issues; and,

• Level of effort devoted to supervision of SD and poverty issues.

Sample Profile 2. Of the 100 operations in the QSA5 sample,

• 69% of operations took SD issues into account during preparation (64% in

QSA4);

• Projects with components that explicitly address gender issues rose from 34% in QSA4 to 44% in QSA5; and,

• 30% were supervised by social development specialists, same as QSA4. Overall Results 3. Of the 86 projects from the QSA5 sample in which SD issues were considered relevant, 82% were rated satisfactory. However, this performance masks substantial differences in the performance of supervision by sub-categories. Monitoring and evaluation (M&E) of SD issues, risk and outcomes continues to be an area of weakness, with 55% of operations in QSA5 rated satisfactory on this dimension, a slight increase on the 49% rating in QSA4. The level of effort and participation, with satisfactory ratings of 79% are about the same levels for QSA4.

56 A QAG Assessment

4. The below chart illustrates trends for overall ratings and on key sub-questions over the past three supervision reviews.

0.00%

20.00%

40.00%

60.00%

80.00%

SDV / POV Effort Effectiveness SD Monitoring Participation

Satisfactory percentage by social sub-ratings

RSA3

QSA4

QSA5

Supervision of social safeguards

5. In QSA5 the number of operations in which safeguards issues were relevant dropped in comparison to QSA4. Safeguards policies applied to 22% of the operations, as opposed to 25% in QSA4. By category, 12% of operations had indigenous peoples issues, 15% had resettlement issues and 6% had cultural property issues (with overlap on a number of operations). Of these, all those operations with indigenous peoples issues and cultural property issues were rated satisfactory with regard to addressing safeguards issues; 81% with resettlement issues were rated satisfactory. Though the overall proportion of operations with safeguards issues declined slightly, the proportion rated satisfactory on those issues increased for both indigenous peoples and cultural property, and remained essentially stable for resettlement.

0%

20%

40%

60%

80%

100%

Indigenous Peoples Resettlement Cultural Property Overall Safeguards

Quality of Social Safeguards Supervision

RSA3 QSA4 QSA5


Performance by Region 6. While the performance by regions in QSA4 had reached general parity, the results of QSA5 indicate that two regions have continued to improve (notably EAP and SAR), two have remained generally at the same level as QSA4 (LCR and MNA) and the remaining two have dropped off (AFR and ECA). These figures should, however, be treated with caution, as in several cases the number of projects per region are too few for statistical significance. Rating by Networks 7. Most networks have remained fairly consistent on overall ratings from QSA4 to QSA5. For both FSE and PREM the number of operations was statistically too small to draw robust conclusions (only 2 FSE projects and 5 PREM operations). PSI operations accounted for 46% of the total, and this network maintained the level of QSA4. The following chart illustrates trends by network over the past three reviews.

Performance on Gender Dimensions 8. Operations in the QSA5 sample performed better on gender issues than operations in past supervision reviews. Of the 100 projects in QSA5, 42% had some form of gender analysis (22% in QSA4), 44% of operations had project components that addressed gender issues (34% in QSA4), and gender issues were supervised in 36% of the operations (21% in QSA4). It appears that gender considerations are increasing in operations, and importantly, the gap between identification of gender issues and supervision of those issues is narrowing. In QSA4, one-third of the operations which identified gender issues as significant did not supervise them. 9. The correlation between supervision of gender issues and poverty/SD ratings continues to be high. For example, of the 41 operations (out of 100) that identified gender issues and supervised them, 95% received satisfactory ratings on supervision of social development and poverty issues, compared to 75% satisfactory rating for projects without gender components.

Satisfactory Percentage by Networks

0%

20%

40%

60%

80%

100%

120%

ESSD HDN PSI PREM FSE Overall

RSA3 QSA4 QSA5

58 A QAG Assessment

Lessons: Elements of Supervision Good Practice

10. Projects rated highly satisfactory tend to focus on achievement of social development outcomes. Those operations which identify social development, poverty reduction, and/or gender outcomes up front as project objectives subsequently pay greater attention to such outcomes and score higher on supervision than do those projects which do not identify such outcomes. The QSA5 review highlighted the following elements of good practice, many of which echo those identifie d in the QSA4 review.

• Results-based management through early identification of SD outcomes.

Those operations which identify SD outcomes during preparation generally focus supervision activities much more directly on behavioral changes – changes in the “rules of the game” – than do those operations which focus only on outputs or inputs. Examples include India AP DPIP, Nepal Basic and Primary Education and Algeria Rural Employment. Though not always successful, operations which clearly identify the changes being sought in the service of SD objectives tend to be much more effective in adjusting their style of implementation based on actual outcomes on the ground.

• Systematic social analysis manifested through the life of the operation. Good practice entails consistent monitoring and evaluation of the progress of the operation with regard to social development outcomes throughout the course of implementation. For example, Russia Coal entails the use of TA in association with SECALs to provide strong support for monitoring, and responding to, the social changes and impacts resulting from restructuring of the coal sector. Ongoing SAs and social impact studies provide support for improvement in the design and implementation of community development programs and pilot activities of community support and employment. The Indonesia KDP has utilized a variety of studies and analyses during implementation (assessments of the effectiveness of local governments in reflecting the needs of local populations, studies on corruption, conflict studies, etc.) which are then “fed back” into the project to improve implementation and the likelihood of achievement of SD outcomes.

• Sustained participation by intended beneficiaries during supervision. The most critical mechanism for determining success or failure in achieving SD outcomes is feedback from the intended beneficiaries themselves. Those operations which include participatory M&E as a part of supervision tend to be more flexible, lead to greater ownership among in tended beneficiaries, and appear to hold promise of greater sustainability of project outcomes.

• Use of Innovative M&E strategies, particularly in CDD operations. Ongoing supervision of SD outcomes, with the active participation of intended beneficiaries, can be a time and effort-consuming activity. The best approaches have used innovative approaches to conduct such monitoring. For example, Indonesia KDP utilizes independent journalists as both monitoring and evaluation mechanisms, and engages civil society in similar roles to achieve what is effectively continuous supervision. Supervision from field offices, as opposed to formal missions from headquarters, also result in more continuous M&E (India


AP DPIP, Indonesia KDP). The use of independent consultants for M&E also seems to improve the transparency of supervision and provides a valuable independent check on progress (e.g., Algeria Rural Employment).

11. Good practice therefore entails (a) focus on social development outcomes enabling results-based management; (b) ongoing social analysis for adaptive learning, improved targeting, gender equity, and understanding of informal institutional behavior; (c) sustained participation enhancing stakeholder ownership and empowerment, with feedback mechanisms to improve project implementation, and (d) use of innovative and field-based M&E strategies. Of course, application of these best practice elements remains uneven. The challenge lies in mainstreaming these practices across the portfolio.

60 A QAG Assessment

ANNEX 7. SUPERVISION OF ENVIRONMENTAL ASPECTS Overview 1. A team of environment specialists assisted QSA panels in assessing the quality of the Bank’s supervision of environmental planning and management in QSA5. Environmental safeguard and other environmental issues were found to be relevant in 72 of the 100 projects in the sample. This annex discusses key findings, strengths and weaknesses of environmental supervision, issues requiring Management’s attention and proposed actions to improve performance. The key areas in which environmental issues were assessed were:

• Extent to which environmental issues have been monitored during supervision;

• Extent to which supervision has assessed and taken into account the views of key stakeholders or attempted to strengthen mechanisms for participation in implementation and/or monitoring;

• Compliance with Bank environmental safeguard policies concerning environmental

assessment, pest management, forestry, natural habitats and, where relevant, safety of dams;

• Effectiveness of supervision of environmental issues; and,

• Level of effort devoted to supervision of environmental issues.

Sample Profile 2. Table 1 provides a profile of the environmental categories of the QSA 5 sample. Twenty-eight projects, all in category C or uncategorized, ranging from fiscal consolidation to labor force development, posed no significant environmental impacts relevant to implementation or supervision. Accordingly, the remaining 72 projects were reviewed and the quality of the Bank’s supervision was rated on the basis of attention given to compliance with the Bank’s environmental safeguard policies and the effective implementation of environmental mitigation and other protective measures. These projects include large-scale infrastructure, community-driven deve lopment, natural resources management and intermediary financing.

Table 1: QSA Sample

Environmental Impact Category

No. of Projects

A 12 B 47 C 38

Other 3 Total 100


Key Findings: 3. Overall, supervision of only 72% of the projects was found to be satisfactory. Rating of satisfactory projects was particularly low in AFR (24%), with SAR scoring relatively low (60%), LCR 78% and all other three regions rated above 90% satisfactory (EAP 92%, MNA 95% and ECA 97%). 4. Triggering of Environmental Safeguard Policies. Environmental safeguard policies were triggered a total of 104 times, with 54 projects triggering just one and 18 projects triggering two or more policies (Table 2).

Table 2: Environmental Safeguards Triggered

Safeguard Policy Projects

Triggered Environmental Assessment (OP/BP 4.01) 73 Natural Habitats (OP 4.04) 9 Forestry (OP 4.36) 8 Pest Management (OP 4.09) 8 Dam Safety (OP 4.37) 4 International Waters (O P 7.50) 1 Disputed Areas (OP 7.60) 1 Total 104

Level of Compliance 5. There were 14 instances of non-compliance with environmental safeguard policies; nine in the case of OP 4.01 - Environmental Assessment (four projects in AFR, three in LCR and one each in MNA and SAR), and five in the case of OP 4.09 - Pest Management (two projects in AFR and one each in MNA, SAR and EAP). During the panel reviews, Task Teams reported that they would take appropriate remedial measures to bring these projects into full compliance. 6. Eleven projects (mostly education and health) were inappropriately assigned a “Category C” at the time of board approval and have given rise to environmental problems during implementation. The significance of these issues was discussed during panel meetings and corrective measures and monitoring arrangements were suggested to ensure compliance with the Bank’s environmental safeguard policies. Implementation of Environmental Mitigation and other Protective Measures. 7. Bank supervision of the implementation of environmental mitigation and other protective measures had a 72% satisfactory rating, lower than that of all previous QSA exercises (Figure 1). Ratings for level of effort and effectiveness of supervision of environmental issues and risks were rated 71% and 68%, respectively. Explanations for the decline may be found in inadequate budgets, unavailability of skilled staff, inappropriate category rating, and the continuing marginal importance assigned to environmental supervision. On a more positive note, EMPs were prepared and are being implemented for 83% of projects sampled. However, their implementation was found to be satisfactory in only 69% of the cases (see para. 10). The views of the stakeholders was satisfactorily taken into account 75% of the time.

62 A QAG Assessment

Figure 1: Quality of Environmental Supervision (%) RSA3 to QSA5

82

101

13 15

0

21

7

72

78

64

0102030405060708090

HS S M U

%

RSA3 QSA4 QSA5

8. Performance by region (Table 3A) and network (Table 3B) demonstrates differences noted in previous assessments. AFR has the lowest rating (24%), SAR (60%) and LCR (78%) relatively low and the other three regions exceeding 90%. Ratings by network suggest that ESD is low (56%) while the other three networks are above 75% or above.

Table 3A: QSA5 Ratings by Region for Supervision of Environmental Aspects*

HS S HS or S M U Region No. of

Projects % No. % No % No. % No. % No.

AFR 19 - 0 24 8 24 8 59 9 16 2 EAP 11 24 3 68 6 92 9 3 1 5 1 ECA 11 16 1 81 9 97 10 - 0 3 1 LCR 13 - 0 78 10 78 10 22 3 - 0 MNA 10 - 0 95 9 95 9 5 1 - 0 SAR 8 3 1 57 5 60 6 20 1 20 1

Bank-wide 72 8 5 64 47 72 52 21 15 7 5

Table 3B: QSA5 Ratings by Network for Supervision of Environmental Aspects*

HS S HS or S M U Network

No. of Projects % No % No. % No. % No % No.

ESD 21 3 1 52 12 56 13 24 4 21 3 FSE 0 - 0 - 0 - 0 - 0 - 0 HDN 12 18 1 66 10 84 11 16 1 - 0

PREM 2 15 1 85 1 100 2 - 0 - 0 PSI 38 6 2 69 24 75 26 22 10 3 2

Bank-wide 72 8 5 64 47 72 52 21 15 7 5

* Note: • Out of 100 projects, 28 posed no significant environmental impacts relevant to supervision • The percentages are weighted averages (as referred to in the main body of the report)


Strengths and Weaknesses of Supervision 9. A clear strength included the use of Environment Specialists from Resident Missions that undertook regular, on-site visits and faithfully reported their findings to the TL. In some cases, this was ma de easier by proactive actions taken by the Borrower and the direct involvement of government environment agency staff or NGOs. Although of varying quality, EMPs are now more regularly included in preparation documents, sometimes being cited in annexes to the PAD. 10. Weaknesses are apparent in EMP implementation, exacerbated by the ill-defined roles of the PIUs, lack of or insufficient coverage of environmental mitigation in operational manuals and major lapses in the quality of reporting in PSRs and Aides-memoire. There are also continuing concerns over the marginal use of Environment Specialists during supervision, the treatment of FIs and CDD projects, and environmental category misclassification. However, most worrying is the high level of non-complia nce with OPs 4.01 and 4.09 – some 19% of the projects reviewed – well above that found in previous QAG exercises. Issues Requiring Further Attention and Proposed Actions 11. Based upon QSA5 results, many aspects of environmental supervision require immediate attention and follow-up actions. Most of these have been noted in previous QAG reports and Management should review how the Regions (with the support of the Environment Department) propose to take concerted action to avoid repetition in the future. Environmental Management Plans (EMPs) 12. Although of variable quality, EMPs or variants were available to guide the implementation of mitigation measures in 83% of the classified projects sampled – an improvement over previous years. However, monitoring their implementation and ensuring satisfactory performance continues to be a major shortcoming. Frequency of site visits by Bank staff and discussions with the Borrower to review needed actions were often insufficient and there is little evidence in project documents of diligent follow-up, including correspondence with the Borrower on actions needed to assure continued compliance with the Bank’s safeguards. Part of the problem lies with the lack of, or poorly defined, environmental supervision arrangements in project supervision plans. Equally, unlike procurement and financial requirements (that are regularly detailed in annexes to the PAD), environmental requirements are rarely stated in commensurate detail. Another failure concerns the varying quality of environmental performance indicators and output – an issue noted in last year’s QSR report. 13. It was also found that changes in project design and/or reallocation of funds to new components were rarely assessed on environmental grounds. Such changes often pose major safeguard risks, and the Regions should alert Task Teams to potential risks (e.g. in one project in the sample, funds were reallocated from a road component to a municipal solid waste component without any environmental and social review).

64 A QAG Assessment

Recommended Actions

• Most Regions, as part of cross-support, are providing guidance on the preparation and implementation of EMPs, including examples of good practice. However, effective EMP implementation requires increased use of Environment Specialists as part of field supervision.

• Regional Environmental Safeguard Specialists should assess changes in project design/reallocation of funds to new components for potential environmental impacts and, as appropriate, a new project review exercise should be conducted.

• Project supervision plans should be more explicit about environmental supervision requirements.

Application of Environmental Safeguard Policies 14. There are 14 instances of non-compliance with safeguards, 9 concerning OP 4.01 - Environmental Assessment and 5 concerning OP 4.09 - Pest Management. Some Task Teams failed to recognize the application of the OP during preparation, some others failed to take action during implementation, sometimes in the face of a compliance violation being indicated by independent study. During panel discussions, TTLs for the relevant projects stated that they were taking appropriate actions to employ appropriate specialists to assure compliance with the OP. This should be followed up by the Regional Environmental Safeguard Units. Recommended Actions

• Ensure that non-compliant projects are brought into full compliance with the relevant safeguard policies.

• Continue staff training under the General Safeguard Policies Training Course. This course should emphasize that safeguards may be triggered even during project implementation and measures need to be developed regardless of any up-front work.

• The Environment Department should convene a meeting with the Regions to agree on a program to assist and ensure that Task Teams comply with OP 4.09. This might include staff from the FAO, as their failure to apply good practice was evident in those projects where they were handling pest management components.

Financial Intermediary and Community Driven Development Projects 15. Environmental supervision of these projects is especially weak and deserves urgent attention. Environmental guidance for FIs was often available (criteria, institutional arrangements, etc.), but Task Teams rarely chose to supervise its application or to conduct spot checks/audits during implementation. Consequently, the adequacy of environmental protection measure for sub-projects could not be determined. Fortunately, in some instances, the borrower applied national regulations and reported progress or otherwise in quarterly reports to the Bank. The CDD projects face similar problems.


Recommended Actions

• Regional environment units should conduct workshops to assist Task Teams to prepare and supervise FI and CDD projects more effectively. Guidance should be pragmatic but flexible enough to accommodate the evolving nature of these instruments as recognized in the recent Environment Update on FIs. The Environment Department is currently developing guidance on the handling of CDD projects; this should be made available to the Regions through workshops and guidance materials.

Operational Manual and the Project Implementation Unit (PIU) 16. The inclusion of environmental management requirements in the Operational Manual is crucial to ensuring environmental due diligence by the project team. However, many manuals included vague statements giving no details of institutional responsibilities, scheduling or budgeting; some were without any mention of the need for environmental management. As an exception, road sector projects were very instructive in providing detailed guidance (especially in AFR), including performance indicators and outputs, and gave Task Teams a rigorous framework in which to supervise and monitor progress. On an allied issue, many Task Teams took the view that environment issues were “taken care of” if the Project Implementation Unit (PIU) hired an environment officer and chose not to evaluate performance. In addition, supervision reporting often adopted the environmental supervision findings of the client (PIU or other unit) without adequate evaluation. Recommended Actions

• Task Teams should ensure that environmental management requirements for projects are satisfactorily reflected in Operational Manuals. Regional Environment Units should assist as required.

• When PIUs have their own environmental staff, Task Teams should assess the capacity of PIUs environmental staff, review the effectiveness of PIU performance, ensuring adequate review of the implementation and reporting of environmental protection measures, including compliance with Bank safeguards.

Quality of Reporting 17. Reporting in PSRs continues to be perfunctory on compliance with safeguard policies and environmental protection performance in general. This is perhaps a reflection of the limitations of the PSR format and the minor importance accorded to environmental aspects by managers. However, aides-memoire were often equally perfunctory in reporting environmental due diligence, especially on problems requiring management’s attention. These are valuable means of recording issues, identifying agreed actions and measures for follow-up. Some of the Regional Environment Units (AFR & MNA) are conducting environmental supervision reviews and these are proving instructive to Task Teams in demonstrating the value of comprehensive reporting on environmental issues during implementation; they also point out instances of non-compliance.

66 A QAG Assessment

Recommended Actions

• The PSR format should be modified to accommodate better reporting of compliance and implementation of environmental mitigation and other protective measures.

• Aides-memoires should be used more effectively to report environmental issues and recommended actions.

• The practice of environmental supervision reviews adopted by some Regions should be considered for possible Bank- wide application.

Role of Environment Specialists 18. The failure to use environment specialists during supervision continues, particularly in relation to Category B projects, some of which are particularly complex/ris ky. During panel interviews, it was learned that TTLs often accepted the need to include such specialists, but were constrained by insufficient budget.1 As in the past, some chose to partially offset this constraint by staff doubling as environment specialists – sometimes with disastrous results. The practice of using Resident Mission staff again served to assure quality of supervision, largely because of regular (sometimes surprise) field visits and immediate follow-up. The value of including environme ntal agency staff in field missions to improve project performance and help mainstream environmental supervision is being recognized in a number of regions. Recommended Actions

• Environment Specialists should be included in project supervision of category B projects at least at mid-term review, and for the most complex ones, every year, and country environment agency staff should be encouraged to join supervision missions on a selective basis.

• Supervision review by environment specialists could be enhanced by their offering recommendations on linking the project with other geographically proximal Bank or donor assisted projects having similar objectives (e.g.. natural resources management/rural development and water management projects) to gain efficiencies and in dealing with the systemic issues.

• Opportunities for giving presentations to government/project staff during supervision should be explored, e.g. safeguards/EMPs.

1 The panel heard that some TTLs used local consultants to overcome this constraint; others availed themselves of funds from regional safeguard units.


Environmental Category Misclassification 19. This persists as an issue affecting most Regions. In 11 projects (mostly education and health) a “C” Category classification was inappropriate despite evidence to the contrary at preparation and subsequent identification of significant adverse impacts during implementation. Some education projects reviewed included the building of hundreds of schools and associated sanitation facilities; some health projects involved use of chemicals to control vectors. However, the panel learned that in all cases the Task Teams have introduced or are introducing appropriate mitigation measures. Recommended Actions

• The region’s environment units should undertake a series of workshops with education and health sector Task Teams to demonstrate the process of EA classification of education and health projects and disseminate the existing guidance documents, identifying mitigation measures and their implementation, including supervision.

68 A QAG Assessment

ANNEX 8. SUPERVISION OF FINANCIAL MANAGEMENT ASPECTS

Overview 1. A team of financial management specialists (FMSs) assessed the quality of the Bank’s supervision of financial management (FM).1 The reviewers were guided by specific criteria and their ratings were based primarily on the Bank’s timeliness in identifying and addressing issues, the frequency and quality of FMS inputs, coverage of FM issues in project documentation, the extent that management support was provided to resolve FM issues (if required), and the degree of support provided by the Bank to the borrower in resolving problems or otherwise improving project FM performance.

Table 1 – QSA5 Financial Management Ratings Number of Projects

Type of Operation

Highly Satisfactory Satisfactory

Marginally Satisfactory Unsatisfactory

Total

Investment 15 61 19 0 95 Adjustment 0 5 0 0 5

2. The level of satisfactory supervision of financial management was 78%. This is somewhat lower than the levels in QSA4 (84%) and higher than QSR (73%). In addition, the results confirm that the quality of FM supervision remains below that of FM Quality at Entry. 2 3. Key findings were that:

• Approaches to FM supervision are inconsistent. Guidance to staff is needed to better define and support good practices.

• Good FM supervision is a team effort – involving effective collaboration within

the regional FM team, the FMS, the TL and Country Office Staff. Strong supervision of FM resulted when the task team worked together to not only identify issues but also follow up to resolve them.

• Supervision of FM was often highly dependent on the TL’s perception of FM. • Weak FM supervision was in many cases due to insufficient human resources

(number of FM specialists) in the regional FM teams. • FM-related supervision often is not well documented, although there are some

good practice examples worth noting.

1 The review also covered disbursement aspects to a limited extent: SOE reviews and the resolution of Special Account problems. 2 The three most recent Quality at Entry (QEA) exercises, which focus on new projects, have judged the quality of compliance with Bank FM policies to be 88% (QEA5), 95% (QEA4), 89% (QEA3).


Table 2 – Sub-data on FM Supervision

Number of Projects

Issue Yes No N.A. Total % Satisfactory

Adequate communication with the borrower on FM issues outside of missions 65 19 16 100 77

SOE reviews were carried out and results were

documented 47 39 14 100 55 Special Accounts issues were dealt with in a timely,

decisive, and effective manner 34 7 58 99 83 Audit reports reviewed timely, by a qualified person 72 17 11 100 81 Appropriate and timely response sent to the borrower

regarding review of audit report 63 23 13 99 73 FM issues identified, and followed up appropriately 71 18 9 98 80 Task team assisted the borrower in working out

problems and/or improving FM arrangements 74 13 13 100 85 Appropriate level of FM-certified staff in supervision 75 22 3 100 77 Documented inputs addressing relevant FM areas 81 16 3 100 84

Key Findings: Strengths and Weaknesses of Financial Management Supervision 4. As shown in Table 2, the reviewers rated particular aspects of FM supervision. The yes/no ratings of these questions are generally consistent, within projects and overall, with the FM ratings assigned to the projects. Of the main FM and disbursement activities of supervision, the most significant outlier is SOE reviews, which, where applicable, were only carried out in 55% of the projects over the two year span. In addition, the nature and documentation of SOE reviews was highly inconsistent, ranging from a sample of a few transactions and brief mention in an aide-memoire, to a region-wide, sophisticated review across projects. Regarding adjustment loans, while these were all rated satisfactory (Table 1), the FM reviewers had considerable difficulty in defining appropriate FM supervision of these operations. Key general findings are:

a) Supervision is generally adequate, but there are substantial weaknesses that should be addressed. With 78% of projects rated satisfactory, FM supervision while not weak, has room for improvement. There are clear steps that can be taken by the Bank to improve FM supervision, which are detailed in the Recommendations section below.

b) There is a lack of clarity in the Bank as to what constitutes good FM supervision.

Other than the QAG criteria, no explanation has been provided to staff as to what FM supervision should entail. Therefore disagreements between staff, and the combination of “old habits” and new but unclear mandates, have continued to confuse staff as to what is required. Some examples of this are that:

70 A QAG Assessment

• Financial analysis is confused with financial management. In many projects,

financial analysts carried out the valuable role of analyzing the financial performance of entities, but their work was incorrectly considered to be addressing financial management aspects of supervision.

• Financial management supervision is often seen as a reactive rather than

proactive process. Some teams continue to focus on reviewing audit reports, supervising the FM aspects of the project only to the extent that the audits indicate FM issues.

• Documentation varies widely. (see below).

c) Highly satisfactory supervision of FM requires a collective effort. The best FM

supervision occurred where the FMS was able to closely monitor the project, while being integrated into the task team. Since most FMSs are in field offices, the importance of linkages to headquarters staff is also important. Strong supervision was found in some large country offices, where the FM team was strong, and had a balance of national and international expertise. More generally, country office FM staff were effective when integrated into task teams, which is particularly important in projects involving complex implementation arrangements. In one community-oriented project, for example, the reviewer noted that: “The decentralized, community driven development approach…poses challenges to financial management, to which the Bank team has been responsive. FM is well integrated into the team’s overall approach. Continuity in FM team composition has helped consistency in supervision…. Overall, the team is to be commended for its support to capacity building in a decentralized financial management system.”

d) TTLs often determined the quantity, and influenced the quality, of FM supervision.

Supervision of FM was ultimately the responsibility of the TTL. Therefore the intensity of supervision was driven by TTL judgments, or in many cases, budgetary concerns. In projects where FM was well supervised, the TL was not only open to FMS involvement, but engaged in FM issues. Poor supervision resulted when there was an FMS/TTL disconnect; in some country offices, local FM staff often did not carry out supervision if not requested by the TTL. In others, TTLs were not sufficiently aware of the FM work being carried out.

e) FM-related supervision often is not well documented. Supervision of FM is not

documented or archived consistently. Documentation of mission work is often separate from that of the team, informal supervision is verbal or in e-mails, and required records are not in one place. While it is understandable, and in many cases even desirable, for FM staff to visit projects independently, documentation of missions as well as other FM supervision activities should be more consistently prepared, be readily available, and be integrated with other supervision documents. The problem may be especially relevant in field offices, where FM staff may have frequent informal exchanges with clients that are never formally documented. While part of the problem is a general one, the FM teams can do better.


f) Weak FM supervision was sometimes due to shortfalls in human and budgetary resources. In some cases, especially in EAP and LCR, TTLs cited the unavailability of FM staff as a reason for insufficient FM supervision. Budget constraints also were blamed, especially during FY01 when task budgets were constrained and TTLs were faced with choosing between competing priorities.

Recommendations for the Financial Management Sector 5. The findings of the QSA5 suggest certain actions that the FM Sector Board should undertake to improve project supervision:

• First, as a general measure to improve the quality of FM work, regional FM teams should more fully adopt quality arrangements. These arrangements should include a process of internal reviews, feedback to (especially junior) staff as to their performance, a system of electronic documentation of FM supervision and its review, and improved identification of, and attention to, projects with the highest FM risks and opportunities.

6. In three other areas, actions are already underway and should be completed. The FM Board should:

• Manage the shift of responsibilities for FM supervision from TTLs (demand driven) to the regional FM teams (risk and opportunity based). In most regions, the Bank’s internal budgetary and accountability practices have been revised to allow the FM team to more easily monitor FM aspects of projects as it sees fit, in collaboration with its internal clients. This will allow for a more appropriate allocation of staff and budgetary resources to FM supervision than in the past. While the TTL still has ultimate responsibility for supervision in a given project, the FM Board should work closely with its internal clients to agree on roles and responsibilities, recognizing the TTL’s role as head of the task team but also the FM team’s responsibility to optimize FM supervision across the region.

• Prepare FM Supervision Guidelines, as per the Board’s FY03 work program.

These guidelines will help to standardize the supervision process and to more clearly define what is “good FM supervision” and how it should be documented. Such guidelines would complement the FM Assessment Guidelines, which have served as an important tool for documenting the FM reviews of new projects.

• Determine the proper role of SOE reviews. This is currently being discussed as part of reforms to the Bank’s audit and disbursement policies and practices.

• Clarify the role of FM supervision in adjustment operations. This is expected to be addressed by the in-process guidelines to staff being prepared by the Financial Management Sector Board.

72 A QAG Assessment

Improving the QSA Process for FM 7. New approaches to the QAG process, applied during QSA5, should be continued in the future. Process improvements of this review included training the reviewers prior to the beginning of the exercise, improving communication between reviewers and the FM coordinator, copying regional FM Managers on draft reviews sent out by QAG, and involving the FM Board Quality Committee in both reviewing the criteria prior to the exercise and providing feedback once the reviews begin. In the future, the following additional steps could help to improve the reviews of supervision:

a) Focusing more on the PSR rating and how it was determined. This rating, introduced only in March 2002, requires the team to make a determination as to the quality of FM arrangements on the project. While it was only applicable to a small part of the review period, reviewers found the responses to the question of “How was the FM rating determined?” to be a good indicator of the team’s approach to FM supervision.

b) Incorporating more explicitly the extent to which interim reports (especially Financial Monitoring Reports) are being reviewed and utilized to monitor the project. Following the simplifications to progress reporting requirements in the “fixing LACI” exercise, QAG reviews should consider the impact of reporting and the use of reports to monitoring the financial activity, outputs, and procurement plan of the project.

c) Aligning FM supervision with revised business processes and objectives. The FM Sector Board is currently studying how the sector can shift its focus more heavily towards impact while still paying appropriate attention to compliance. This study will assist in re-defining “quality” FM work, and this new definition should be reflected in the FM Supervision Guidelines and future QAG supervision reviews. Some initial ideas would be for FM, and perhaps QAG, to carry out reviews of the performance of supervision on a country, or even regional, basis, in order to more closely look at progress regarding some of the items (e.g. risk and opportunity based approaches, regional quality arrangements for FM) mentioned above.


ANNEX 9. SUPERVISION OF PROCUREMENT ASPECTS

Overview 1. The assessment of procurement supervision was done by specialized procurement reviewers. The reviewers conducted in-depth assessments of various aspects of procurement including planning, implementation, progress monitoring, addressing procurement issues and problems during project implementation, post reviews, compliance with loan agreements and Bank policies, and development of institutional procurement capacity. Results for the QSA sample indicate that 91% of the projects benefited from satisfactory supervision of procurement aspects (compared to 92% in QSA4). Key Findings 2. Key findings were as follows:

• Percentage of projects with highly satisfactory procurement supervision has remained same at 10% between QSA4 and QSA5

• Percentage of projects with satisfactory procurement supervision has also remained same (92% in QSA4 and 91% in QSA5).

• Procurement supervision of projects in HDN network (Health, Education and Social Protection) has improved slightly in QSA5 to 86% (83% in QSA4).

• Although procurement plans were prepared at entry for most of the projects, they were often not adhered to or used for monitoring progress during implementation.

• Supervision reporting, documentation by task teams on procurement status, were found to be the weakest aspect of procurement supervision.

Table 1-A: QSA5 Ratings by Region for Procurement Supervision

HS S HS or S M U Region

No. of Projects % No. % No % No. % No. % No.

AFR 25 7 3 74 17 81 20 17 4 2 1 EAP 15 16 2 80 12 96 14 5 1 - 0 ECA 19 2 1 96 17 98 18 2 1 - 0 LCR 16 22 2 62 11 84 13 16 3 - 0 MNA 11 - 0 100 11 100 11 - 0 - 0 SAR 8 10 1 83 5 93 6 7 2 - 0 Bank-wide 94 10 9 81 73 91 82 9 11 0.4 1

Note: Out of the sample of 100 projects, there were 6 projects without any procurement.

74 A QAG Assessment

Table 1-B: QSA5 Ratings by Network for Procurement Supervision

HS S HS or S M U Network

No. of Projects % No % No. % No. % No % No.

ESD 21 4 2 81 15 84 17 16 4 - 0 FSE 5 - 0 91 4 91 4 9 1 - 0 HDN 19 15 2 71 15 86 17 14 2 - 0 PREM 5 9 1 82 3 91 4 9 1 - 0 PSI 44 12 4 84 36 96 40 3 3 1 1 Bank-wide 94 10 9 81 73 91 82 9 11 0.4 1

Note: Out of the sample of 100 projects, there were 6 projects without any procurement. Please note that when the Projects are divided by Regions or Sectors, the number of projects in some Regions and in some Sectors are too few to be statistically significant.

Strengths and Weaknesses of Procurement Supervision

3. There was not much variation in the quality of procurement supervision. There were some common features among the well supervised projects, which are:

• Prompt responses provided to procurement issues raised by the Borrower.

• TORs outline d procurement supervision tasks.

• BTORs and aide-memoires addressed procurement implementation issues.

• Supervision team include d a Procurement Specialist (PS) or a Procurement Accredited Staff (PAS).

• Project procurement plans are updated periodically. 4. The general weaknesses in the supervision were:

• Lack of written reports on post-review of procurement contracts below the prior-review threshold.

• Action plan/ recommendation for capacity building of implementing agencies not followed through.

• Even projects with specialized procurement needs are designed with the traditional approach and do not design specific procurement arrangements for such purposes. (for fertilizers, pesticides, live stocks, crop plants etc).

• Quality of supervision by PS and PAS is uneven. The PAS looks after mainly the technical and overall (as TTL) aspects of the project and then does not provide the necessary support and/or supervision of procurement function for the project.


Issues Requiring Further Attention 5. There are two issues, which may require further attention:

(i) Overall quality of procurement supervision by staff who are PAS, compared with Procurement specialist have been found inferior. Therefore, this issue needs to be followed up by the procurement sector board.

(ii) Proper documentation on procurement supervision continues to be a problem. Panels encountered difficulties retrieving in a timely manner the key correspondence and progress reports related to procurement issues. Where documented information was missing, panels relied on verbal information provided by the task team.

Conclusions

6. The following points will be helpful for effective procurement supervision:

• A procurement plan is essential. The procurement plan at appraisal should be the “base” and should be updated/revised as necessary or at least annually.

• Procurement plans should be monitored at least every quarter and reviewed regularly during supervision. The PS/PAS should help the implementing agency design a Procurement Monitoring Table that includes details such as the bid/contract number, nature of bid/contract, procurement method, name of supplier, contractor or consultant, key procurement dates, value, origin of bidder, and other relevant information. The Financial Management Report has a section for procurement, which covers this aspect for contracts not subject to prior review.

• The PS/PAS responsible for the project should communicate regularly with the implementing agency and should be available to provide prompt, quality responses to procurement issues raised by the Borrowers. (Decentralization facilitates this, however, evidence shows that a satisfactory level of service can also be provided from HQ).

• The supervision mission’s TOR should clearly state the supervision tasks related to procurement. During this mission, project staff should:

(i) review the procurement progress against the plan and revise as necessary; identify and address procurement issues affecting (or having the potential to affect) the implementation of the project;

(ii) identify and address pending contractual issues from various parties (PIU, contractor, and consultant);

(iii) conduct post reviews of the procurement decisions below the prior-review threshold and also review documents relating to contract variations, price adjustments, etc. for prior-review contracts;

(iv) review the procurement capacity of the implementing agency and address training needs of the implementing agency staff, including conducting workshops, and

(v) visit project sites and conduct actual physical inspection.

76 A QAG Assessment

• The procurement rating in the PSR must be justified by the write up in the implementation status and the issues sections. Whenever a PS/PAS participates in a mission, an annex on procurement progress should be attached to the aide-memoire or a separate report should be written. The report should summarize procurement activities, such as current status, planning of remaining activities, problems/issues and actions/agreements with the Borrower.

• Supervision missions follow-up letters should confirm the mission’s findings and recommendations, summarizing the above procurement problems/issues.

• The BTOR of the PS/PAS should include the post-review report in the recommended format, and it should be sent to RPAs and recorded in the regional monitoring system. If a post review exercise detects non-compliance with the legal agreements, then the report should include proposed remedial actions.

• The PS/PAS is responsible for the procurement supervision of projects and, therefore, should be a party to any procurement related communication with the Borrower.


ANNEX 10. SUPERVISION OF MONITORING AND EVALUATION ASPECTS

Overview 1. The assessment of monitoring and evaluation (M&E) in supervision received special attention in QSA5, as a team of specialized reviewers assisted the QSA5 panelists in assessing 100 projects in the sample. The M&E ratings in the main questionnaire were drawn from detailed comments and sub-ratings which assessed (i) the appropriateness of the M&E system to assess progress towards outcomes and implementation, (ii) the use by the Bank of M&E information in supervision (outcomes, outputs, implementation progress, and risks), and (iii) the degree of recognition, follow up, and support to the borrower when M&E issues arose (such as weak or non-functioning M&E systems). 2. The level of satisfactory supervision for M&E was 65%, a decline from QSA4 (79%). It should be noted that in this review a team could receive a satisfactory rating in M&E in supervision without an appropriate M&E system in place (at entry), if the Bank team made significant efforts to improve project M&E. Many times, however, efforts were well intended, but failed to be “technically” at par.

Key Findings 3. The key findings of the QSA5 M&E review are:

• There are major shortcomings in the quality of the M&E systems in place.

Only 42% of the projects had appropriate M&E systems, which includes clear project development objective, measurable indicators to track progress towards outcomes and consideration of borrower capacity.

• There is an increased recognition that there are problems with the quality of

the M&E system in place. However, the teams either do not have suffic ient support to rectify the system (financial or human) or they are overconfident that the team can restructure/redefine the M&E system themselves. Few projects were able to address identified weaknesses in M&E sufficiently.

• Teams tend to focus on monitoring physical implementation and less on progress towards outcomes. The predominant focus of Bank supervision on outputs and activities is not in line with the Bank’s new business environment to move towards outcomes. Even in Mid-term Reviews, many teams focus more on outputs and processes than on assessing whether the project is progressing towards the project development outcomes.

• Overall, the quality of reporting in the PSR is poor as is the use of key performance indicators (KPIs) to substantiate DO ratings. The specialist panelists observed that most M&E information is buried in aide-memoires or back to office reports with only a few anecdotal examples of managers requesting improvements in using KPIs in the PSR. Only 28% of the projects use KPIs to substantiate the DO rating, which is related to the overall weakness in realism of project performance ratings.

78 A QAG Assessment

• Few teams use M&E information to advise borrowers on modifications to project implementation (outside of traditional input, contracting, output advise, etc.) and in most of the projects reviewed, borrowers do not appear to own the M&E system that was “put in place” at entry. This often results in borrowers not using this system as a project management tool. 3

Element of Good Practice in M&E in Supervision 4. Highly Satisfactory. Six projects received a Highly Satisfactory rating, representing examples of good practice in M&E in supervision. These have several common features, with each project having a good results-based M&E system at entry which was used for decision-making by the Bank to influence the Borrower. Specifically:

• The project development objective was clear, realistic given the strategy and resources, targeted, and practical to measure (e.g. specification of intended beneficiary groups, baseline and end-of-project targets). It was very clear what the project sought to directly achieve (such as improved silk production of a targeted group) and to what higher order objective it contributed (such as decreases in poverty).

• Borrower ownership of the system and capacity to sustain and use the M&E information was considered at entry and fostered during supervision. There were clear indications in borrower reports that borrower ownership of the M&E system existed and that it was used to report project progress.

• These teams not only used the system in place for monitoring outcomes, outputs, and risks, but they also monitored the effectiveness of the system itself, and made efforts to improve it. For example in a few cases, the original baselines values for indicators or data collection mechanisms were found to be optimistic given the implementation conditions, so the team worked with the borrowers to improve these.

• These teams recognized the need for specialized M&E input, suc h as defining M&E for institutional changes, and recommended expert support for the borrower, as opposed to trying to undertake this advice themselves.

• The team used the Bank’s PSR to track progress and substantiated the DO and IP ratings with performance indicators (e.g. PDO specific outcome indicators, when data was available, or intermediate outcomes indicators).

5. Management Attention. It was encouraging to see some country teams using the CPPR to focus on systemic problems in M&E which may be hampering the use of M&E by borrowers for project management (at the country level). This may indicate increased attention by management. Unfortunately, there were only a few cases where the country director’s and sector manager’s comments in the PSRs reflected an increased concern for the quality of M&E in operations; most of these were in AFR.

3 Given the desk review nature of the QSA, it was not possible to systematically capture this dimension, which would require field visits and more in depth work.


Issues Requiring Further Attention and Recommendations 6. The poor quality of the M&E systems assessed in QSA5 as well as in previous QAG reviews demonstrates that a significant problem remains at the Bank. Task Teams are no more likely to design and use appropriate M&E systems to support good project and program management than they were three years ago, despite the Bank’s significant focus on improving the Bank-wide capacity in M&E. The following five areas were identified by the Special M&E Review Panel as key concerns that must be addressed to improve this situation 7. Incentives for Improved M&E (both Bank and Borrower). While there is anecdotal evidence that some managers are increasing their focus on M&E, most of the poor M&E reporting in the PSRs goes unnoticed by management. The degree of focus on M&E by teams suffers due to other priorities in supervision. More importantly, M&E does not seem to be owned by the borrower nor is the capacity of the borrower sufficiently assessed to both conduct and use M&E for management of operations.

Recommendation: Managers must regularly demand that teams design and use appropriate M&E systems in projects and programs . While much emphasis is being placed on the need for better M&E by the Board, MD’s, SRM and OPCS, it is necessary to send strong messages from Regional and Sector management. For borrowers, the Bank must carry a clear message on how M&E is useful for project management and not just useful for the Bank’s own reporting. The upcoming board paper on staff incentives in managing for results is an opportunity to address in depth these issues.

8. Lack of Skills in M&E to support Teams. QSA5 identified the concern that teams need additional capacity in the appropriate design and effective use of monitoring and evaluation systems. Projects reviewed by the Panel showed a lack of understanding of how to frame a good project development objective and construct realistic indicators, how to use M&E for better supervision and to support better management of the project by the borrower, and how to leverage supervision to assess and strengthen borrower capacity for monitoring and evaluation. In addition, assessing and tracking changes in institutional performance is one of the weaker areas of M&E in supervision, due in part to inadequate systems and baselines at entry. Teams tend to rely on counting how many people went on training as opposed to how effective this was in building skills. This is particularly concerning in the case of LILs where the KPIs were especially weak, but essential to assess performance. The basic premise of the use of LILs, that the Borrower and the Bank want to learn about the outcome in order to decide whether a larger operation is justified, seems to be missing from the M&E system.

Recommendation: Strong messages of the importance of M&E must be coupled with adequate resources and the regions have to own the effort. The Centers are helping the Regions and networks build new skills in M&E through the Monitoring and Improvement Program. However, it is clear that the Regions need to own the effort and a clear mandate from management must be coupled with dedicated funds to employ expertise in the area of M&E to enhance operational quality. Just in time support for teams should be available in the regions and the center should develop more targeted training on operational issues in M&E, not just technical, within the core curriculum for results based approaches M&E.

80 A QAG Assessment

9. Confusion of the extent of M&E required at the project level. The review found that teams have difficulty in focusing on which development objective is important for project management (i.e. a clear PDO focused on realistic outcomes) and which is a higher order objective to which the project contributes (sector impact, poverty reduction, etc.). Often teams feel that they must measure how a project contributes to poverty reduction while others feel the project should only be responsible for tracking completed activities and at most outputs.

Recommendation: Clarify the responsibility of lending operations for M&E and M&E that is best undertaken in the CAS or other ESW. This could decrease the burden on task managers to show “higher order impacts” and would allow simplification of some M&E systems. In addition, the messages on the importance of M&E for project management should drive M&E design at the project level and sufficient support to use the logical framework as a way to think through the logic of the operation and show how M&E is critical in project management should be provided. The need to involve the borrower in design is critical.

10. Borrower Ownership and Capacity: Too often the M&E systems appear to be designed to fulfill only a Bank requirement and do not appear to be owned or used by the borrower for project management.

Recommendation: Increased focus on building client capacity is essential. Task teams have a role in sending appropriate messages of the importance of a focus on results and providing guidance on how the borrower may improve their results focus. In moving forward with mainstreaming evaluation capacity development, where appropriate the Bank should include a focus on M&E in the public investment portfolio and develop similar tools for assessing project level capacity for M&E. Innovative work on diagnosing strategies for improving capacity and incentives for clients to use M&E should be continued and scaled up. WBI’s role would be important for training and capacity building as an integral part of regional strategies to improve M&E.

11. Adjustment Operations : Most adjustment operations tracked whether or not the actions in the policy matrix were undertaken, such as laws written, and did not monitor past this, such as whether or not the law was enacted and/or enforced. In this assessment use of these policy matrices was considered satisfactory for M&E in supervision, as there are different Bank requirements for adjustment and investment operations. Unfortunately the current policies for adjustment operations do not allow adequate monitoring of results. While outcomes may take several years to mature, there are leading indicators of policy changes which could be better monitored for adjustment operations.

Recommendation: Efforts to strengthen the M&E framework for lending and non lending, aligned to the CAS, 4 should clearly look to intermediate outcomes for tracking policy changes, and not just that actions in the policy matrix were undertaken, such as laws written. Adjustment operations would benefit from a focus on national level ownership and use of result based M&E as a part of public sector management.

4 Better Measuring, Monitoring and Managing for Development Results: Implementation Action Plan, December 9, 2002.


Bank Task Team View of M&E

During the fall of 2002, OPCS conducted focus groups with Task Managers from the AFR, EAP, ECA, LCR and MNA regions, and with mid-level Managers (Sector Managers, Portfolio Managers, and others). The purpose was to better understand the underlying issues of poor performance in M&E. There was strong consensus across groups of the need to strengthen M&E at all levels, especially in the CAS and project design. The main systemic issues and suggested actions were:

1. Focus: The Bank still tends to evaluate activities in terms of disbursement rates with M&E focusing on inputs and outputs, not outcomes. Staff suggested that the Bank should work from the CAS down, rather than from the project up.

2. Responsibility for M&E: Most viewed M&E as a client responsibility, but said that client capacity for M&E is often weak. The Bank has a greater role to play in building capacity among clients, and in helping them understand that credible and timely M&E can help them reach development objectives.

3. Time and Resources: Respondents described TTLs as heavily overburdened with a my riad of specialized expectations. They cannot be specialists in everything, nor should the Bank ask them to be. Rather, TTLs (and others) need access to an internal M&E service and support function, especially when they are designing a project with unusual features or high risk. Respondents across regions and positions described the resources available to them for staff, experts, time, or money dedicated to M&E as woefully inadequate or even non-existent. All groups discussed the critical importance of increasing financial and staff resources for credible M&E. Respondents, including Mid-level Managers, said their resources in this area are virtually non-existent. They especially need funds to hire M&E experts, as needed. Creating a cadre of experts “on call” in the Bank (and in the field) would be useful to either type of respondent.

4. Incentives: Respondents could name few incentives and fewer supports for taking M&E seriously. Other barriers include the Bank’s bureaucratic red tape, lack of consensus, failure to place M&E higher on the Bank’s list of priorities, insufficient supervision of TTLs, poor data collection by borrowers, almost non-existent M&E training, overloading TTLs with too many missions, overloading Mid-level Managers with too many supervisees, cancellation based on disbursement rates, and excessive time spent administering trust fund monies.

82 A QAG Assessment

ANNEX 11. SUMMARY RESULTS OF TTLS, PANELISTS AND OBSERVERS SURVEY

Following completion of the QSA5 panel interviews and assessment reports, TTLs, Panelists as well as Observers were asked to participate in a survey to help QAG obtain feedback on QSA5. While the TTL survey has been a long-standing feature of QSAs, the panelist and observer surveys were newly introduced under QSA5. Of the 100 TTLs, 197 Panelists, and 27 Observers who participated in QSA5, 60 TTLs (60%), 96 Panelists (49%) and 15 Observers (56%) responded to the survey.

A. TTL Feedback on Project Supervision

SUPERVISION BUDGET

1. When asked to assess the appropriateness of their supervision budgets for FY01 and FY02, about 70% of TTLs responding to the survey considered the supervision budgets to have been about right, while 30% responded that budgets were too little. Those considering the supervision budget inadequate indicated that if they had additional resources they would spend a third of it for more frequent and/or longer field visits, another third to engage additional technical experts, and the remaining third to ensure compliance with fiduciary and safeguard policies, and to improve M&E. The results obtained following QSA4 were broadly similar. TTLs continue to place a great deal of emphasis on the technical analysis underpinning their projects and feel the need to spend more time in the field. Interestingly, TTLs intend to spend about 12-15% of the additional supervision resources on M&E aspects, an area where Bank supervision continues to be weak.

MANAGEMENT SUPPORT

2. When asked about the quality of management support, 89% of the responses indicated that Sector Managers provided satisfactory guidance and support to facilitate project supervision. The level of satisfaction grew to 92% when asked whether managers made timely decisions on critical supervision issues, and the timeliness of Sector Managers’ feedback on PSR and post-mission follow up activities was found to be satisfactory at 91%. As for Country Management, 83% of the respondents expressed satisfaction in the way the Country Department used the CPPR or other venues to resolve project related problems. Overall, the level of satisfaction with management support in supervision is high and appears to have improved slightly since QSA4.

IT QUALITY

3. From the standpoint of IT quality, 50% of the responses indicated satisfaction with the SAP system, 60% were satisfied with IRIS, 52% were satisfied with the PSR, and BW received only a 27% satisfactory rating. The level of satisfaction with the IRIS and BW seems to have gone down since QSA4. More efforts are clearly needed by the institution to further integrate all of these systems and make them more reliable and user-friendly for operational staff.

USEFULNESS OF PSRS

4. Only 58% of the TTLs responding found the PSR in its present form useful and only 33% found the PSR user friendly. Only 53% of the TTLs reported that the ir managers used the PSR to provide inputs in the supervision. These findings are essentially the same as in previous years and corroborated by the feedback received from panels who found reporting as the weakest area in Bank project supervision. The fact that only about half the TTLs felt that managers use PSRs as a


management tool is also not surprising as most PSRs reviewed by the panelists failed to find many meaningful comments from the managers. Supervision management continues to be carried out mostly through mission debriefing, emails, aide-memoires and mission follow up letters.

5. An overwhelming majority of TTLs continue to find the PSRs as not user-friendly, too standardized to serve as a tool for devising and pursuing supervision strategies, and therefore, not really used by TTLs or their managers for project management. This lack of enthusiasm for the PSRs and the failure to use them meaningfully is clearly evident in the fact that out of the four dimensions of supervision the QSA5 exercise evaluated, the one area where the performance was satisfactory in only half the cases was in the realism and quality of supervision reporting. Since the Bank considers the PSR as the single most important document to report on the health of the Bank portfolio, a serious and sustained effort must be made to address the issues that the quality and use of PSR is facing.

MONITORING & EVALUATION (M&E)

6. M&E has been a weak area of Bank supervision and, therefore, several specific questions were included in this years survey regarding M&E. Only 31% of the TTLs responding felt that they had sufficient resources to adequately implement an outcome focused M&E system for supervision. Of those TTLs who did not feel that they had enough resources, 54% felt that they could use support from within the Bank, 41% felt that they would utilize outside consultants, and 49% felt that they need to improve their own M&E related skills. Managers provided guidance to about 22% of the responding TTLs in the effective use of M&E in supervision, and only 41% of the TTLs felt that Borrowers find the M&E for Bank operations useful. This is clearly an area where focused effort across regions is required.

FIELD OFFICE SUPPORT PROVIDED TO TL

7. Approximately 83% of the TTLs reported that they receive support from the field office in the supervision of their projects and 56% indicated that decentralization is having positive effects on the quality of project supervision. TTLs also appear to be satisfied with the quality of field office support received for procurement (68%), general task management (71%) and financial management (71%). Responding TTLs indicated limited satisfaction with the level of support from field offices in aspects related to environment (30%), social aspects (38%), and M&E (29%). The level of satisfaction with field office support appears to have leveled off although opportunities remain to strengthen capacity in several areas which can be immensely beneficial to project supervision.

TTL Feedback on PSR

“This is a user-hostile system. From the field it takes very long time to enter data and for that reason is rarely if ever used as a project management tool “.

“The changes in PSR made it look more like a form. The design was more appropriate to feed computer based data collection machines rather than for managerial review and discussion”.

“We do not use it as a project tracking system as it should be. It is partly due to its format, which is not very user friendly; partly because most of our effort is poured into the development of a good Aide Memoire (which we share with Govt.), and partly because our managers do not use it for follow up discussions”.

“Did not find any use of PSR in supervision as all what it contained were available in detail in mission reports/aide memoires”.

84 A QAG Assessment

STAFF TIME INPUT FOR QSA5

8. TTLs report that, on average, they and their task team members have spent 61 hours (45 hours of HL and 16 hours of ACS staff) preparing for, and participating in, the QSA5 process. This average is double that of QSA4 (29 hours). One of the reasons for this might be that QSA5 covered a two year review period and task teams spent more time in assembling the documentation requested for the review. However, this increasing trend in the task team time required for these exercises is worrisome since task teams are not provided with any additional budget to participate in QAG assessments and with all the budget and time constraints TTLs face, the reluctance to undergo QAG reviews and cooperate with QAG panels could become a problem.

QAG PERFORMANCE

9. TTLs gave the overall QSA5 process a rating of 89% satisfactory, same as in QSA4. Assessment panels, the assessment approach and the assessment process all received satisfactory ratings of 86% and above. When questioned on the overall usefulness of QSA5, 71% of the responding TTLs gave it a satisfactory rating. QSA5 as a learning experience received a 72% satisfactory rating; 77% found the review process as enhancing accountability within the Bank; and 74% also felt that the process is useful in identifying systemic issues. Considering that QSA5 was a desk assessment conducted primarily to provide feedback to senior management on the efficiency and effectiveness of the Bank’s supervision work, the fact that close to 3 out of 4 responding TTLs found value in these reviews is encouraging. Opportunities still remain, however, for devising new ways to add value to the QSA assessments by offering more detailed feedback to the task teams during the interviews and - possibly - through more systematic follow -up meetings.

10. Recommendations for QAG to be drawn from TTL comments include:

• Further improve the selection of panelists and panel configurations to reflect more country knowledge.

• Routinely include Sector Managers in the interview so that they internalize QAG recommendations better.

• Giving panelists sufficient advance notice and preparation time, specifically in terms of developing meaningful and timely lists of questions for task teams.

• Allocating additional time for interaction between panels and task teams, possibly including more frequent follow-up meetings after completion of the main interviews.

• All panelists (including specialist reviewers) should attend the interview.

B. TTL, Observer and Panelist Feedback on QAG’s Performance 11. QAG’s performance in QSA5 was assessed by TTLs, Panelists and Observers against five main aspects: (i) assessment panel; (ii) assessment approach; (iii) assessment process; (iv) usefulness of QSA; and (v) overall performance. The first four main aspects include several sub-questions. The table below presents satisfactory or better ratings given by each set of respondents:


NR = Not Rated N = Number of respondents 12. For the first time this year three separate questionnaires (for TTLs, Observers and Panelists) were used by QAG and the questionnaires produced valuable insights into the various perceptions concerning QAG’s performance. Overall, the following observations can be made:

• QAG’s overall performance received an overall 89% satisfactory rating from TTLs, and a 100% satisfactory rating from both panelists and observers;

• The panel skill mix, the fairness and transparency of the process, the role of the moderator and the quality of QAG administrative support received high ratings.

• Ratings given by the TTLs for panel familiarity with country specific issues was somewhat low, something QAG must take into account in selecting future panelists.

• While Panelists and Observers found the QSA to be very useful, TTLs were somewhat less enthusiastic. While part of this can be explained by the fact that TTLs find undergoing a QAG review an additional and non-funded burden, QAG must also better disseminate the main objectives of such assessments and better explain the relationship between individual assessments and the broader Bank quality enhancement agenda.

13. Some TTLs felt that there had been some major omissions by the panel and complained that panels had failed to fully appreciate the project and country context. This is also reflected in the low rating given by observers to the panels’ familiarity with country specific issues. Some TTLs were dissatisfied with the fact that their comments on the draft assessment were not fully reflected in the final assessment. While some of these concerns can be expected, there are opportunities for improving TTLs perception of these assessments by: (i) providing additional opportunities for the task team to meet with panel to discuss specific issues on which there is a difference of opinion; and (ii) generalize the use of brief Regional rejoinders for all instances where there is a disagreement between a team and a panel.

% N % N % N

Assessment Panel 94 47 98 60 100 10

• Panel skill mix 92 60 99 96 100 15

• Panel familiarity with country specific issues 69 59 88 95 87 15

• Panel fairness and professionalism 92 60 100 96 100 11

Assessment Approach 86 44 100 59 100 12

• Panel focused on relevant issues 88 57 100 95 93 15

• Were there major omissions 89 52 99 83 91 11

• Panel’s preparation 92 59 96 95 87 15

• Appropriateness of the time allotted for the interview 89 60 96 96 67 15

• Individual panelists views taken into account when rating the project NR NR 100 95 100 8

Assessment Process 90 47 98 49 100 12

• Fairness and transparency 88 60 98 95 100 15• Panel listening to the Task Team 85 60 99 95 100 15• Task team listening to the panel NR NR 96 95 100 15• Role of the moderator 88 59 97 76 87 15• Timely availability of background documents NR NR 95 95 NR NR• Quality of QAG administrative support 83 59 99 95 100 8

Usefulness of QSA 71 44 100 57 100 12

• To enhance accountability 77 59 96 95 93 15

• As a learning experience 72 60 100 95 100 15• To identify systemic issues 74 59 100 95 100 14

5. Overall QAG Performance 89 57 100 95 100 12

TL Panelists Observer

4.

3.

2.

1.

86 A QAG Assessment

Attachment A

FY01-02 QUALITY OF SUPERVISION ASSESSMENT Results of TTL Supplementary Survey

1. Was the FY01 and FY02 project budget for doing good quality supervision: FY01

About Right 66% Too Much 3% Too Little 31% NA 0%

FY02

About Right 71% Too Much 0% Too Little 29% NA 0%

2. If FY01and FY02 budget was too little, assume you were given addit ional $10,000 each FY.

Please distribute these additional budget resources among the expenditure items we list below.

Additional $10,000

Additional $10,000

FY01 FY02 (a) More frequent missions and/or longer field visits 34% 30% (b) Hiring additional technical inputs 34% 37% (c) Ensuring compliance with Bank's fiduciary and safeguard

polices: i) social 4% 4% ii) environment 1% 2% iii) financial management 6% 6% iv) procurement 6% 7% v) M & E 15% 12% (d) Other (specify) 0% 2%

3. Sector Manager (a) Guidance and support provided by Sector Manager to facilitate your supervision of this

project? HS 19% S 70% MS 9% U 2%

(b) Timeliness of Sector Manager’s decisions on critical supervision issues?

HS 21% S 71% MS 6% U 2% (c) Timeliness of Sector Manager’s feedback on PSR and post-mission follow up activities?

HS 13% S 78% MS 5% U 4%


4. Country Department (a) How well did the country department (Country Director and/or Country Anchor/Portfolio Manager) use the CPPR or other venues with the Borrower to resolve problems affecting this project?

HS 20% S 63% MS 16% U 1%

In carrying out your supervision duties, how has new Bank IT affected the quality of your 5. supervision work?

Level of Satisfaction

System

Satisfied Unsatisfied N/A SAP 50% 43% 7% PSR 52% 46% 2% IRIS 60% 25% 15% Business Warehouse 27% 21% 50% Other 33% 11% 56%

6. Usefulness of PSRs (a) As TTL, do you find that the PSR in its present form useful? Yes 58% No 42% NA 0% (b) In your experience, do managers use the PSR? Yes 53% No 45% NA 2%

(c) Is the PSR user friendly?

Yes 33% No 67% NA 0%

7. Monitoring and Evaluation

(a) Did you have sufficient resources to adequately implement and use an outcome focused M&E system for supervision?

Yes 31% No 61% NA 8% If not, which of the following was the main area where you would need support? (Select all

that apply)

22 In-Bank advise on M&E

17 External Consultants

21 More budgetary resources

20 improving your own M & E related knowledge Skills

88 A QAG Assessment

(b) Does your manager provide guidance in the effective use of M&E in supervision?

Yes 22% No 62% NA 16%

(c) In your opinion, do Borrowers find the M&E for Bank operations useful?

Yes 41% No 41% NA 18%

8. Does the country field office provide support for the supervision of your project?

Yes 83% No 10% NA 7%

9. If yes, please answer the following about the types and level of satisfaction of support provided to

you: Level of Satisfaction

Type of Support

Satisfied Uns atisfied N/A (a) General Task Management 71% 18% 11% (b) Procurement 68% 16% 16% (c) Environment 30% 22% 48% (d) Social Development 38% 11% 51% (e) Financial Management 71% 14% 15% (f) M&E 29% 36% 35% (g) Other 29% 21% 50%

10. Is decentralization having a positive effect on the supervision quality of your project? Yes 56% No 27% NA 17%

11. What suggestions would you and your team have to improve the quality of the Bank's supervis ion?

Please comment here:

12. Time Input: With regard to the QSA exercise, kindly estimate how much time your task team (TT) has spent in preparing for and participating in this process?

Hours: HL Hours 45 ACS Hours 16

Note: In the following five questions we ask you to rate the performance of QAG on a four-point rating scale

(HS = Highly Satisfactory, S = Satisfactory, M = Marginal, U = Unsatisfactory).

13. Assessment Panel: HS 13% S 81% M 4% U 2% (a) Panel skill-mix in relation to the project's type and complexity HS 10% S 82% M 7% U 1%


(b) Panel's familiarity with country specific issues?

HS 5% S 64% M 27% U 4% (c) Panel's fairness and professionalism

HS 25% S 67% M 8% U 0%

Assessment Approach: HS 18% S 68% M 14% U 0%

14.

(a) Panel focused on relevant issues?

HS 16% S 72% M 12% U 0% (b) Were there major omissions?

HS 10% S 79% M 11% U 0% (c) Panelists were prepared for the interview?

HS 24% S 68% M 7% U 1% (d) Time allotted for the interview was appropriate?

HS 17% S 72% M 10% U 1%

15. QSA5 Process: HS 13% S 77% M 10% U 0% (a) Fairness and Transparency of the assessment process?

HS 18% S 70% M 12% U 0% (b) Panelists listening to the Task Team's point of view?

HS 28% S 57% M 15% U 0% (c) Role of the Moderator?

HS 17% S 71% M 7% U 5% (d) Quality of QAG Administration support?

HS 14% S 69% M 10% U 7% (e) Did QAG adequately communicate logistics of the assessment?

HS 19% S 72% M 7% U 2%

90 A QAG Assessment

Usefulness of QSA5: HS 14% S 57% M 25% U 4%

(a) Usefulness of the QSA5 process overall for enhancing accountability in the Bank?

HS 14% S 63% M 17% U 6%

(b) Usefulness of the QSA5 assessment as a learning experience for the Task Team? HS 15% S 57% M 22% U 6%

16.

(c) Usefulness of the QSA5 process to identify systemic issues HS 8% S 66% M 22% U 4%

17. Quality of the QSA5 process overall HS 5% S 84% M 9% U 2%

Improvements: What suggestions would you and your team have to improve the quality of QSA including its learning aspects?

18.

Please comment briefly:

q s fy01/02 (qsa5) public disclosure authorized upervision...

Documents