THE WORLD BANK February 2015

Afghanistan Third Party Monitoring

1

The Afghanistan ProgramThe Context:– The Afghanistan program consists of both IDA and trust fund resources;– Total active portfolio: $3.2 bn– The Afghanistan Reconstruction Trust Fund (ARTF) started in 2002 to

provide recurrent cost financing for the Afghan Government’s operating budget, funding non-security wage bill and O&M;

– ARTF: $7.9 bn paid in since 2002 by 33 donors, – ARTF: $7.4 bn committed, $6.3 bn disbursed to gov for recurrent and

investments; active portfolio $2.4 bn ; – IDA: $2.7 bn committed, $2.3 bn disbursed; active portfolio $767m– Current active portfolio: 31 projects (11 IDA, 15 ARTF and 5 co-financed).

2

Why TPM?• Afghanistan faces several challenges, including:

– Low capacity environment;– Difficult access; – Challenging security context;– Rough terrain, complex geological structure, disperse and remote project sites;– A large trust fund program with high donor scrutiny and pressure for accountability. Need to strengthen supervision, access data from the field, report to donors.

• The ARTF Supervisory Agent – key characteristics:– Carries out asset verification, quality assurance, and data mapping of national infrastructure

projects;– Has conducted a total of 11,239 site visits since the program start in 2001;– Provides the Bank with strengthened outreach to all 34 provinces of Afghanistan, including

highly insecure areas; – Focused on infrastructure quality and safeguards issues in education, rural roads, irrigation

and community development;– Monitored 4 projects worth a total of US$2.25 billion;– Increasing economies of scale/program efficiencies: Cost per site visit: Year I: $4600, Year IV:

$2060– Developed feedback systems to ensure deviations are rectified and construction quality is

improved, resulting in increased ministry appreciation of monitoring value;– Collaborative efforts with the ministries helped to improve program performance and foster a

mutual push for better projects. 3

To Consider before/during Preparation?• Be clear on objective but ensure room for flexibility: Afghanistan SA

started off with infrastructure, but now branching into other areas such as safeguards and FM. Initial focus was on reporting to donors but shifting the focus towards quality assurance during construction and beneficiary input;

– TPM by theme: The objective, focus and use of the reporting differs for infrastructure, financial management, livelihoods, social and env. Safeguards etc.

– Different objectives: TPM monitors social outcomes in Health.

– Audit vs ??:“Normally” TPM refers to an audit agency focusing on fiduciary aspects. Afghanistan focuses on quality assurance, verification and access to data – different skills needed, different contracts;

4

Continued…• What agency to hire?: The credibility of the third party is critical but will

often depend on context. In– International contractor was needed to ensure donor buy-in;– Government more comfortable with local counterparts; – The right set-up, sophisticated systems were needed;– in Afghanistan compromise with international contractor, expat

oversight and local counterparts• Contracting: The Bank has very limited experience with Bank

contracting directly – this takes preparation; • TPM ownership – Gov/Donors/Bank?:

– Ideally we want gov and Bank to co-own to ensure oversight and buy-in and therefore sustainability of investment;

– Donors would like to own but sensitive - we need balance in what can be shared. 5

The SA - Monitoring Approach(es)• The database, CMORE, is central to everything: data collection, monitoring, security tracking, and information

processing;• Smartphone technology allows for engineers to work in austere and dangerous areas;• Risk management and mitigation is done through multiple tiers of control including GPS verification of staff

locations and projects;• Information is published in an online database and mapping application searchable catalogue that geospatially

displays data; • Detailed inspection reports, analysis of identified patterns, recommended actions shared with Bank and ministries; • Actions to address deviations are agreed and correct systems to prevent similar problems from reoccuring and

increase impact on the ground.

Two Methodologies:Third Party Monitoring• SA field engineers travel to sites and complete comprehensive reports.• Pros: High level of detail, skilled engineers on site, complete coverage of the project including social and other

data,;• Cons: Cost to travel to site, sites are seen once or twice per year.Local Monitoring• Community member selected to send daily information to SA/ministry in Kabul. Engineer provides oversight;• Pros: Low cost, increases community buy-in and support, information comes in near real-time, deviations and

rectifications can be seen immediately.• Cons: Monitoring is fairly simple as the members are not engineers, limited to projects in or near the community.

6

How it works?

7

The Process in Pictures

8

Lessons Learned• Cooperation and collaboration with the GOA ministries is essential and includes monthly meetings,

weekly reporting, joint field visits, and joint work plans.• New to both Bank and ministries, learning curve to be expected. Key challenge to develop systems;• Critical to convince ministries of program value – should not be seen as “policing function” but rather

input for collaboration. The TPM contractor key in this regard to strike right balance and communicate well.

• TPM allows the Bank to access data from all over the country, benefitting monitoring of both quality of infrastructure, output/outcome results and project effectiveness;

• Multifaceted dialogue builds awareness within ministries on issues related to program monitoring and infrastructure quality assurance;

• Helps build capacity within the government by “raising the bar” on monitoring and reporting standards;

• Provides donors, the Bank and the government with a credible system of data collection and verification which contributes to effective policy making and ARTF credibility;

• Addresses individual needs within CDD, transport, irrigation, livelihoods, education and health;• Economy of Scale works, more project sites lower cost per site visit.

9

Challenges:• Expensive investment. Cost decreasing over time though due to efficiencies. Working

towards cost sustainability: introducing new component for community based monitoring;• Significant potential risk if data is not used by bank and ministry teams;• A flexible contractor is important to ensure we can respond to emerging needs – safeguards

an example in ARTF program.• TPM to carry on in coming years but will continue to evolve in response to needs in the

program and context – will have to align with government’s monitoring vision as it evolves;• Management of relationship with government technical team, tendency for being defensive,

sometimes misunderstood as policing.

Challenges

TPM in Transport ProjectsProjects:NERAP- US$232million, closed Dec 2013, covered 32/34 provinces, over 200/368 districts, over 500 contracts;ARAP-US$332 million, ongoing, covers 33/34 provinces, over 250/368 districts; over 600 rehab. contracts and over 500 maintenance contracts;

Justifications for TPM:• Dispersed and remote project sites, impossible for Bank team to monitor all project sites;• Sub-projects in almost every province including in secure provinces, not cleared for Bank staff to visit;• Expensive investment, ensure the right quality and scope;• Complex engineering structures require special expertise to monitor;• Feasible for such type of contract, economy of scale;

TPM Scope:• Visual quality evaluation of the roads, bridges and structures against defined technical specifications;• Evaluation of the construction methodologies used by the contractor;• Assessment of the contractor’s resources on site;• Review of the engineering designs against site and identifications of design faults for improvement;• Visual check of the quality of construction material;• Evaluation of contractor’s performance;• Other ad-hoc check and data collection as per the task team request.

11

TPM in Transport Continued..Advantages:

• Over 1266 site visits by TPM engineers since inception, which otherwise not possible with traditional bank supervision;

• Significant Improvement in overall sub-projects quality;• Each rehabilitation sub-project gets at least one visit ( 100% representation);• Reliable and accessible data from the site;• Opportunity for remedial actions before too late;• Knowledge sharing between TPM-Consultant and government engineers;• Better control of problem sub-projects and prospect for timely decision making;• Flexibility for ad-hoc and special data collection from site;• Screening contractors and identifying the frequent bad performers;

• Feasibility of special analysis of data and preparation of analytical reports;

• Weekly technical meetings to discuss deficiencies, deviation and agree on remedial actions

12

Challenges:• Security risk still an issue;• Cost of monitoring, sustainability;• Availability of sites for monitoring; always need for a bunch of project to

justify the cost;• Sometimes communication between TPM and government engineers

becomes an issue; culture of being defensive;• Ownership of the government, not yet fully accepted and recognized;• Quality of reports and findings highly depend on capacity of the monitoring

engineers on site; counter check has been challenging;• TPM services not mature in the country, limited number of qualified

consultants, local consultants not yet developed as a result high cost for service

• Cost per sites is not a comprehensive indicator; should be linked to the size of investment; linked to performance of the project ( quality, disbursement, no. of deviations, on time completion of sub-projects) , etc.

TPM in Transport Continued..

13

TPM in Education SectorProject:Education Quality Improvement Project (EQUIP) II is a USD 460 million basic/secondary education project. Coverage is nationwide (insecure and secure provinces). Project is implemented entirely using government systems right up to community level. School shuras are responsible for the majority of school construction.

TPM Scope Three functions: implementation support; capacity building; external supervision. • Works with clients to monitor progress with meeting outputs/objectives;• Builds capacity within governments by working closely with government engineering departments;• Provides due diligence in project implementation (compliance with technical standards/physical

audit/safeguards compliance etc.)

Justification for TPM:• Country context : highly insecure, governance-challenged environment• Bank capacity: limited ability to conduct field visits• Community capacity: responsible for school construction but quality can be mixed; • Government Capacity: lack of qualified engineers, weak monitoring systems, weak or non-existent technical

standards (e.g., DRM)14

TPM in Education continued..Advantages:

• Complements Bank supervision and Government monitoring;

• “Raises the bar” by having high monitoring standards;

• Brings in new technology (satellite imagery, mobile phone apps) as well as innovations in monitoring (community monitoring).

• Captures knowledge across the country as well as across the Bank portfolio. Allows client/Bank to analyze systemic deficiencies/trends and put in place mitigation measures.

• TPM can be contracted to conduct monitoring of physical assets (IRD) and/or social indicators (Johns Hopkins - health service delivery.

• Provides independent verification of government systems/findings (“ghost schools”);

• Direct and indirect capacity building of counterpart agencies.

15

Challenges:• Linking physical/fiduciary audits would make system more effective; • Payment to TPM (per site) is much higher than what government or Bank

teams have for monitoring;• Institutional model of TPM varies and some are more effective than others;• Contractual accountability: TPM in the long term should be contracted by

client and accountable to the client.• TPM is an independent monitor but is dependent on government

counterparts for information, access to project sites, etc. TPM is only as relevant as the quality of information/cooperation they get from government.

• Regulatory environment of the country can influence the effectiveness of TPM

TPM in Education continued..

16

TPM in Rural Development SectorProject Third National Solidarity Project: 2010- sept 2016: (US$1.5 Billion) to build, strengthen and maintain Community Development Councils (CDCs) as effective institutions for local governance and socio-economic development.• Est. over 41,000 elected Community Development Councils (over 90% of rural

villages) constructed over 85,000 sub-projects (US$1.5 in block grants over 10 years)

TPM Scope and advantagesSample size: Total of 800 TPM sites this year (<1%)

Provide assurance: • 34 ARTF donors, complaints handling tool for WB team,

Evolved from physical verification after completion towards enhancing quality construction during the 4 stages of construction • Results: changed technical engineering designs, learning during site visits for

communities and project engineers, corrections in poor construction etc.); 17

TPM in Rural Development continued..Monitor the MIS• High reliance on NSP’s database (by WB, Donors, GoA)• Serve as a means to verify the integrity and accuracy of NSPs own data

collection and MIS by the FPs and NSP monitors;

High risk areas (for WB, NSP monitors and FP engineers)• No govt. presence in district (No District Governor & No District Police Chief,

functions from other nearby village or provincial center), Located on insurgency corridor; CDC members can’t disclose their identity in other villages or officials; Travel to district center is extremely risky; Travel with project documents is impossible; Permanent AGEs check posts; Regular military operations)

Fiduciary monitoring • A first pilot (100%);• Move the focus away from communities towards the systems at the Province;• FM gives greater autonomy to NSP;• Contribute to greater fiduciary assurance and identify gaps for NSP IV design

18

Challenges and lessons learnt:• Requires full commitment from the WB Task team to manage it

(provide data, context, manage relationships, regular coordination etc)

• Reporting needs to be backed up by local context knowledge (case of latrines, construction material, safeguards, grading for CDD type of sub-projects);

• Local paid monitors and the community voluntary model;• Standard approach to TPM changed to suite individual project needs

(Not all ministries need capacity building); • Building trust and a good working relationship between the Project

team and TPM agent takes time

TPM in Rural Development continued..

19

Social Safeguards in TPMGradual inclusion of safeguards and gender• Safeguards monitoring needs:

Original contract for TPM did not include safeguards and gender Development of project specific questions regarding safeguards and gender Close cooperation btw IRD and project staff to develop questions IRD’s engineers were trained on social safeguards to understand issues.

• The key safeguards questions/output indicators: Availability and implementation status of safeguards instruments at sub-projects

site, (i.e. ESMP, abbreviated RAP with strip plan) Records of consultation/training sessions with communities, including PAFs (i.e.

number of consultation and training sessions to date, etc.) Functionality of grievances handling (i.e. number of grievances submitted/handled,

key issues, etc.) Gender data (i.e. consultation meeting with project affected women, girls

enrollment ratio, female staff, female CDC, etc.) Environmental Safeguards (i.e. safety measures, safety equipments,

quarry/borrow pit management, etc. )

Benefits and use of data

In addition;• Project staff will report follow up actions on TMP to WB in monthly format

• TPM findings help us to discuss shortcomings with project implementers during WB supervision implementation mission.

• The implementing partners’ safeguards staff are now more accountable to communities, particularly to affected people with regards to land issues.

Follow up of IRD TPM findings:

TPM findings Follow up actionsProject level GRC is in place-but no grievances were recorded”.

Public awareness program among the affected communities regarding GRM services. (ARAP about 187 grievances submitted- Last mission findings).

No sub-project had evidence of women being consulted during planning.

Recruitment of female staff to conduct consultation with project affected women.

Land for a number of schools through community donation were not proper location. The donated landwere low-lying and or flood plain areas, which are prone to flooding.

Site selection criteria developed and are beingapplied.

Information on affected land/assets (i.e. size with number of PAFs) and land disputes information.

Those safeguards figures, which are being reported is very helpful.

Challenges & opportunities Challenges:• The PMUs’ staff was initially a bit defensive, and hesitant to accept findings of TPM

related to social safeguards.

• Strengthening relational btw the implementing agencies and the TMP is crucial for success. And the implementing partners shall provide response and feedback to TPM reports.

• Limited capacity of TPM engineers on social safeguards. Only reporting on availability of reporting data but no qualitative information on process.

Opportunities:• TPM has strengthened safeguards performances, and it has helped PMUs

safeguards team to be more accountable to communities and WB.

• The TPM experiences, particularly use of smart phone has proved very useful in fragile situation.