Impact Evaluation of Labor-Based Road Work in the Pacific

ADB Pacific Department

Aaron Batten, Christopher Edmonds, Daisuke Misuzawa, and Craig Sugden

John Standingford, Lead Consultant, Transport Economist Lincy Pendeverana, Community Survey Expert

July 2012

CONFERENCE ON IMPACT EVALUATION: METHODS, PRACTICES, & LESSONS

The labor-based methods are a construction technology using labor substantially supported by limited equipment instead of heavy equipment handled by a little labor force. Such methods provide an important avenue for promoting poverty

alleviation in infrastructure development because they contribute to income generation and job creation in local communities—particularly in rural areas where there are few wage-based employment opportunities.

The LB methods have been used for small-scale infrastructure development, including road, irrigation, land development, forestry, water-supply, sanitation and small buildings.

In cooperation with partner governments in Pacific Developing Member Countries and funding from the Government of Japan, ADB-supported projects have employed labor-based methods for road maintenance and rehabilitation of unpaved rural roads in a number of PDMCs. The ongoing evaluation is assessing impacts of road maintenance and rehabilitation projects in Papua New Guinea, Solomon Islands, and Timor-Leste.

Impact Evaluation of Labor-Based Road Work in Pacific

What are the changes in living standards of households in communities located near labor-based and traditional capital-intensive road rehabilitation and maintenance projects?

Are communities near LB projects more likely to express positive attitudes toward the road projects?

Do workers employed in LB road projects show evidence of improved post-project employment experiences?

Do communities participating in LB projects demonstrate greater commitment to road maintenance subsequent to the completion of the project?

What types of social and environmental externalities associated with LB and tradition road maintenance and rehabilitation projects and practical importance in the communities and households neighboring road project areas?

Questions to be addressed in the evaluation

Study areas and basic methods

In Papua New Guinea and Solomon Islands: Survey research based impact evaluation with treatment and control groups tracked over time.

Key issues: defining comparator(s), rigorous sampling, logistics of conducting surveys in a timely manner with available resources

Leaving open possibility of ‘econometric controls’ if treatment and control groups appear to be non-comparable

In Timor-Leste: Piloting of a lower cost, simpler evaluation method that stakeholders can adopt for other rural development programs.

PNG: Select localities based on roads, every 5th house method (citation) Sampling groups, households proximate to: (1) LB Road

Maintenance and rehabilitation, (2) Road maintenance using capital-intensive methods, and (3) No road maintenance

SOL: GIS sample frame based on remotely sensed structures Sampling groups, households proximate to: (1) LB road

maintenance, (2) Road maintenance using capital-intensive methods, and (3) No road maintenance

TIM: Based on national population and housing census data for assisted aldeias (i.e. sub-villages) supplemented with focus groups discussion and targeted household surveys in assisted communities. Control group to be identified ex-post.

Sample selection, and definition of treatment and control groups

Household (individual enumeration at residence)

Community (focus group)

Business (individual enumeration at business)

Transport service provider (individual enumeration at

transport stops or terminals)

Road condition (direct observation and measurement

along roads in the study areas)

Surveys collected in PNG and SOL

Treatment 2:

Mulitaka

Treatment 1:

Eagel-Kero

Control 1:

Recent

rehabilitation

Control 2: No

rehabilitation

Papua New Guinea Study

Two roads in adjacent similar areas on the outskirts of Honiara.

One road being upgraded under ADB supported Second Road Improvement Project, while the other is to be rehabilitated entirely by government.

Sample selected based on remotely sensed census of structures in the areas located near the two roads, as validated in the early field work for the study.

Solomon Islands Study

SOL Summary of Household Interviews

Constellation PSU

Google

Earth

pins

House-

hold

inter-

views Ratio

3 11 4 36%

4 40 22 55%

5 46 23 50%

6 30 10 33%

7 44 10 23%

9 46 19 41%

10 31 14 45%

11 30 10 33%

12 26 10 38%

13 34 16 47%

16 61 19 31%

17 35 12 34%

18 22 0 0%

19 28 5 18%

23 116 39 34%

20 67 31 46%

21 48 15 31%

22 83 28 34%

Total 798 287 36%

St Martin Road

'The Y'

Black Post

Road (Central)

Black Post

Road (South)

The $3m Our Roads Our Future project (Grant 9142), funded from the Japan Fund for Poverty Reduction, is extending earlier pilots of community-based infrastructure delivery

The budget for monitoring is small, and the method would ideally be one that is simple enough for government agencies and other stakeholders to adopt

Timorese communities are ‘over surveyed’, and the government has asked partners to maximize the use of government data and minimize one-off surveys

The control group cannot be set ex-ante, as there are many rural infrastructure programs underway and their future location is not known

Timor-Leste Study

Baseline data has been prepared at the aldeia (ie sub-village level) from the 2010 population and housing census, which hopefully will be updated by the 2015 census

Changes in living standards to be measured by changes in asset holdings rather than more costly income/ consumption data

Relative progress of assisted and unassisted groups to assessed by changes in ranking against a village level asset index

Additional context drawn from socio-economic data of the census (eg access to basic services) and targeted information collection

Timor-Leste Study

The PNG, SOL, and TIM governments are supportive to provide data and documentation.

Recruitment of consulting services and procurement of data collection are conducted in a timely and effective manner.

The timing of road project implementation, enable surveys to be conducted pre- and post-project completion within the suggested timeframe for the IE study.

Key Assumptions

Non-response and/or logistical difficulties in conducting survey make it difficult to collect the data required for the IE on a timely basis.

Insufficient impacts from the road maintenance and rehabilitation projects accrue to households and communities over roughly a 6-month period to enable measurement of significant differences across baseline and follow up surveys in treatment and control households and communities.

Implementation risks

How comparable are project sites?

Ethical concerns - collection of data in ‘non-

treatment’ villages can lead to expectations from residents

Implementation risks

Duration of IE study – long term nature of changes study

attempts to measure

Costs of data collection, particularly on PARD overhead

(staff time, staff travel) not covered by the IE RETA

Uncertain outcomes – costs and benefits of undertaking

rigorous IE studies

IE consumer appreciation of the difference between

results of rigorous IE study versus more typical

process/implementation evaluation

Expected differences and pros/cons between the team’s

approach and Timor Leste approach using census data

Key issues looking ahead