11

Monitoring and Monitoring and Evaluation in ECA Evaluation in ECA RegionRegion

Land Thematic Group RetreatLand Thematic Group Retreat

November 19-20, 2007November 19-20, 2007

22

ECA OverviewECA Overview

ECA Region – 29 countries ranging ECA Region – 29 countries ranging from recent EU entrants to IDA from recent EU entrants to IDA countries.countries.

Several post-conflict countries..Several post-conflict countries.. Post-communist transition to market Post-communist transition to market

economies.economies. Land records universally out-of-date Land records universally out-of-date

and neglected. and neglected. Privatization, de-collectivization, Privatization, de-collectivization,

restitution issues to be resolved. restitution issues to be resolved.

33

ECA OverviewECA Overview

Project TypeProject Type ClosedClosed ImplementatioImplementationn

PreparationPreparation

Land reform, Land reform, agriculture, agriculture, farm farm restructuringrestructuring

55 33 11

Registration, Registration, CadastreCadastre

55 1212 11

Land Use Land Use ManagementManagement

11 11

44

ECA Overview cont.ECA Overview cont.

Approximately 30 projects in some 20 Approximately 30 projects in some 20 countries.countries.

Supporting governments to restructure state Supporting governments to restructure state and collective farms, privatize land, create and collective farms, privatize land, create modern cadastre and registration systems. modern cadastre and registration systems.

More recent projects also trying to support More recent projects also trying to support development of better land policy, state development of better land policy, state land management and land use planning. land management and land use planning.

The overall objective is to better develop The overall objective is to better develop land markets to support investment and land markets to support investment and more economic development.more economic development.

55

Monitoring vs. EvaluationMonitoring vs. Evaluation MonitoringMonitoring

– Throughout project implementation. Throughout project implementation. – To improve performance, aid in decision To improve performance, aid in decision

making, improve accountability, and enhance making, improve accountability, and enhance learning. learning.

– All current projects are regularly monitored.All current projects are regularly monitored. Evaluation Evaluation

– What did the project achieve? And what What did the project achieve? And what difference did it make?difference did it make?

– ICRs are completed at the end of each project. ICRs are completed at the end of each project. – But very few (if any?) impact evaluation carried But very few (if any?) impact evaluation carried

out in the ECA region on land projects.out in the ECA region on land projects.

66

Monitoring in ECA Land ProjectsMonitoring in ECA Land Projects

Early projects (farm restructuring and Early projects (farm restructuring and registration/cadastre) heavily focused on inputs registration/cadastre) heavily focused on inputs and outputs:and outputs:– Number of titles issuedNumber of titles issued– Crop yieldsCrop yields

Monitoring in ECA land administration projects Monitoring in ECA land administration projects still focused on outputs (number of registrations), still focused on outputs (number of registrations), but more data being collected on land markets but more data being collected on land markets (price/m(price/m22; mortgage market) and customer ; mortgage market) and customer satisfaction. Moving towards outcomes and satisfaction. Moving towards outcomes and impacts.impacts.

In many countries difficulties with collection of In many countries difficulties with collection of baseline data.baseline data.

77

Monitoring IndicatorsMonitoring Indicators

Difficulties of comparability and consistency Difficulties of comparability and consistency of indicators across projects and countries.of indicators across projects and countries.

Often indicators are overly ambitious; too Often indicators are overly ambitious; too many indicators or the data is not available.many indicators or the data is not available.

Indicators at appraisal may no longer be Indicators at appraisal may no longer be relevant at completion.relevant at completion.

Important that monitoring data is used by the Important that monitoring data is used by the project – PIU, implementing agencies – for project – PIU, implementing agencies – for decision making and understanding.decision making and understanding.

Not just for reporting to the Bank team.Not just for reporting to the Bank team.

88

EvaluationEvaluation Completed projects in countries across the Completed projects in countries across the

ECA region from Slovenia to Tajikistan.ECA region from Slovenia to Tajikistan. ICRs for 7 projects – 4 registration and ICRs for 7 projects – 4 registration and

cadastre projects; 3 land reform/farm cadastre projects; 3 land reform/farm restructuring projects.restructuring projects.

ProjectProject RatingRating IEGIEG SustainableSustainable IEGIEG

Kazakhstan Registration Pilot (2000)Kazakhstan Registration Pilot (2000) SS MSMS LL UU

Russia LARIS (2003)Russia LARIS (2003) SS SS LL LL

Armenia Title Registration (2004)Armenia Title Registration (2004) HSHS HSHS HLHL HLHL

Slovenia Registration (2005)Slovenia Registration (2005) SS SS LL LL

Romania General Cadastre (2006)Romania General Cadastre (2006) SS HSHS LL LL

           

Azerbaijan Farm Privatization (2003)Azerbaijan Farm Privatization (2003) HSHS HSHS HLHL HLHL

Tajikistan Farm Privatization (2005)Tajikistan Farm Privatization (2005) SS SS LL LL

99

Lessons LearnedLessons Learned

Institutional:Institutional:– Need political will to address issues of land policy Need political will to address issues of land policy

and institutional reform.and institutional reform.– This kind of reform takes time and does not This kind of reform takes time and does not

always fit the Bank’s schedule – but if we stick always fit the Bank’s schedule – but if we stick with it, we can achieve results (LARIS, Romania).with it, we can achieve results (LARIS, Romania).

– Single agencies work best, but are not always Single agencies work best, but are not always feasible; do we have a back-up plan?feasible; do we have a back-up plan?

– Supporting the institutions to move from output Supporting the institutions to move from output based, command and control to service focused based, command and control to service focused and regulatory.and regulatory.

1010

Lessons LearnedLessons Learned

Capacity Building:Capacity Building:– Necessary to building human and Necessary to building human and

institutional capacity to carry out and institutional capacity to carry out and sustain the reforms in the land sustain the reforms in the land administration sector.administration sector.

– Also must build capacity to manage the Also must build capacity to manage the project and significant large contracts project and significant large contracts (IT).(IT).

– Not just training, but professionalization Not just training, but professionalization of staff.of staff.

1111

Lessons LearnedLessons Learned Automation and Information TechnologyAutomation and Information Technology

– IT can play a role in helping to improve efficiency IT can play a role in helping to improve efficiency of the land administration agencies, but it is not a of the land administration agencies, but it is not a cure-all.cure-all.

– Not just a HW/SW issue – includes business Not just a HW/SW issue – includes business process reengineering, understanding different process reengineering, understanding different users and their requirements.users and their requirements.

– Both the Bank and the client continue to Both the Bank and the client continue to underestimate the complexity and management underestimate the complexity and management requirements of these large IT implementations.requirements of these large IT implementations.

– Have had better success with incremental, home Have had better success with incremental, home grown solutions (Moldova, Kyrgyzstan), than with grown solutions (Moldova, Kyrgyzstan), than with ‘big bang’ approach (Romania, Bulgaria).‘big bang’ approach (Romania, Bulgaria).

1212

Lessons LearnedLessons Learned

TechnicalTechnical– Use of private sector surveyors has been Use of private sector surveyors has been

positive – more effective than public positive – more effective than public sector.sector.

– Competition/ICB brings prices downCompetition/ICB brings prices down– Use of new technologies, GPS, OPM, Use of new technologies, GPS, OPM,

CORS, means surveys can be done cheap CORS, means surveys can be done cheap and accurate; $10/parcel or less!and accurate; $10/parcel or less!

– Encouraging use of sporadic registration Encouraging use of sporadic registration in urban areas.in urban areas.

1313

Building Land MarketsBuilding Land Markets

But do we really know the impact?But do we really know the impact? Anecdotal evidence:Anecdotal evidence:

– In 2001, the National Bank of Romania In 2001, the National Bank of Romania recorded 1,948 mortgage loans; in 2005 recorded 1,948 mortgage loans; in 2005 they recorded 82,675 mortgage loans they recorded 82,675 mortgage loans

– In Slovenia real estate taxes increased In Slovenia real estate taxes increased by 15% between 1999 and 2004.by 15% between 1999 and 2004.

– In Armenia the number of real estate In Armenia the number of real estate transactions more than doubled transactions more than doubled between 2000 and 2003.between 2000 and 2003.

1414

Poverty AlleviationPoverty Alleviation

ECA projects have not promised too ECA projects have not promised too much in this area – but it is clear that much in this area – but it is clear that we do not know the poverty impacts we do not know the poverty impacts our land projects have.our land projects have.

Is this an area for future evaluation?Is this an area for future evaluation?

1515

TransparencyTransparency

Transparency is important for land Transparency is important for land administration systems – part of good administration systems – part of good governance.governance.

But difficult to measure... How do we But difficult to measure... How do we monitor increased/improved monitor increased/improved transparency?transparency?

Is transparency also a question for the Is transparency also a question for the monitoring and evaluations systems? Is monitoring and evaluations systems? Is the monitoring data shared publicly? the monitoring data shared publicly? Should it be and how?Should it be and how?

1616

Assessing PerformanceAssessing Performance Projects seek to improve delivery of Projects seek to improve delivery of

services to clients. services to clients. Regular customer surveys allow us to Regular customer surveys allow us to

measure performance of the LA agencies measure performance of the LA agencies and change in customer attitudes over and change in customer attitudes over time.time.

Regular customer surveys performed in Regular customer surveys performed in Serbia, Macedonia, Croatia, Kyrgyz Serbia, Macedonia, Croatia, Kyrgyz Republic.Republic.

Currently data is used project by project – Currently data is used project by project – can we use across projects to talk about can we use across projects to talk about results?results?

1717

Assessing Performance -ExampleAssessing Performance -Example

Continuing successful implementation of the Land Continuing successful implementation of the Land Real Estate Registration Project in Kyrgyz Real Estate Registration Project in Kyrgyz Republic.Republic.

Further support to the implementing agency – Further support to the implementing agency – Gosregister – from BNPP to build capacity within Gosregister – from BNPP to build capacity within Gosregister to measure performance in terms of Gosregister to measure performance in terms of stakeholder feedback.stakeholder feedback.

Specifically to develop capacity and tools for staff Specifically to develop capacity and tools for staff to measure client satisfaction and derive to measure client satisfaction and derive evidence-based recommendations; and conduct evidence-based recommendations; and conduct an initial survey on quality of services delivered an initial survey on quality of services delivered and collect baseline data in order to measure and collect baseline data in order to measure improvements over time.improvements over time.

Further information: Further information: http://go.worldbank.org/PYKD1UU640http://go.worldbank.org/PYKD1UU640

1818

Future PlansFuture Plans

Croatia will do a follow-up study/evaluation Croatia will do a follow-up study/evaluation of staff who participated in the training of staff who participated in the training program over the course of the project program over the course of the project (2004-2008).(2004-2008).

Consultant working on a desk review of Consultant working on a desk review of monitoring and evaluation of land projects monitoring and evaluation of land projects in ECA. Report should be ready by early in ECA. Report should be ready by early 2008.2008.

Bring together client survey results?Bring together client survey results?