pcm - project cycle management, training on evaluation

1

Project Cycle Management

Training Course on "Evaluation"

2

Definitions

Evaluation is the systematic assessment of the worth and merit of some object.

Evaluation is the systematic acquisition and assessment of information to provide useful feedback about some object.

3

Evaluation: A DefinitionAn evaluation is an assessment, as systematic and objective as possible, of an on-going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, develop-mental efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both partner countries and donors.

OECD, DAC, 1991

a

4

The Project CycleKey characteristics: breaks down the life of a project/

programme into phases defines key decisions, information

requirements, & responsibilities at each phase

each phase must be completed for the next to be tackled with success

allows for corrective measures during implementation, based on monitoring and mid-term evaluation

draws on evaluation experience for the design of future programmes & projects

a

5

When to do evaluations?

Mid-term Mid-term evaluationevaluation

End of End of project/ project/ programme programme evaluationevaluation

Ex-post Ex-post evaluationevaluation

During implemen-tation

At the end of implemen-tation

After imple-mentation

Also used as term: Ex ante evaluation

a

6

Evaluations: What for?

To improve

Decision making

Resource allocation

Accountability

Through:

Informing the public

Informed key decision-making processes

Encouraging on-going organisational learning

7

Logframe Basics

‘... IF results are delivered, AND assumptions hold true, THEN the project purpose will be achieved ...’

Objectively Verifiable Indicators

Intervention Logic

Sources of Verification

Assumptions

Overall Objectives

Project Purpose

Results

Activities Means Cost

Pre-condi-tions

8

Good indicators

Do partners and stakeholders agree that this indicator makes sense to use?

Owned

Will the data have utility for decision-making and learning?

Useful

Can data be collected easily, on a timely basis at reasonable costs?

Accessible

Is the data consistent or comparable over time?Reliable

Is the definition precise and unambiguous about what is to be measured?

Objective

Does the indicator directly represent the objectice it is intended to measure?

Valid

9

Indicators: An Example

Objective: Pollution load of wastewater discharged into the Blue river is reduced

Select the indicator: Concentration of heavy metal compounds (Pb, Cd, Hg)

Define the targets:• Define the quantity: Concentration of heay metal compounds (Pb,

Cd, Hg) is reduced by 75% compared to year x levels …(particular attention should be paid to the availability of baseline information)

• Define the quality: ... to meet the limits for irrigation water ...• Define the target group: ... , used by the farmers of Blue village, ...• Define the place : ... in the Blue river section of the District ...• Determine the time: ... 2 years after the project has started

10

Special challenges facing the donor agencies (1)

Work in many different countries and contexts

Have a wide diversity of projects in multiple sectors

Often focus on capacity building and policy reform, which are harder to measure than direct service delivery activities

11

Special challenges facing the donor agencies (2)

Are moving into new areas such as good governance, where there's little performance measurement experience

Often lack standard indicators on results that can easily be aggregated across projects

Are usually only minor actors affecting impacts, with subsequent problems in attributing them to their agency's activities

12

Special challenges facing the donor agencies (3)

Typically rely on outcome and impact data collected by partner countries, who have limited technical capacity and resources, with subsequent quality, coverage and timeliness problems.

13

Criteria for Evaluating Development Assistance

Relevance Efficiency Effectiveness Impact Sustainability

14

The Logframe and the 3 basicevaluation criteria

Overall Objectives

Project Purpose+ Assumptions

Results+ Assumptions

Activities+ Assumptions

+ Pre-conditions

allocation

action

utilisation

change

Efficiency

Means

Impact

Overall Objectives

+ Assumptions

Results+ Assumptions

Activities+ Assumptions

allocation

action

utilisation

change

+ Pre-conditionsMeans

Overall Objectives

Project Purpose+ Assumptions

Results+ Assumptions

Activities+ Assumptions

allocation

action

utilisation

change

+ Pre-conditionsMeans

Effective-ness

Project Purpose

15

Logframe and basic evaluation criteria: Major questions

Overall Objectives

Project Purpose+ Assumptions

Results+ Assumptions

Activities+ Assumptions

+ Pre-conditions

allocation

action

utilisation

change

Means

Efficiency

Effective-ness

Impact

How were inputs and activities converted into Results?

How well did the Results contribute to the achievement of the Project Purpose?

Which benefits on society and sector?

16

The 4th evaluation criterion: Relevance

Overall Objectives

Project Purpose+ Assumptions

Results+ Assumptions

Activities+ Assumptions

+ Pre-conditions

allocation

action

utilisation

change

Means

Efficiency

Effective-ness

Impact

Quality of planning and adaptation, including relevance of problems to correct beneficiaries, OVIs, means, cost, assumptions, risks

How were inputs and activities converted into Results?

How well did the Results contribute to the achievement of the Project Purpose?

Which benefits on society and sector?

Relevance

17

The 5th evaluation criterion: Sustainability

Have and will services and benefits be maintained?

Quality of planning and adaptation, including relevance of problems to correct beneficiaries, OVIs, means, cost, assumptions, risks

How were inputs and activities converted into Results?

How well did the Results contribute to the achievement of the Project Purpose?

Which benefits on society and sector?

Overall Objectives

Project Purpose+ Assumptions

Results+ Assumptions

Activities+ Assumptions

+ Pre-conditions

allocation

action

utilisation

change

Means

Efficiency

Effective-ness

Impact

Relevance

Sustain-ability

18

Summary: The 5 evaluation criteria

The fact that the results were obtained at reasonable cost, i.e. the cost, speed and management efficiency with which Means/Inputs and Activities were converted into Results, and the quality of the Results achieved.

Efficiency

The effect of the project on its wider environment, and its contribution to the wider objectives summarised in the Overall Objectives.

Impact

The appropriateness of project objectives to the real problems of the intended beneficiaries that it was supposed to address, and to the physical and policy environment within which it operated.

Relevance

The likelihood of a continuation in the stream of benefits produced by the project after the period of external assistance has ended.

Sustainability

How well the Results contribute to the achievement of the Project Purpose, and how Assumptions have affected project achievements.

Effectiveness

19

Relevance= The extent to which the aid intervention is suited to the priorities and policies of

the target group, partner country and donor

Possible questions: To what extent are the objectives of the programme still valid? Are the activities and outputs of the programme consistent with the

broader objectives and do they contribute to the attainment of these objectives?

Are the activities and outputs of the programme consistent with the intended impacts and effects?

… … …

20

Efficiency= Efficiency measures the outputs – qualitative and quantitative – in relation to the

inputs. It is a term which signifies that the aid uses the least costly resources in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted

Possible questions: Were the activities cost-efficient? Were objectives achieved on time? What were the major factors influencing the achievement or non-

achievement of the objectives? … … …

21

Effectiveness= A measure of the extent to which an aid intervention attains its objectives

Possible questions: To what extent were the objectives achieved/are likely to

be achieved? What were the major factors influencing the achievement

or non-achievement of the objectives? … … …

22

Impact= The positive and negative changes produced by an intervention, directly or

indirectly, intended or unintended.

Possible questions: What has happened as a result of the programme or project? What real difference has the intervention made to the

beneficiaries? How many people have been affected? …… …… ……

23

Sustainability= Sustainability is concerned with measuring whether the benefits of an

intervention are likely to continue after donor funding has been withdrawn.

Possible questions: To what extent did the benefits of a programme or project continue

after donor funding ceased? What were the major factors which influenced the achievement or

non-achievement of sustainability of the programme or project? … … …

24

Principles of evaluation Integrative part of planning and management: closely connected to decision

making processes Impartiality & independence of the evaluation process in its function from the

process concerned with policy making, the delivery and management of assistance (= separation of evaluation and responsibility for the project/ programme/policy)

Credibility depending on expertise and independence of the evaluators & transparency to be seeked through an open process, wide availability of results, distinction between findings and recommendations

Usefulness: relevant, presented in a clear and concise way, reflecting the interests and needs of the parties involved, easily accessible, timely and at the right moment improved decision-making

Participation of stakeholders (donor, recipient...); if possible: views and expertise of groups affected should form integral part of the evaluation; involving all parties capacity building

DAC 1991

25

Evaluation, Monitoring and Audit (1)

once or twice, essentially at the end or 'ex-post' drawing lessons from the past in order to orient future policies and actions but also during implementation: mid-term evaluation to (re-) orient implementation

When?

external evaluators specialised in the subjects evaluatedWho?

in-depth analysisHow?

mainly analysis of the efficiency, effectiveness, impact, relevance and sustainability of aid policies and actions

What?

Evaluation:

26

Evaluation, Monitoring and Audit (2)

regularly, several times per yearWhen?

internal and external (staff, monitors.....)Who?

rapid and continuous analysis, immediately useful to improve on-going actions; of key importance to improving performance

How?

mainly analysis of efficiency and effectiveness (i.e. measuring actual against planned deliverables); is a systematic management activity

What?

Monitoring:

27

Evaluation, Monitoring and Audit (3)

.

during or after implementationWhen?

external, professional auditorsWho?

verification of financial records (financial audit)How?

traditionally checks whether financial operations and statements are in compliance with the legal and contractual obligations. More concerned with compliance, but better financial management can also contribute to improving current and future actions. More recently: Performance audit is strongly concerned with questions of efficiency amd good management

What?

Audit:

28

Evaluation

Evaluation is the systematic assessment of the value and merit of some object (in this case, a project or a programme)

Evaluation is the systematic acquisition and assessment of information to provide useful feedback about the evaluated object.

29

Evaluation: major criteria

Relevance Efficiency Effectiveness Impact Sustainability

30

Relevance= The extent to which the aid intervention is suited to the priorities

and policies of the target group, partner country and donor

Possible questions: To what extent are the objectives of the programme valid for the

beneficiaries? Are the activities and results of the programme consistent with the

overall objectives? … …

31

Efficiency= Efficiency measures the outputs – qualitative and quantitative – in relation

to the inputs. It is a term which signifies that the aid uses the least costly resources in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted

Possible questions: Were the activities cost-efficient? Are the unit costs comparable to … … …

32

Effectiveness= A measure of the extent to which an aid intervention reaches its

objectives

Possible questions: To what extent was the project purpose achieved/is likely to be achieved? What were the major factors influencing the achievement or non-

achievement of the project purpose? … …

33

Impact= The positive and negative changes produced by an intervention,

directly or indirectly, intended or unintended.

Possible questions: What real difference has the activity made to the beneficiaries? How many people have been affected? … …

34

Sustainability= Sustainability is concerned with measuring whether the benefits of

an activity are likely to continue after donor funding has been withdrawn.

Possible questions: To what extent did the benefits of a programme or project continue after

donor funding ceased? What were the major factors which influenced the achievement or non-

achievement of sustainability of the programme or project? … …