LIBERIA

INDICATOR CHARACTERISTICS

How closely do the indicators used measure the results?

LIBERIA

2 2

INDICATOR CHARACTERISTICS: OBJECTIVES

Use 2 key questions to create indicators Use 7 indicator criteria to select/improve

indicators

LIBERIA

3 3

What is a Performance Indicator ?(ADS Def. p 67)

3

A particular characteristic or dimension used to measure intended changes defined by a Result Framework.…

A phenomenon we observe that tells us there is a problem and whether it is being solved.

The measurement that shows how and by how much a project is achieving its intended results.

LIBERIA

4 4

An example of an Indicator

4

Body temperature : an indicator that may show that illness is present.

In order to use this indicator to diagnose illness, it is necessary to know, at a minimum, how far above the normal level the temperature is.

LIBERIA

5 5

Role of Indicators in a Results Framework

5

Indicators apply at all levels of the project hierarchy of results.

Indicators are used to measure change over time. The result statement will show the direction of change desired (increase or

decrease) NOT the indicator.

Example: Percentage of children ages 12 to 24 months in the target area who are fully vaccinated. (NOT Increased

percentage)This indicator tells us the phenomenon to observe, not which

way the change might move.

LIBERIA

66

Context IndicatorsReality surrounding a development problem is usually greater than USAID’s manageable interest.

Example: Damages caused by forest fire reduced.

Tracking rainfall as a context indicator informs the analysis of USAID efforts to control forest fires.

6

LIBERIA

7 7

If a Result is the amount of change achieved or a problem solved: What is a

Problem?A discrepancy in somebody’s head between a Perception (based on current data) and an Ideal (based on ideal data) floating on an Emotion supported by Values

7

P(cd) / I(id) E

V

Reality

Data

LIBERIA

8 8

To formulate Indicators for a Results Framework, ask these key questions

1. How do we know it is a problem? = Data· What data generated the problem through which of

the 5 senses?

2. What phenomenon are we observing? = Indicator· What will we observe to see whether the problem is

solved?8

LIBERIA

99

Table Exercise V

9

Each table receives an envelope with cards. On each card is a problem statement or an indicator.

Instructions:Match the indicators to the appropriate problem

statement and post the matches on a flipchart page or on a table.

10 min.

LIBERIA

10 1010

The loss of tropical glaciers in the Andes impacts the hydrologic cycle and raises grave concerns over future

water supplies throughout the region (Barnett et al., 2005).

The chronic lack of planning capacity, linkages, and communication among and between stakeholder groups is resulting in an increase in conflict between lowland and highland populations

in Peru and other Andean countries (Oré 2005).

Number of years of glacial water supply remaining

Number of water resources management-related conflicts

Problem Statements…Matched with Indicators

Tropical glaciers loss rate

LIBERIA

1111

Coastal regions are home to nearly 70 per cent of the country’s population and that produce 60 per cent of its agricultural GDP

Water from the Santa watershed also powers three hydroelectric stations with a total capacity of 271 MW, or 5.5 per cent of the total national hydroelectric output for Peru, that directly and indirectly serves 1,697,221 people throughout the region

• percentage of agricultural GDP of coastal regions

• Ratio of current capacity to original capacity of three hydroelectric stations

• percentage of country population in coastal regions

LIBERIA

1212

Water resources management in the watershed basins of Santa/Piura Chira improved

Indicators:1.Tropical glaciers loss rate2.Number of years of glacial water supply remaining 3.Number of water resources management-related conflicts4.Percentage of country population in coastal regions5.Percentage of agricultural GDP of coastal regions6.Ratio of current capacity to original capacity of three

hydroelectric stations

LIBERIA

13

1313

Decision-making processes function

effectively# of active

highland/lowland decision making processes

Politically balanced decisions made# of consensus

decisions reached

Conflicts over water avoided

# of water related conflicts

Investment in improved water resources

management increased$ value of qualified project

investmentsPolitical will to change

water management decision making generated# of working groups formed

Good representation at the conference secured

% of targeted stakeholders who attend

MCS Case: Indicators with Associated IRs

LIBERIA

15 15

Task VI b: Create Indicators for the Results Framework

1. Using any Problem Description/Analysis, create one or more indicators for each Result. Write each indicator on an A4 paper and place it under the corresponding result in your Results Framework. Ask these questions:

How do we know it is a problem?Ask this question until you get to the data that

generated the problem in your heads.

What phenomenon are we observing that generated the data?

Ask this question to define the indicator(s)2. Add any informative Context Indicators.45 min

15

LIBERIA

16 16

Indicator Job Description and Criteria

ObjectivePractical Useful for Management Direct Attributable to USAID efforts TimelyAdequate (ADS 203.3.4.2– p 16-17)

16

Job Description: To tell the most truth about whether a result is being achieved at the least cost when you need to hear it.

USAID’s characteristics of good performance indicators:

LIBERIA

1717

Objective?• Unambiguous and

operationally precise about–What is being measured–What data are being

collected • Uni-dimensional --

measures only one thing• Consistent over time

Result: - Performance of export firms improved

Imprecise Indicator: - Number of successful export firms

Precise Indicator: - % of export firms experiencing annual increase in revenues of at least 5%

LIBERIA

18

18

PRACTICAL?• Data are available when required for

decision making

• Data can be obtained at reasonable cost

LIBERIA

19 19

USEFUL?• Is the indicator useful for management?• Which indicators are most meaningful at a

given point in time?

LIBERIA

2020

DIRECT?• The indicator closely

tracks the result it is intended to measure.

Result: • Citizens’ knowledge of their

rights expanded

Direct Indicator: • % survey respondents able to

identify 3 or more key civil rights

LIBERIA

PROXY INDICATORS• Indirect measures of the intended result. • Use if data for direct indicator is unavailable or not

feasible to collect.

Result: - Citizens’ knowledge of their rights expanded

Direct Indicator: - % survey respondents able to identify 3 or more key civil rights

Proxy Indicator: - Number of civil rights cases brought to court by targeted community

organizations

LIBERIA

22

22

If there had been no USAID activity, would the measured change have been different?

ATTRIBUTABLE?• Indicators selected for the PMP should measure

changes that are clearly and reasonably attributable, at least in part, to USAID effort.

• Attribution exists when the links between USAID outputs and the results being measured are clear and significant.

• A simple way to assess attribution:

LIBERIA

23

23

TIMELY?• Indicators should be available when they are

needed for decision making.

– Are the data available frequently enough?

– Are the data current?

LIBERIA

24

24

ADEQUATE?• Taken as a group, the indicator and its companion indicators

should be the minimum necessary to capture progress towards the desired result.

• How many indicators? – As many as are necessary and cost effective for management

purposes• Too many indicators results in information overload• Too few indicators could be misleading• Use the “necessary and sufficient” rule to select the

minimum number per result

RULE OF THUMB2-3 PER RESULT

LIBERIA

25

2525

Table quiz: Selecting indicators (“x” rejection criteria) (20 min.)

Result: Citizens’ awareness increased

Direct Objective Useful Practical Attrib. Timely Adequate

1. No. of people trained

2. Ratio of people trained/target population

3. % of people trained to target population

4. Ratio of cumulative No. of people trained to target population

5. No. of courses delivered

6. Average pre- and post-test scores of citizen awareness

7. Random focus group pre- and post-test scores of citizen awareness

8. Key informant interviews

9. No. 4 and No. 7

LIBERIA

26

26

Indicators

26

Quantitative: Number, amount, ratio, percentage, proportion, average

scores, rating, weighted or non-weighted index, etc.

Qualitative: Description of the status of an intended achievement,

documented observations, representative case descriptions, perceptions, opinions of value, photos, etc.

LIBERIA

2727

INDICATORS: QUANTITATIVE, QUALITATIVEOR BOTH?

QUALITATIVE• Expert opinion on

comprehensiveness of a law

QUANTITATIVE• Dollar value of plantains exported

BOTH Country score on “Corruption

Perceptions Index”

LIBERIA

282828

Example of Indicators at Different Levels

Incidence of diphtheria, pertussis and tetanus in young children

No. of children vaccinated against diphtheria (D) No. of children vaccinated against pertussis (P) No. of children vaccinated against tetanus (T) No. of vaccines provided to clinics

Under-five mortality rate

Input/Activity

Output

Outcome

Impact

LIBERIA

29

2929

Disaggregation and Gender ADS 203.3.4.3 (p 17-18): Performance management

systems and evaluations at the AO and project or activity levels must include gender-sensitive indicators and sex-disaggregated data when the technical analysis supporting the AO, project or activity to be undertaken demonstrate that: The activities or their anticipated results involve

or affect women and men differently; and If so, this difference would be an important factor

in managing for sustainable program impact.

LIBERIA

30 3030

Water resources management in the watershed basins of Santa/Piura Chira improved1. Tropical glaciers loss rate2. Number of years of glacial water supply remaining 3. Number of water resources management-related conflicts4. % of country population in coastal regions5. % of agricultural GDP of coastal regions6. Ratio of current capacity to original capacity of three hydroelectric stations

Water resources management improved# of WRM investments validated both scientifically and politically

Politically balanced decisions made# of consensus decisions reached

Investment in improved water resources management increased$ value of qualified projects

Conflicts over water avoided# of water related conflicts

Political will to change water management decision making generated# of working groups formed

Effective decision making processes function# of active highland/lowland decision-making processes

Stakeholder capacity increased# of shared planning tools used in decision making

Good representation at the conference secured% of targeted stakeholders who attend

?

LIBERIA

31 31

Indic/Criteria Objective Pract. Useful Direct Attrib Timely Adeq.

Assessing Performance Indicators Against Criteria Result: Water resources management in the watershed basins of Santa/Piura Chira improved

31

Tropical glaciers loss rate

Number of years of glacial water supply remaining

Number of water resources management-related conflicts

% of country population in coastal regions

% of agricultural GDP of coastal regions

Ratio of current capacity to original capacity of three hydroelectric stations

YES ? YES NO NO NO ?

YES NO NO NO NO NO ?

YES YES YES YES NO YES ?

YES NO NO NO NO NO ?

YES NO NO NO NO NO ?

YES YES NO NO NO YES ?

LIBERIA

32 3232

Indicator/Criteria Objective Pract. Useful Direct Attrib. Timely Adeq.

Assessing Performance Indicators Against Criteria Result: Water resources management in the watershed basins of Santa/Piura Chira improved

Number of water resource management investments validated both scientifically and politically

Number of water resources management-related conflicts (context indicator)

Number of Hectares (HA) classified as “conserved”

Number of Hectares (HA) classified as “restored” YES YES YES YES YES YES

YES YES YES YES YES YES

YES YES YES YES NO YES

YES YES YES YES YES YES

LIBERIA

33 3333

Context Indicators 1. # of water resources management-related conflicts2. Rate of glacial melt

Result: Water resources management in the watershed basins of Santa/Piura Chira improved1 . Number of Hectares (HA) classified as “restored” 2 . Number of HA classified as “conserved”3. Number of water resource management investments validated both scientifically and politically

Critical Assumptions1. National policy supports watershed basin regional planning

IR1: Models for restoration/conservation validated1. Number of models replicated 2. Number of replications

IR2: Water management decision making restructured1. Cases of highland/lowland decision making processes functioning

IR3: Investment in improved water resources managem. increased1. $ value of qualified projects

MCS Case Indicators

LIBERIA

34 34

TASK VII: Select Good Performance Indicators

1. Select one Result with multiple Indicators2. Use the Indicator Criteria table to put your indicators

on the left with the 7 standards across the top. Do your indicators meet all the criteria?

3. Fill in the boxes with “Yes” or “No”.4. Anything need to be changed?

30 min.

34

LIBERIA

35

35

Session 3. Summary

• Use 2 key questions to create indicators• Use 7 indicator criteria to select/improve indicators• Reality doesn’t have problems, we do.