Methodological Proposal for the Evaluation of the Relevance and Scope of Indicators of Social Programs

Gilberto MoncadaConsultant

Banco Mundial Noviembre 2008

1. Background

Methodological proposal developed in the framework of Fees For Services with the National Evaluation Council of Mexico.

The proposal has to be piloted with the indicators of different programs in order to contrast its utility in different situations and make refinements.

The proposal can serve as a base for the development and application of a methodology for the evaluation of indicators in different contexts.

2. Objetive of the methodology

To facilitate the measurement and evaluation of the relevance and scope of inidcators used by social programs, with the objective of refining/improving them.

Are the selected indicatoirs the most adequate to measure the advances of social programs? How can we examine if these indicators are the best?

3. Focus of evaluation

Indicators of social programs created through a logic framework process.

Indicators at the level of: outcome, objective and component.

4. Area of evaluation

Dimensions of evaluation Categories to be evaluated Variables

General Variables related to basic capacities of indicators

Consensus in the formualtion: information capacity, dissimination and access.

Specific aspects of basic capacities of indicators as defined by a guide of questions

Quality of the indicator

CREAM criteria: clear, relevant, economic, adequate, monitorable.

Specific aspects of the quality of an indicator as defined by a guide of questions.

Use of the indicator

Use of the indicator Specific aspects of the use of an indicator as defined by a guide of questions.

5. Method

5.1 Guide of Questions

5.2 Information Collection

5.3 Question Scoring

5.4 Weights and Points

5.5 Metrics of evaluation

5.1 Method: Guide of Questions

The evaluation should follow a guide of questions related to the variables.

It is proposed that the variables can relate to a group of indicators and/or single indicators.

  Each dimension of evaluation will be evaluated on the basis of the

results of the questions.

5.2 Method: Collection of Information

Desk review of key documents( matrix of indicators, evaluation reports, registers, normative documents, benficiary information, databases etc.)

In depth interviews with operators of the program, superior authorities, external and internal users of the indicators of a program.

The guide of questions should be answered by the evaluation team based on the information obtained.

5.3 Method: Question Scores

Every question has response alternatives which represent a scale: Maximum score is 1; Second is 0.5; Third is 0.25; Lowest score is 0.

Some question are binomial: yes/no (value of 1 or 0). Each response must be justified clearly and concisely, with evidence. If evaluating a group of indicators rather than an individual indicator (in for

example, basic capacities) is indicator is given the corresponding score.

5.4 Method: Weights and Points

1123112345621

Dimensión de

evaluación

Ponderación de la

dimensión Criterios Ponderación de

cada criterio

Puntaje máximo

por pregunta

Puntaje total

máximo Variables generales de calificación Variables generales

PG 1 Proceso de consenso para formulación del indicador

Pg 1 1 (1Pgt.) 1

Capacidad informática Pg 2 1 (1Pgt.) 1

Difusión y acceso a información de indicadores

Pg 3 1 (5Pgt.) 5

100% 7 Variables de calificación a nivel de cada indicador Calidad P 1 Claridad P 1 1 (3Pgt.) 3

Relevancia P 2 1 (2Pgt.) 2

Economía P 3 1 (1Pgt.) 1

Monitoreable P 4 1 (8Pgt.) 8

Adecuado P 5 1 (3Pgt.) 3

Unicidad P 6 1 (1Pgt.) 1

100% 18 Utilidad P 2 Uso de la información del

indicador P 1 1 (5Pgt.) 5

100% 5 Total 100% (30 Pgt.) 30

Calculation of weights

It is proposed to determine the relative weights of the different dimensions of evaluation (general variables, quality and use), as will as the crtieria in each dimension (CREAM etc), through a process of consulting national experts.

The experts should use a scoring table, using a scale of 1 to 10 to indicate the value they would attach to each dimension and criteria given their experience.

The results obtained through this consultation can serve as the basis of determining the average weight of each dimension and criteria.

5.5 Method: Metric to Evaluate IndicatorExample of the calculation for the criteria: quality

1111111122222222333333334444444455555555666666661

Criterios CREMA

Ponderación p(i)

Puntaje total alcanzado

Puntaje total máximo

Valor relativo

Normalizado Promedio ponderado

de calificación Claridad p 1 x 1 X 1 (x 1 /X 1 ) P 1 (x 1 /X 1 )

Relevancia p 2 x 2 X 2 (x 2 /X 2 ) P 2 (x 2 /X 2 )

Economía p 3 x 3 X 3 (x 3 /X 3 ) P 3 (x 3 /X 3 )

Monitoreable p 4 x 4 X 4 (x 4 /X 4 ) P 4 (x 4 /X 4 )

Adecuado p 5 x 5 X 5 (x 5 /X 5 ) P 5 (x 5 /X 5 )

Unicidad p 6 x 6 X 6 (x 6 / X 6 ) P 6 (x 6 / X 6 )

Total 100% V 1 = Σ valor de cada

criterio

Score of indicators

It is proposed to consider three rankings for each dimension using three equal ranges. Low quality/use/general capacity 0 – 33.3 Medium quality/use/general capacity 33.4 – 66.7 High quality/use/general capacity 66.8 – 100

The total score of the indicators in terms of its relevance and scope is found estimating the average weight of each dimension. Low relevance and scope 0 – 33.3 Medium relevance and scope 33.4 – 66.7 High relevance and scope 66.8 – 100

6. Methodology to evaluate the quality of indicators at the level of activitiy

Proposal is to use a guide of 10 questions that explore the basic characteristics of an indicator.

Binomial reponses (Yes/No) which represent a values of 0 and 1. The points recievd by each indicator are determined by the following

ranking:  Low Quality 0 – 3 Medium Quality 4 – 6 High Quality 7 – 10

Muchas gracias