Defining Traits Rapid Evaluations

Session 2

1

Session structure

Introduction – Rapid Evaluations…a new concept?

Objectives – Define REs – Specify particular traits of REs

Main topics – Definition – Key constitutive elements – Rationale, timing and users

Session Re cap

2

Previous session Re Cap (Evaluation, M&E and Rapid Evaluations for Government Programs)

The reasons for rapid evaluations

The concept of evaluation

Program Evaluation and Program Monitoring

M&E systems

Different Types of evaluations

Rapid Evaluations introduced

Rapid Evaluations and Information

3

Course Overview

1. Evaluation, M&E and Rapid Evaluations for Government Programs.

2. Defining Traits of Rapid Evaluations

3. Methods and techniques for Rapid Evaluation

4. Rapid Evaluations in practice I

5. Rapid Evaluations in practice II

6. Institutional context, application and use of Rapid Evaluations

4

Rapid Evaluations may be regarded as a subset of a larger family of evaluation methodologies, known in other fields as rapid appraisals, rapid assessments, desk reviews, etc.

The novelty of Rapid Evaluations is its systematic and more general use by government agencies in order to have better mechanisms for decision making.

Rapid Evaluations have benefited from the use of these methodologies in other fields or contexts.

Appraisals

The purpose is to develop a preliminary, qualitative understanding of a situation. Developed and applied mostly in anthropological studies.

Assessments

Research approaches that are intended to be efficient and cost effective; savings are found in reduced sampling and more efficient data analysis. Common in medical, environmental and biological sciences.

Desk Research (or Reviews)

Mostly used in the context of market research it refers to the use of secondary information or information readily available in order to conduct analyses of various topics.

Rapid… Rapid evaluations

5

Definining Rapid Evaluation

Starting point: working definition

We understand Rapid Evaluation (RE) as an evaluation approach that draws on different research methods and techniques and mostly on existing information to quickly, yet systematically, collect, analyze and synthesize evidence on key aspects related to public interventions, such as design, outputs, processes, outcomes and impacts.

Using RE is extremely useful in the face of important constraints of time and resources to provide valuable decision making inputs for budget, planning and to improve program performance.

6

Key constitutive elements of Rapid

Evaluations

Scope (content)

Information

Methods

Use

Budget & Time

7

Scope

A Rapid Evaluation:

Captures and assess the core traits of programs relevant for decision making.

Most commonly an RE:

– Examines the coherence of the program design

– Establishes connection between design and implementation

– Looks broadly at all the links of the results chain, from inputs to outputs to impacts.

– Reviews performance indicators

– Clarifies existing evidence on outcomes or impacts.

8

Scope

Always includes evaluation of program logic (design evaluation)

Plus some assessment of implementation (organization, process or management evaluation)

Plus a review of available performance or impact indicators (performance evaluation).

9

Information

Where does the information for RE come from?

A mix of data sources

– Documental information already available becomes a fundamental source.

• Documental information can come from a program's monitoring and information systems, previous studies, analysis or evaluations.

‒ Other types of information sources derived from:

• Observation

• Interviews

• Meetings

‒ Generally little or no additional systematic primary data collection

10

Methods

There is no unique strict method for REs

RE methods are generally adaptable to scope and context

– The scope of an RE depends on programs’ maturity, stage, or on specific information needs.

– Context refers to available information, development of M&E

systems and accountability needs.

11

Methods

But common to REs is the aim to systematically and analytically synthesize important programs´ information in order to be useful for accountability purposes and decision making.

The challenge is to know what information to consider, what aspect of a program to evaluate and how information gets analyzed and synthesis is developed.

12

Methods

REs use:

‒ Within a national, region or sector, a standardized methodology to analyze a set of programs.

‒ A homogeneous set of information to facilitate some (to the extent of possible) comparability across programs.

‒ Replicable instruments and processes.

REs aim is to

‒ Be applied to a significant number and a broad range of programs

‒ Be Implemented serially in the context of “systems of evaluation”

13

Methods

REs may apply ‒ Different types of methods, qualitative and quantitative, to analyze

a program ‒ Other methods to gather information, such as focus groups,

observation, interviews. ‒ A combination of analytic instruments, like the LogFrame matrix,

short questionnaires, etc.

REs do not

◦ Apply sophisticated quantitative methods on the basis of extensive primary data collection

◦ Draw conclusions on the impact and final outcomes on their own

14

Use

REs are intended as “decision making” tools

‒ Identify key messages and findings for decision making

‒ Contribute to recommendations formation

‒ Continuous program improvements

‒ Executive format and style of the results

15

Budget and Time

REs ‒ Are designed to be completed in a short period of time

Cost efficiency and time opportunity are a consequence of other constitutive dimensions ‒ Scope and methods

‒ Information used

‒ Coverage and scale

‒ Relevance for decision making

16

Key constitutive elements of Rapid Evaluations

• Decision makers at different levels

• Accountability

• Improvement of programs

• Budget decisions

• Mixed Data sources

• Documents/ Field

• Cost Effective

• Adaptable to context and scope

• Replicable

• Review

• Standardization

• Design/Logic framework

• Implementation (process, organization, planning)

• Results/outcomes Scope Method

Use Information

17

Rationale (for evaluation in general)

Two basic reasons for government:

“Internal”

– Need to enhance public sector performance

– Evidence based decision making

“External”

– Political accountability and transparency

18

Rapid Evaluations: Why?

Flexible and timely tool that allows to inform policy and budgeting decisions in the short run.

Generalising and transversal tool that permits to undertake a global comprehensive review of a program.

Standardized methodology that can be applied serially and allows obtain similar types or categories of performance information that facilitate the knowledge across programs

19

Rapid Evaluations: Why?

Manageable and adaptable tool that permits evaluating a number of programmes at the same time

Constructive methodology that permits to produce and consolidate information when lacking

…

20

Timing

Different aspects to consider about the timing of RE

– Country context and characteristics of M&E systems

– Policy/Program Cycle

– Program stage and maturity

21

Country context and characteristics of M&E systems

Aspects that are worth considering

– Degree of institutionalization of the government’s evaluation process

– Agencies and actors involved in the evaluation system

– M&E systems

– Evaluation capacity

22

Rapid Evaluations: When?

Projects/programs with….

– Upcoming key decision making points in the short run

– Mix of questions about design, process and performance (outputs, impacts)

– Generic lack of knowledge about performance

– Good quality secondary data

In preparation for a more focused operation or impact evaluation

….

23

Rapid Evaluations: When?

Timing of RE and the policy cycle

– When program has been

designed, or needs to be redesigned.

– When program is implemented

– After program implementation to assess results

– Periodical RE for continuous program improvement

• Rapid Evaluation

• Rapid Evaluation

• Rapid Evaluation

• Rapid Evaluation

(Re) Design Implement

ation

Results

Evaluation and

Budget Decisions

24

For whom?

Central Agencies

(Planners, budget

holders)

Legislators

Program Managers

(operators, implementators)

Civil Society Organizations

Citizens

25

For budget holders, central planning agencies

Pressure on agencies to perform (continuous improvement) – By reviewing their budget requests in the light of their past

performance (effectiveness and efficiency)

– By monitoring the implementation of recommendations and adjustment plans that have emerged from evaluations

Expenditure prioritization – Which programs which should be cut back, eliminated or expanded?

– Budget-linked performance targets

– Assess how far the agency is responsible for any failure to meet target

Accountability towards the general public

26

For program managers /implementers

Strategic directioning

– Improve program design and achieve higher outcome effectiveness

Internal management

– Improve programme implementation and achieve higher efficiency and cost effectiveness

Internal (within government) and external (towards citizens) accountability

27

REs for budget holders, central planning agencies

Evaluations timing must fit in with budget preparation

– Means evaluations usually need to be quick

– Results before final budget decisions taken

Similar types or categories of performance information allow a more comprehensible view across programs and facilitate some decision making

Uniform evaluation approach (metric)

28

RE for budget holders, central planning agencies

Evaluation reports designed to tell budget decision-makers what they need to know

– How effective the program is

– Whether it is inherently ineffective

– Or could be effective with better implementation

Focus on design evaluation permits questioning the reason of existence of programs

– Inform if the design is adecuate to resolve the problem

– Highlight issues of inter-institutional coordination

29

REs for program managers/ implementers

Integral assessment – Feedback that links design, implementation, outputs and

results

– Identify bottlenecks by looking at the “big picture” and highlighting “core” issues

Formative approach – Consolidates (non systematized) information in a unique

place

– Constructive approach in opposition to external “assessment” approach

30

Civil Society and Citizens

Increasing pressures for transparency and accountability

– There are increasing initiatives from Civil Society Organizations to analyze performance of governments

– Demands from citizens to know uses of public budget.

Importance of generating and facilitating access to quality information of government interventions

Transparency of evaluation process and evaluation by external agents give greater legitimacy to governments

31

Session ReCap

Definition of Rapid Evaluation

Key constitutive elements of Rapid Evaluations

– Scope

– Methods

– Information

– Use

– Timing

Rationale for RE

Users

32