Unpacking Impact and Influence:Approaches to Monitoring & Evaluating

April 28, 2014

TTI PEC Africa workshop - Nairobi

Courtney Tolmie (Results for Development Institute)

A few notes to start

This session focuses on Organization M&E. That said, most of the lessons can be scaled down to the project level.

You will guide the session – and giving real time feedback:

2 | R4D.org

Way too fast. Please slow down.

Just right.

You can speed up.

How do we think about Impact and Influence?

1. What IMPACT does your organization seek to have?

2. What and who do you work to INFLUENCE as an organization?

3. What is the link between IMPACT and INFLUENCE in the work that you do?

3 | R4D.org

Monitoring and Evaluation – the link to Impact and Influence

In other words, a way of following whether you are doing what you planned to do and whether you are achieving the IMPACT you wanted to, including Policy INFLUENCE.

4 | R4D.org

Monitoring: a continuing function that aims primarily to provide the management and main stakeholders of an ongoing intervention with early indications of progress, or lack thereof, in the achievement of results.

Evaluation: systematic and objective assessment of an on-going or completed project, program, or policy, and its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact, and sustainability.

Definitions from the World Bank - http://siteresources.worldbank.org/INTBELARUS/Resources/M&E.pdf

What Makes an organization-level M&E System Effective?

Defining Impact and how we get there (TOC)

Defining how to measure impact & outputs

(indicators)

Carrying out M&E (Methods)

Using results to improve

work (Feedback

Loop)

5 | R4D.org

M&E Systems – Defining a Theory of Change

Defining Impact and how we get there (TOC)

Defining how to measure impact & outputs

(indicators)

Carrying out M&E (Methods)

Using results to improve

work (Feedback

Loop)

6 | R4D.org

Defining a Theory of Change Clarifying your

organization objectives and desired impacts

Mapping out what is required to achieve these impacts

Linking to the activities and actions your organization will take to achieve the impact

Also highlighting assumptions (including context issues)

Similar structures - logframes

M&E Systems – Defining Indicators

Defining Impact and how we get there (TOC)

Defining how to measure impact & outputs

(indicators)

Carrying out M&E (Methods)

Using results to improve

work (Feedback

Loop)

7 | R4D.org

Defining Indicators

Turning what you hope to achieve into what you can actually measure

Making indicators SMART

Value (and tradeoff) in collecting indicators on impact and outputs

M&E Systems – Designing Methods

Defining Impact and how we get there (TOC)

Defining how to measure impact & outputs

(indicators)

Carrying out M&E (Methods)

Using results to improve

work (Feedback

Loop)

8 | R4D.org

Defining and designing Methods

Identifying how to actually collect data on the indicators that you have defined

Timing – monitoring, evaluating or both

Things to consider – capacity, resources, what you are trying to achieve

M&E Systems – Ensuring Feedback Loops

Defining Impact and how we get there (TOC)

Defining how to measure impact & outputs

(indicators)

Carrying out M&E (Methods)

Using results to improve

work (Feedback

Loop)

9 | R4D.org

Ensuring Feedback Loops

The most forgotten step of M&E systems

Results for M&E can help improve programs and ultimately impact

Best to have formal structure to build results back into future work

Moving from culture of M&E to MEL

Mapping areas for development

Based on your existing M&E system, assess how you are doing in each element of effective M&E

Scale from “we don’t do this at all” to “we do this very well on all areas of work”

10 minutes to map your organization – then we will come back to review

10 | R4D.org

Defining Theories of Change

11 | R4D.org

Defining Theories of Change

Examples from other organizations Tips from the group about how to do it well Ideas for moving forward

12 | R4D.org

13 | R4D.org

Build evidence on better ways to

budget for higher education

Share recommendations with Minster of Education and

Finance

New budget passed with

recommendation

It would be easier if organization TOCs looked like this:

But this is not realistic

WHY NOT?

• TTs are trying to achieve bigger changes than just this• A single TT is going to likely need to take a lot of different

actions to get to that end impact• A single TT is not going to individually achieve this impact

14 | R4D.org

15

16 | R4D.org

Note about Theories of Change

A good TOC for influencing policy is incredibly complex.

Your organization may not try to affect all parts of the TOC – and thus may not choose to do M&E for all factors that go into policy influence.

But good to be aware.

Defining Theories of Change

Examples from other organizations Tips from the group about how to do it well Ideas for moving forward

17 | R4D.org

Defining Theories of Change – TOC Pitfalls to avoid from Stanford Social Science Review

1. Confusing accountability with hope2. Creating a mirror instead of a target3. Failing to take external context into account4. Not confirming the plausibility of your theory5. Creating a theory that is not measurable6. Assuming you have figured it all out

18 | R4D.orghttp://www.ssireview.org/blog/entry/six_theory_of_change_pitfalls_to_avoid

What lessons can you share from your organizations –

things that have worked and what has not worked?

Defining Indicators

19 | R4D.org

Defining Indicators

What types of indicators work well, especially for measuring policy influence and impact?

Tips from the group about how to do it well Ideas for moving forward

20 | R4D.org

Examples of good indicators – for the full organization While the focus of this program is on PEC, it is worth noting

that there are multiple components of a think tank’s performance worth monitoring:

21 | R4D.orgFrom Raymond Struyk’s forthcoming book – Improving Think Tank Management

Public Policy

Perspective

Funder Perspectiv

e

Internal Business

Perspective

Innovation & Learning Perspectiv

e

Financial Perspectiv

e

Examples of good indicators – for the full organization While the focus of this program is on PEC, it is worth noting

that there are multiple components of a think tank’s performance worth monitoring:

22 | R4D.orgFrom Raymond Struyk’s forthcoming book – Improving Think Tank Management

Public Policy

Perspective

Funder Perspectiv

e

Internal Business

Perspective

Innovation & Learning Perspectiv

e

Financial Perspectiv

e

Examples of good indicators for PEC impact and influence Some helpful indicators from the center for International Governance Innovations:

23 | R4D.orgFrom www.cigionline.org/blogs/tank-treads/communications-and-impact-metrics-think-tanks

Indicators of Exposure:• Media mentions• # and type of publications• Scholarly citations• Government citations• Think tank ratings

Indicators of Resources:• Quality, diversity, stability of

funding• Characteristics of researchers• Quality, # of networks,

partnershipsIndicators of Demand:• Events• Digital traffic and engagement• Access to officials• Publications sold, downloaded

Indicators of Policy Impact & Quality of Work:

• Policy recommendations considered or adopted

• Testimonials

• Quality of think tank work

Other indicators we have seen used

24 | R4D.org

If the objective is …Specific policy recommendation implemented

Change in the debate

… Possible indicators are:

Invitation to share findings with specific audiences (government, others)

Attendance of decision makers at dissemination events

(if available) Information on issue being debated in policy discussions

Request for research by policymakers on the new topics

Greater debate on topic in media (if available) Information on issue being

debated in policy discussions

Longer term – access to policymakers

Individual meetings with policymakers

A note about impact indicators …

In general, these are not going to be attributable.

Attribution is difficult Almost without exception, it takes a village.

This does not mean impact indicators are not helpful – especially if seeking to improve results.

25 | R4D.org

Defining Indicators

What types of indicators work well, especially for measuring policy influence and impact?

Tips from the group about how to do it well Ideas for moving forward

26 | R4D.org

Defining Indicators

What types of indicators work well, especially for measuring policy influence and impact?

Tips from the group about how to do it well Ideas for moving forward

27 | R4D.org

Designing methods for M&E

28 | R4D.org

Designing methods for M&E

What common options are others using? Tips from the group about how to do it well Ideas for moving forward

29 | R4D.org

Options for M&E approaches

30 | R4D.orgTable from ODI Background Note – A guide to monitoring and evaluating policy influence (Harry Jones)

Does not include common impact evaluation methods – RCTs, process tracing, etc.

Designing methods for M&E

What common options are others using? Tips from the group about how to do it well Ideas for moving forward

31 | R4D.org

Designing methods for M&E

What common options are others using? Tips from the group about how to do it well Ideas for moving forward

32 | R4D.org

Ensuring Feedback Loops

33 | R4D.org

Ensuring feedback loops

Do results feed back into the think tank automatically?

Tips from the group about how to do it well Ideas for moving forward

34 | R4D.org

Connecting M&E Results to Think Tank Changes

Two potential reasons to undertake M&E:

To achieve the second, M&E can’t stop at data analysis – it needs to feed back into the think tank.

35 | R4D.org

ACCOUNTABILITY to funders and others

BETTER RESULTS for think tank impact and influence

Questions to ask to design a strong Feedback Loop (Knowledge Management)

36 | R4D.org

The Question

WHO should hear the M&E results?

WHEN should they get the information?

The Answer

(1) Who is the best position to exploit the information – capacity and interest

(2) More generally, anyone who SHOULD know

(1) At a time when they can act – capacity and opportunity

(2) Generally more than once

WHAT should they receive?

(1) Format – useful but not necessarily fancy (2) Ideally comparisons to baseline data

In general, the Feedback Loop is more likely to be effective if designed early – proactive rather than reactive.

Built from Raymond Struyk’s forthcoming book – Improving Think Tank Management

Ensuring feedback loops

Do results feed back into the think tank automatically? Tips from the group about how to do it well Ideas for moving forward

37 | R4D.org

Ensuring feedback loops

Do results feed back into the think tank automatically? Tips from the group about how to do it well Ideas for moving forward

38 | R4D.org

A final word

39 | R4D.org

Conclusion

M&E is not easy – and requires capacity. This is especially true for organization-level M&E and for policy influence M&E. Match M&E plan to capacity

Moving from M&E to MEL (L = “learning”)

Linking the two objectives of M&E – accountability and better results Some methods and indicators can address both – where

possible, don’t double your work Example – how can you use the IDRC monitoring indicators for

PEC to help you achieve better results?

40 | R4D.org

Other Great Resources (in addition to those referenced on this powerpoint)

ODI work on RAPID

On Think Tanks page on Monitoring and Evaluating Influence

(http://onthinktanks.org/topic-pages/topic-page-monitoring-and- evaluating-influence/)

Research 2 Action PEC page

41 | R4D.org