it’s the learning, not the result which counts most in evaluation

It’s the learning, not the result which counts most in evaluation

Randall PearceTHINK: Insight & Advice

The 7th Australasian Better Boards Conference5 July 2013

Evaluation – What is it?

‘A systematic way of answering questions about projects, policies and programs’

1. Is it needed or worthwhile?2. Is it having an effect?3. At what cost?4. How could it be improved?5. Are there better alternatives?6. Are there any unintended consequences?

NFP Evaluation – What it isn’t

Who evaluates?

75%

25%

Do you measure the impact of your work?

YesNo

Source: New Philanthropy Capital 2012

Why do (or don’t) they evaluate?

Funders tend to ignore impact measurement

Data collection interferes with client relationships

Impact measurement detracts from priorities

You don't need evaluation to tell you what works

Funding is main reason for conducting evaluations

There is too much pressure to measure results

There is not enough funding for impact evaluation

Charities should report negative results too

Measuring impact makes NFPs more effective

0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00%

17.50%

21.20%

26.20%

29.50%

34.10%

37.50%

60.20%

72.30%

77.50%

Attitudes toward impact evaluation among charitiesTop two box scores (n=1,000)

Source: New Philanthropy Capital

What do they gain?

Other

Increased publicity

Don't know

Improved strategy

Increased funding

Able to show how you make a difference

0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00%

0.90%0.90%

2.10%3.70%

4.40%6.50%

8.70%9.10%

9.80%10.70%

18.40%24.60%

Benefits gained through evaluation(n=755)

Source: New Philanthropy Capital

Who should evaluate?Advantages Disadvantages

Internal • Better overall & informal knowledge

• Less threatening/known to staff• Less costly

• May be less objective• Evaluation could be a part-time

responsibility• May not be trained in evaluation

External • More objective• Able to dedicate time and

attention• Greater evaluation expertise

• Needs to learn about organisation and culture

• Unfamiliar to staff• More costly

When to conduct evaluation?Form: Stage of Program Purpose:

Proactive Evaluation

Program Start

Program Completion

Before a program starts

To synthesis information to inform program design

Clarificative Evaluation

During program development

To clarify the program design and how it operates

Interactive Evaluation

During program delivery

To improve program delivery. Involves stakeholders in the evaluation.

Monitoring Evaluation

Once the program is in a settled stage

To monitor program progress for accountability and improvement

Impact Evaluation

During or after program implementation

To assess what has been achieved, to learn and be accountable

Source: K Roberts (adapted from Owen and Rogers, 2006)

Dispelling mythsTheory of change?

• Not needed because the evaluator will reconstruct the logic of the actual program, not the theoretical model:• Foundational activities• Activities• Outputs• Immediate outcomes• Intermediate outcomes• Long-term outcomes• Organisational goals

Dispelling mythsMountain of data?

• Most data is just information…we are looking for insight into what it means

• Historical data is more valuable than a mountain of current data

• Your evaluator should identify the few ‘dashboard’ measures that you will need to evaluate

• Once an evaluation has been conducted you can use the dashboard forever

Dispelling mythsA wad of cash?

• Think of what is at stake versus the internal budget allocation – any activity with a value in excess of $200K should be evaluated

• Governments and foundations often allow for 10% to be spent on evaluation

• There are many ways that NFPs can reduce the cost of evaluations

Using the results of evaluation• Share them…as widely as you can• Some evaluators will agree to write a

summary which protects the egos of those involved

• Action learning/research is a participative approach based on a four part cycle of: taking action, measuring the results of the action, interpreting the results and planning what change needs to take place for the next cycle of action

• The best projects conclude with a Summit workshop

Beyond program impact evaluation

Learning along the way

Documentation

• Documents success and failures• Summary of key documents in one place• Timeline/sequence of events• Isolates key measures for the future• Supports performance appraisal for staff and

board• Helps orient staff, volunteers and contractors

Learning along the wayFull cost accounting

• Full costs and expenses need to be calculated to arrive at the true financial picture

• Need to include:• Budget allocation• Cash donations• In-kind services• Pro-bono services

INCOMECash Contributions

Member contributions 348,000.00$ Donors 1,168,000.00$

Sub-total cash contributions 1,516,000.00$ Subtotal in-kind contributions 926,346.00$ TOTAL INCOME 2,442,346.00$

EXPENSEOperationsSub-total operations expense 881,346.00$

CommunicationsSub-total communications 803,500.00$

Field and coordinationSub-total field and coordination 470,970.00$

TOTAL EXPENSE 2,145,816.00$ GST adjustment (97,600.00)$

NET EXPENSE 2,048,216.00$

Learning along the wayFull value assessment

• Captures all non-financial outputs in addition to financial information

• For example, while social media produces a host of measures, there are no financial equivalents as there are in traditional media (i.e. TARPs)

• Need to identify data sources for year-on-year comparison in future

Learning along the wayOrganisational behaviour and Governance

• Qualitative research reveals issues around organisational behaviour and governance which can impact outcomes

• Project governance can be examined independent of personalities to pinpoint areas for change/improvement

Learning along the wayRelationship building

• The evaluation process has been described as ‘cathartic’ by key players

• Helps diffuse tensions that build up during a campaign

• Provides stakeholders a voice/builds goodwill for the future

• Aids communication ‘across the political/media divide’

Over to you…

Questions

For more information, contact:Randall Pearce

+61 2 9358 6664randall.pearce@thinkinsightadvice.com.au

NOTE: For a copy of this presentation, please provide your business card at the end of the session or email