moving from evaluation to learning peter york

Program Evaluation:

Moving from Evaluation to Learning

2

Session Agenda

• Introductions

• The Need for Learning

• Why Nonprofits Aren’t Learning

• Evaluation vs. Evaluative Learning

• Nonprofit Evaluative Learning – Steps, Processes & Tools

The Need for Learning

4

Leadership = Success

Leadership is key to:

Growth

Sustainability

Lifecycle Advancement

Effectiveness

5

There’s More to Leadership…

• Advancing effective communication of mission and vision to

internal and external stakeholders

• Engaging internal and external stakeholders in planning

• Taking decisive action when faced with challenges

• Making decisions anchored in cost-effectiveness

• Demanding accountability that includes demonstrated

success with those being served/targeted

Only one in four nonprofit organizations is well led…

6

The Key to Leading Is Learning

Only one in four nonprofit organizations is an effective learner…

The following organizational learning behaviors are significantly and

singly the biggest predictor of organizational leadership:

• Creating sophisticated financial, operational, programmatic and

environmental data-gathering and learning processes

• Spending time leveraging program data for making meaning,

decisions, designs and plans, not accountability or validation

• Infusing learning into ALL planning, anchored in program success

• Taking immediate and decisive action, particularly at a human

resource level, as indicated by strategy research findings

Why Aren’t Nonprofits Learning?

8

The Investment Model Is All Wrong

• There’s no “outcome synchronicity” between the

investor and investee

• Effectiveness and accountability must be viewed

through the measurement of proximate effect

• Proximate cause-and-effect is the only way!

• Learning requires understanding the cause, not the

effect

• There’s no Research & Development for

programs/initiatives

9

What Results Do Investors, Business Leaders, & Consumers Want?

Example: In-Home Support Services to New Parents, For-Profits vs. Nonprofits

Investors

(Funders)

Business

LeadersConsumers

For-Profit Results:Just Give Me (or Show Me) the Direct

Results, Please…

Nonprofit Accountability:Just Give Me the Direct Results, But Somehow Prove to

“Them” That We Can Do Much More…

Recuperation, parent-child bonding, healthy

adjustment to family change, tools for care and

feeding, stress-reducing routines and habits, better

communication skills with providers

• Same Direct Results as the For-Profit Business

• Child Development, School Readiness, Crime Reduction

• Parental Reduction in Child Abuse & Neglect, Maternal Health

Nonprofit

Business

Leaders &

Clients

Funders

(Investors)

Why Evaluation Doesn’t Always

Lead to Learning

11

Why Evaluation Isn’t Achieving Learning

1. Makes uncontrollable community impact indicators the

metric of accountability

2. Evaluates the “whole” program/strategic effort to determine

if it worked, rather than the combinations of program

components to determine how it works

3. Aspires to a scientific research design ideal that seeks

widespread generalizability, rather than contextual

applications

4. Gathers data from the wrong source

Evaluation vs. Evaluative Learning

13

The Difference Between Evaluation and Evaluative

Learning

Evaluation: the owner determines success by overall

wins, losses and trophies.

Evaluative Learning: the coach/manager figures out how

to improve the odds of success for the next game by

analyzing the play of the previous ones.

14

ResourcesWhole

ProgramShort-Term Outcomes

Long-Term Outcomes

Community / Social Impact

EvaluationDetermining a “Whole” Program’s Long-Term Impact to Judge Social Value

15

Program Resources

Program Elements

Achievable Outcomes/

Next Actions

Studies say…

Community / Social Impact

Evaluative LearningMeasuring Program Success for “Best Practice” Improvement

16

Six Learning Practices that Facilitate Sustainable

Growth

Organizations are significantly more likely to sustain and grow if they

engage in the following evaluative learning behaviors:

NOT evaluating to decide if a whole program works; rather, figuring out

when and how it works.

Gathering data from recipients.

Determining outcomes by listening to recipient changes and next

actions.

Engaging people in making meaning from data.

Developing performance management processes, tools and systems.

Prioritizing cost-per-outcome.

17

Evaluative Learning Facilitates Sustainable Growth

41% of nonprofits

grow faster than

the rate of inflation

over a three year

period

Nonprofits where

leaders engage in

these behaviors

are 2.5 times

more likely to

grow faster than

the rate of

inflation.

7%

5%

4%

2%

Conducting All R&D Behaviors (0% of all NPs)

Conducting Many R&D

Behaviors (5%)

Conducting Some R&D Behaviors

(45%)

Conducing Very Few R&D

Behaviors (25%)

Conducting NO R&D Behaviors

(25%)

Avera

ge A

nn

ual

Gro

wth

Rate

Average Annual Growth Rate (Mean Per Year, Based on Three Consecutive Years of Data)

18

Guiding Steps for Conducting Evaluative Learning

1. Measure achievable outcomes, next actions, direct results.

2. Gather lots of data QUICKLY, by quantifying the qualitative.

3. Ask the participant/recipient.

4. Analyze data for patterns of cause-and-effect.

5. Gather a few key illustrative stories.

6. Spend time making meaning and making improvements.

19

Step One: Identify Your Achievable Outcomes

• It’s all about the path to behavior change:

– Awareness +

– Knowledge +

– Attitude +

– Motivation +

– Skills +

– Opportunity +

– Behaviors =

___________________

Habits

=

20

Evaluative Learning Processes to Try

1. Positive Deviant Experiment

2. Literature Review to Make the Case for Proximate Outcomes

3. Proximate Logic Model Creation

4. Survey Data Project

5. Qualitative Data Project

6. Creation of Ongoing R&D and Performance Management Tools

Thanks!