measuring and using outcomes for volunteer programs

July 10, 2014

Measuring and Using Outcomes for Volunteer Programs

2014 Greater Richmond Association for Volunteer Administration Conference

Outcome measurement overview: What, why, how

Identifying outcomes

Identifying your volunteer program’s outcomes

Identifying indicators, data sources, data collection methods

Identifying indicators, data sources & data collection methods for your volunteer program

Using outcome data to learn and improve

Parting advice

Resources

Measuring and Using Outcomes for Volunteer Programs 2

Topics

What

3Measuring and Using Outcomes for Volunteer Programs

ACTIVITIES

What the program does with inputs to fulfill its missione.g.

feed and shelter homeless families

provide job training educate teachers

about signs of child abuse

counsel pregnant women

provide mentors for youth

OUTPUTS

The direct products of program activitiese.g.

number of classes taught

number of counseling sessions conducted

number of educational materials distributed

number of hours of service delivered

number of participants served

OUTCOMES

Benefits or changes for participants during or after program activitiese.g.

new knowledge increased skills changed attitudes

or values modified behavior improved condition altered status

Source: Measuring Program Outcomes: A Practical Approach. United Way of America, 1996

INPUTS

Resources dedicated to or consumed by the program

e.g.

money staff & staff time volunteers &

volunteer time facilities equipment &

supplies

Program elements


nothow many worms

the bird feeds its young,but

how well the fledgling flies

Outcome:


Outcome: Benefit or change for intended beneficiaries or target group

e.g., Parents will use more effective discipline methods

Outcome indicator: The specific statistic used to show how fully the program is achieving the outcome

e.g., The number and percent of parents who, after completing the program, explain rules and consequences and apply consequences each time a rule is broken

Outcome target: Numerical objective for a program’s level of achievement on an outcome

e.g., 75% of parents will explain rules and consequences and apply consequences each time a rule is broken


Outcome vs. indicator vs. target

Don’t set targets for outcome achievement until you have collected 1 or 2 rounds outcome data

Until you know your current level of achievement you won’t know what target is reasonable; you’ll have to guess at a target

Whether you meet, exceed, or fall below the target won’t reflect your effectiveness; it will reflect how well you guessed


A note about targets

Experimental research

Program evaluation

Performance measurement

A replacement for tracking inputs, activities, and outputs


Outcome measurement is NOT. . .

Show whether what you are measuring is the right outcome for your program’s activities

Prove statistically that your program caused the outcome

Explain why this level of outcome was achieved

By themselves, tell how to improve the outcome


Outcome findings will NOT. . .

So…


Why


Increaseeffectiveness

Communicatevalue

Why measure outcomes?


Provide direction for staff

Identify training needs

Improve measurement and/or program

Support long-range & annual planning

Guide budgets & justify resource allocations

Suggest outcome targets

Focus board members’ attention on programmatic issues


Internal uses of program outcome findings

Recruit talented staff and volunteers

Promote the program to potential participants and referral sources

Identify partners for collaboration

Enhance the program’s public image

Retain and increase funding


External uses of program outcome findings

Executives of 298 programs agreed or strongly agreed – outcome measurement had helped their program:

Focus staff on shared goals (88%)

Communicate results to stakeholders (88%)

Clarify program purpose (86%)

Identify effective practices (84%)

Compete for resources (83%)

Enhance record keeping (80%)

Improve service delivery (76%)

Nearly all (89%) would recommend outcome measurement to the director of a similar program


Benefits of program outcome measurement

Source: Agency Experiences with Outcome Measurement. United Way of America, 2000

How


7. Improve your system

6. Analyze and report your findings

5. Try out your outcomemeasurement system

4. Prepare to collect data on your indicators

8. Use your

findings

1. Get ready

2. Choose the outcomes you want to measure

3. Specify indicators for your outcomes

17

Outcome measurement: 8 steps


Source: Measuring Program Outcomes: A Practical Approach. United Way of America, 1996

Outcomes are appropriate for the program

Indicators provide useful data on outcome achievement

Methods – data sources and data collection procedures – are sound

Tools measure what they’re supposed to and are appropriate for the population

Analysis shows level of achievement for different segments of the target audience and different program characteristics


In good outcome measurement …







Parting advice

Resources


Topics

The specific results(s) sought for a specific audience

(clients, participants, visitors, patrons, volunteers, etc.)


“Outcome”

First question:



Not “What will we measure?”

Instead “What result do we want our activities to achieve for whom?”

Results for volunteers

e.g.:• Volunteers provide

accurate information• Volunteers make a

long-term commitment

Results for clients/ visitors/patrons

e.g.:• Clients receive desired/

needed information• Patrons recommend the

setting to others

Volunteer programs: Results for two audiences


Volunteer program administration activities

Volunteers’ activities







Parting advice

Resources


Topics

Worksheet A:Activities of your volunteers


Identifying your volunteer program’s outcomesPart 1




setting to others


Worksheet B Section II:Outcomes you want your volunteers to create






setting to others


IF … these activities

THEN … this change



26

Activities Outcomes: An “If…then” chain


IF … these activities

THEN … this change in knowledge

THEN … this change in behavior

THEN … this change in condition/status

27

A simple “If…then” chain


Worksheet C:If … then chain linking volunteers’ activities and outcomes for target audience






setting to others


Are the outcomes really outcomes?

Do they describe changes in or benefits for clients/ participants/beneficiaries, rather than something about the program?

Is the longest-term outcome meaningful for program clients/participants/beneficiaries?

Will the change or benefit make a real and lasting difference for them?

Is the logic logical?

Are the activities sufficient to produce the intended outcomes for these clients/participants/beneficiaries?

29

Testing your “If … then” chain


Mentors meet with at-risk teens for an hour each week. Mentors stress the importance of

education, may help teens with homework.

At-risk teens know homework is important.

At-risk teens graduate from high school.

At-risk teens meet district grade requirements.

At-risk teens complete homework regularly.

Act

iviti

esO

utco

mes

30

Are the activities sufficient to achieve the outcomes?


Worksheet D:Testing your If … then chain



Outcome A

Outcome C

Outcome B

Activities

1 & 4

Activities3 & 5

Activity 2

Worksheet E:Impact you want to have on your volunteers










Worksheet F:If … then chain linking volunteer program administration activitiesand outcomes for volunteers






Worksheet G:Testing your If … then chain

Outcome A

Outcome C

Outcome B

Activities

1 & 4

Activities3 & 5Activity 2



Requires time, thought, discussion

Benefits from input by volunteers, clients/visitors/patrons, board members, and engaged donors as well as staff

Evolves over time

Provides content for your program’s “logic model”

Is worth the effort

ACTIVITIES

OUTPUTS

OUTCOMES

INPUTS



Stimulates discussion of the program’s purpose

Creates a shared understanding of how activities are intended to link to results

Often prompts adjustments in activities and intended results

Helps communicate the program’s intent to stakeholders

Makes sure useful data is collected – and unnecessary data is not

Guides review of data to identify improvement opportunities

Often leads to improved outcome performance because now everyone understands the results the program is aiming for


Programs report that defining outcomes and creating logic models …







Parting advice

Resources


Topics

Specific statistic used to show how fully the outcome is being achieved

Often the number and percentage of clients/patrons/ volunteers who, e.g.:

- attain specific knowledge

- demonstrate a needed skill

- display a certain behavior

- achieve a desired status

38

“Indicator”


Number and percentage of volunteers in the rotating gallery who can answer 25 FAQs about the current exhibit correctly

Number and percentage of activity volunteers in the senior center who use techniques for engaging withdrawn seniors

Number and percentage of fund-raising volunteers who renew their commitment for 1 year of service

Number and percentage of children in the activity program who read at least 5 books with their volunteer reading buddy

Number and percentage of patrons in the volunteer-staffed gift shop who make or add to a purchase

39

Examples of indicators


Questions to ask to generate options:

How will we know if the outcome is achieved? What will we see? What can we count or measure?

If you’re stuck, ask:

How would we know if the intended result is not achieved? What would we see?

If we were successful in one instance but not another, how would we know? What would we see in one instance but not the other?


Identifying possible indicators

Does it capture the central point of the outcome? (e.g., if the central point is behavior, it tracks behavior – not knowledge)

Is it specific enough that everyone interprets it the same way?

Is it unlikely to be influenced by factors others than the outcome?

Do key stakeholders accept it as a measure of the outcome?


Testing a possible indicator

Case/client records

Documents

Data files

The volunteers/clients/visitors/patrons themselves

Others who know the volunteers/clients/visitors/patrons

Other individuals

Physical environment


Possible sources of program outcome data

Record review

Document review

Data file review

Questionnaire

Interview

Observation

Testing

Mechanical measurement


Possible methods of collecting program outcome data







Parting advice

Resources


Topics



Worksheet H:Indicators, data sources, data collection methods for your outcomes

Outcome Indicator(s) Data Source(s)Data Collection

Method/Tool

Outcome A

Outcome B

Outcome C







Parting advice

Resources


Topics

Calculate overall outcomes — # and % of target audience achieving the outcome

e.g.:

Total number served 1,000

Number achieving outcome 535

Percent achieving outcome 53.5%



Calculate overall outcomes

Break out results by:

level of outcome

relevant characteristics of target audience

relevant program characteristics

combinations

to see where you are having more and less success



Mentors meet with at-risk teens for an hour each week. Mentors stress the importance of education, encourage school attendance, occasionally help with homework.

At-risk teens achievepassing grades.

At-risk teens earn better grades.


At-risk teens meet district attendance requirements.


At-risk teens attend school regularly.

Outcomes by level of outcome

60%

85%









Outcomes by client characteristic: Parent attended teacher conference

Yes75%

No45%









Outcomes by program characteristic: Mentors meet monthly with mentee’s teacher

Yes55%

No40%









Outcomes by client & program characteristics: Boys only, by mentor’s sexMentor is

a Woman

45% 70%

80% 80%

Mentor is a Man



Break out results by target audience and program characteristics

With caution, compare current outcomes with:

previous outcomes

outcome targets

outcomes of similar programs

relevant standards





With caution, compare current outcomes with other points of reference

Analyze findings

Review and discuss on a regular basis

Seek input from program volunteers, clients/patrons and others

Recommend improvements



Logic (theory) problems: Assumptions about the relationship among activities and outcomes are questionable (e.g., will an hour of contact a week really change behavior?)

Program problems: The logic is sound, but the program wasn’t implemented as planned (e.g., 55% of teens missed more than half of the sessions)

Measurement problems: An issue with how the data was collected means the numbers are questionable (e.g., many more responses from older teens than from younger ones may skew results)


In analyzing findings for improvement opportunities, look for . . .



With caution, compare current outcomes with other points of reference

Analyze findings

Implement improvements and track results

Celebrate learning!

Repeat









Parting advice

Resources


Topics

Make it work for YOU

Your first priority in outcome measurement


Don’t kill what you won’t eat


Meg PlantzHelping organizations improve lives through collaborative community change

[email protected]
