measure what matters karen macdonald, bgca midwest leadership conference october 16, 2009

29
Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

Upload: samuel-bennett-ryan

Post on 01-Jan-2016

230 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

Measure What Matters

Karen MacDonald, BGCA

Midwest Leadership Conference

October 16, 2009

Page 2: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

“Not everything that counts can be counted, not

everything that can be counted counts.”

Page 4: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

“In God we trust, all others bring data.”

Page 5: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

“Data is the new sexy!”

Page 6: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

Why is Outcome Measurement important?

Page 7: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

• Era of accountability

• Doing good is not enough

• Stakeholders are asking more difficult

questions

The Importance of Measurement

Page 8: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

Formal Program Evaluation Outcome Measurement

Does the program work? How well is it working?

• Proving • Improving

• Research tool • Management tool

• Applies to programs • Applies across org

• Analyzing • Learning

• Theory of Change • Best Practices

Program Evaluation vs Outcome Measurement:

Page 9: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009
Page 10: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

Key Elements of Impact

• Safe, positive environment• Fun• Supportive relationships• Opportunities and expectations• Recognition

Page 11: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

Making Every Day Count

• Longitudinal Study • Tracked 7th and 8th graders

for 30 months• Linked levels of

attendance at Boys & Girls Clubs to outcomes

• Finding – MORE IS BETTER

Page 12: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

Outcome (All changes are positive changes)

Minimum Level of Attendance Over 30-Months Related to Positive Change in

Outcome

52 Days 122 Days 244 Days 366 Days

School effortConfident in abilityIntegrityCommunity service Initiate marijuana use Future connectedness

ShynessAggressionInitiate weapon carryingStopped by policeInitiate drinking Initiate Smoking

Number times skippedInitiate sexual activity

Negative peers

Page 13: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

Four things led to teen participation…

1. Fully engaged in a variety of activities

2. Friends came to the Club3. Involved in leadership roles4. First became involved as

preteens or even younger

Page 14: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

Making Every Day Count affirms…

• Attendance matters• Participation matters• Fun matters• Relationships matter• Programs matter

Page 15: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

Annual VisitsDefinition: The number of times a member

attends the Club over the course of the year.

• It’s important to track this metric.• 52 - 104 times per year is the baseline• Increase attend levels of all members

regardless of current attendance patterns

Page 16: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

A big thank you to…• Boys & Girls Clubs

of Greater Milwaukee

• Boys & Girls Clubs of Omaha

Page 17: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

Planning a Measurement Strategy:Key Steps

• Identify key organizational priorities and select

key indicators

• Track attendance and program participation• Locate and collect data• Understand and use data• Communicate results

Page 18: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009
Page 19: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

•Review sources that outline your priorities

including your strategic plan

•Review the needs of your community

•Involve your board by asking questions

Identify Organizational Priorities

Page 20: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

Success Equation Components

A + B + C = D A + B + C = D

What are you ultimately trying to accomplish?

Priority OutcomesWhat changes in behavior or condition are required?

IndicatorsHow do we measure progress

against our outcomes?

Key StrategiesWhich

programs or activities will

drive results?

© 2008 Mission Measurement LLC – as adapted from Benchmarking for Nonprofits, by Jason Saul

Page 21: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

• Use your success equation and priority outcomes as your guide

• Start with the basics: attendance and program participation

• Continue with more complex indicators: changes in knowledge/attitude and changes in behavior

• Use the Outcomes Framework as a guide

• Select indicators that are feasible to collect and credible to communicate

Select Indicators to Track Progress

Page 22: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

•Practical – Can you realistically capture this data without a lot of cost or difficulty?

•Actionable – Will the data allow you to make decisions in a timely manner?

•Relevant – Does it answer the “so what” question?

•Controllable – Is it within your sphere of influence or control?

Characteristics of Good IndicatorsCharacteristics of Good Indicators

Page 23: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

• more time spent going to the Club was linked to positive change

• research supports the importance of regular attendance and the links to positive outcomes.

• at least 52 times per year is a baseline • Clubs should strive to increase the attendance levels of all

members • Average daily attendance is an important measure of

Club’s operational capacity, but does not measure individual members and is therefore not an indicator of changes for youth.

Track Attendance and Program Participation

Page 24: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

Locate and Collect DataLocate and Collect Data• Collect on a program-by-program basis• Leverage existing tools, i.e. test or surveys• Use Club-wide surveys when warranted• Tests or Surveys can be administered by hand or electronically• Results from the tests or surveys can be tracked in the member’s management system.

Page 25: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

Outcome Indicator Indicator Definition Location of data Frequency Collect Where

Report

Academic Achievement

Indicator 1

Indicator 2

Indicator 3

Locate and Collect DataLocate and Collect Data

Page 26: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

• Improve program and/or youth results • Share progress and success

• Share results alone • 50% of members participated in community

service.• Share results compared past performance

(growth/change)• Participation in community service activities has

increased by 25%.

Understand and Use ResultsUnderstand and Use Results

Page 27: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

Indicator Sample Impact Statement Links to External Research

Number of volunteer hours completed by Club members

75% of Boys & Girls Clubs of Main Street members participated in volunteer service activities in 2008 and contributed a total of 1,000,000 hours of service.

Young people who volunteer are likely to do better in school, graduate, maintain positive attitudes towards society and avoid risky behaviors (4)

Simply based on regular member attendance and program participation, Clubs can demonstrate the impact they are having by linking participation to external research on positive activities:

Page 28: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

• Explore and define how measurement can and will be used

• Setting the right tone and creating ownership and buy-in

• Setting the process in motion • Identify the processes and plans needed to bring

measurement to life

Engage your StaffEngage your Staff

Page 29: Measure What Matters Karen MacDonald, BGCA Midwest Leadership Conference October 16, 2009

Available Resourceswww.bgca.net/demonstratingimpact

• Demonstrating Impact Guide• Outcome Measurement Framework• BGCA - Public/Private Ventures Study• Distance Learning courses

Abigail Diner, [email protected] MacDonald, [email protected]

Romero Brown, [email protected]