1

Why are you doing what you’re doing?OUTCOMES-BASED EVALUATION FOR PLANNING AND IMPACT

MIKE CRANDALL & SAMANTHA BECKER

UNIVERSITY OF WASHINGTON INFORMATION SCHOOL

2

Goals for Today’s Session◦ Introduce the basics of outcomes-based evaluation◦ Introduce the key elements of an OBE plan◦ Walk through the OBE process together◦ Prepare you for your own OBE

3

Why Evaluation?WHAT IS OUTCOMES-BASED EVALUATION?

HOW IS IT USEFUL TO ME?

4

What is outcomes-based evaluation?

Defines Documents

Indicators of achievement

Methods for data

collection

Needs of community

Goals of providing services

HowWhy

SuccessRoom for

improvement

Focuses

5

Practice brings added benefits

Outcomes-based evaluation is part of a culture of performance management, and when done well:

◦ Fosters a culture of organizational learning ◦ Fosters intentionality, clarity, and specificity◦ Helps clarify goals of services◦ Prevents mission creep◦ Gets staff involved ◦ Frames successful advocacy◦ Is useful for decision making◦ Is integrated into organization (not just one person’s job)

6

The processAN OVERVIEW OF THE 9-STEPS TO AN EVALUATION PLAN

7

9 steps to an outcomes evaluation

Assemble team Conduct needs assessment Map change

Develop logic model

Develop evaluation questions

Select indicators

Identify data sources

Create analysis plan

Employ reflective practice

8

Getting startedLAYING THE GROUNDWORK FOR YOUR EVALUATION PLAN

9

Step 1: Assemble your working team

Program staff/managers

Library friends

Patrons/end-users

Other orgs/agencies/departments

People who can help you:◦ Understand the needs of your

community◦ Collect and analyze data◦ Understand expectations in the

political environment◦ Define success◦ Roll up their sleeves and do the work

10

Step 2: Needs assessment

Resources/Input Activity Output Outcomes Impact

11

Documented need• Gather sources of data about your community and the

population you serve:• Census data• Labor/business statistics• Schools• Municipal surveys• Strategic plans• News articles/op-eds• Mayor’s speech

◦ Assemble a community profile like the case studies

12

Field work for needs assessment◦ Find out about other community needs◦ Outreach to populations that don’t come into the library◦ Discovery of outcomes from library use and how change occurs

(what works)◦ Expectations about what the library should do/role in the community

◦ Can be done through◦ Interviews◦ Focus groups◦ Community forums◦ Observations◦ Surveys

13

Step 3: Map the theory of change

• Express how you see your activities as creating change in your target population to bridge what you do with what you want to accomplish for the community.

Resources/Inputs Activities Outputs Outcomes Impacts

Theory of change

Theory of change: Having, Knowing, Doing

Need

Books

Reference

Public technology

Programs

Ability to do

Potential to

become

to help your patrons get here?

And why/how do you think it helps?

What do you do here…

15

Theory of change examples

Impact intention:Help people gain self-sufficiency

Activities:◦ Public computer sessions◦ Digital literacy training◦ Job search classes◦ Resume preparation

Theory of change◦ Access to computers enables◦ Learning digital literacy skills leads to◦ Having skills necessary to fill job vacancies

and look for work so that◦ Patrons are able to get hired in order to◦ Earn an income so they can◦ Be self-sufficient

Digitization impact intention:Help community have increased access to digital library resources

Activities:◦ Digitization training for library staff◦ Community repository◦ Access to databases

Theory of change◦ Digitization training enables◦ Libraries to gain knowledge and skills to create

local digital collections leads to ◦ Libraries take a coordinating role in building

community collections so that◦ Libraries build partnerships, increasing their

visibility, resource base, and sustainability so that

◦ Residents access digital holdings and realize enhanced research outcomes

Activity 1WRITE A THEORY OF CHANGE ABOUT THE PROGRAM YOU WANT TO EVALUATE

17

Teen STEM program TOC

Teens in our community go to one of the best high schools in the county, have afterschool activities, and generally come from households with higher incomes and educational achievement,

As a result of these other resources available for teens in Pebble Pond, library use among teens has gone down.

The library wants to engage teens and provide targeted services for them so that they begin attending teen programming,

So that librarians can begin to build relationships with teens, so that the library can create a teen advisory committee

So that the library can hear more easily from teens, so they can make better programmatic decisions, so that teens like coming to library programs, so that the number of teens reached by the library increases, so that

Libraries can begin introducing STEM topics into programming and working with schools to align STEM activities with school curriculum, so that

Teens have enrichment activities and supportive resources that will help them engage in STEM topics, so that they gain confidence and experience with STEM occupations, so they feel more inclined to pursue other STEM experiences, so that they study STEM fields in college

18

Developing metricsTHREE STEPS TO HELP FIGURE OUT WHAT TO MEASURE

19

Inputs•Program resources used to produce outputs and outcomes•Dollars•Program staff•Equipment

Activities•Activities and services delivered; expected to lead to outcomes• Number trainings provided to grantees

Outputs•The quantity of activities completed•Hours of training attended

•Number of people served by portal

•Number of item-level views

Outcomes•Short-term - changes in knowledge, awareness, skills

•Intermediate - changes in actions, behaviors, policies

Impacts•The big picture results of what the program did•Relates to your mission

•Aspects of the clients or community that you are trying to change

Step 4: Develop logic model

Your Planned Work Your Intended Results

______________________________________________________________________________

20

Teen STEM program logic model

Inputs•Teen librarians•Teen library space/room•Programming budget

Activities• Form and maintain a teen advisory

committee• Conduct a teen needs assessment• Library staff coordinate with schools• Library develops and delivers teen

STEM programs

Outputs• The number of meetings of the teen

advisory committee• Actionable conclusions from the

needs assessment• Number of new partnerships with

schools• Number of new teen STEM programs• Hours of teen participation in new

STEM programs

Outcomes• Early• Increase in the number of teens

visiting the library• Increase in the number of teens

engaged in library leadership activities

• Increase in the numbers of teachers who regard the library as an educational partner

• Intermediate• Teens have increased confidence in

STEM participation• Teachers and the library regularly

coordinate curriculum with library programming to support STEM education

Impacts• Teens have greater support for

pursuing STEM-related education and career paths

• More teens from Pebble Pond end up in STEM-related educational programs

• More teens from Pebble Pond enter STEM-related fields

• Businesses recognize Pebble Pond for having STEM-ready young workforce and relocate.

• The average earnings for residents of Pebble Pond increases

21

Activity 2DEVELOPING YOUR EVALUATION QUESTIONS

22

Step 5: Develop evaluation questions

Identify what you want to learn about your program and its results

◦ Who will use what you learn and for what purposes?

Logic model provides a framework when brainstorming

Questions will guide your choice of indicators, data collection, and reporting

23

Teen STEM evaluation questionsIn

puts Teen librarians

Teen library space/roomProgramming budget

Activ

ities Form and maintain a

teen advisory committeeConduct a teen needs assessmentLibrary staff coordinate with schoolsLibrary develops and delivers teen STEM programs

Out

puts The number of

meetings of the teen advisory committeeActionable conclusions from the needs assessmentNumber of new partnerships with schoolsNumber of new teen STEM programsHours of teen participation in new STEM programsNumber of teen visits to the library

Out

com

es EarlyIncrease in the number of teens visiting the libraryIncrease in the number of teens engaged in library leadership activitiesIncrease in the numbers of teachers who regard the library as an educational partnerIntermediateTeens have increased confidence in STEM participationTeachers and the library regularly coordinate curriculum with library programming to support STEM education

Impa

cts Teens have greater

support for pursuing STEM-related education and career pathsMore teens from Pebble Pond end up in STEM-related educational programsMore teens from Pebble Pond enter STEM-related fieldsBusinesses recognize Pebble Pond for having STEM-ready young workforce and relocate. The average earnings for residents of Pebble Pond increases

Why has teen use declined?

Does STEM programming entice teens to come to the

library?Does engaging with

schools/teachers result in better programming?

What aspects of the program support STEM confidence?

Do we have enough

resources to support a

STEM program?

24

Step 6: Select your indicators

Indicators need to meet certain utilitarian standards. Beyond the actual content of the indicator, they should also be:

Specific, unique and unambiguous

Observable, practical, cost effective to collect, and measurable

Understandable and comprehensible

Relevant (measures important dimensions, are appropriate and related to the program, that are of significance, predictive and timely)

Time bound

Valid, providing reliable, accurate, unbiased, consistent, and verifiable data

(Hatry, 2006)

25

Frame outcomes as indicators

Turn your expected outcomes into measurable indicators, e.g. we want library staff to increase their technology skills

◦ Number of library staff able to complete digitization tasks who were unable to do so previously

◦ Number of library staff reporting increased technology confidence, knowledge/skills

◦ Number of library staff who report applying new technology knowledge/skills to other tasks/projects

For Teen STEM program◦ Increase in the number of teens visiting the library◦ The monthly change in the number of teens visiting the library

26

Look for existing indicators

◦ Many organizations and researchers have invested time and money in developing good indicators for a variety of social/public programs, as well as data collection procedures you can borrow.

◦ Using existing indicators has the added bonus of providing you with benchmarks to compare against, since you will be using the same metrics

http://www.communityindicators.net/

27

Making the planTHE FINAL STEPS FOR OPERATIONALIZING YOUR EVALUATION PLAN

Step 7: Identify data sourcesQUANTITATIVE DATA/COLLECTION

Get at the “what”◦ Descriptive data about your

population◦ Programmatic data◦ Survey data

Often used in summative evaluation & formal reports

QUALITATIVE DATA/COLLECTION

Get at the “how” and “why”◦ Interviews◦ Focus groups◦ Survey data (open-ended

questions)◦ Observations◦ Interactive/instructor collected

Often used in formative evaluation & for storytelling

28

29

Best practices for collecting original data

Understand your ethical obligations◦ Respect your subjects◦ Don’t ask more than you need to

know◦ Know your human subjects

requirements

Have a plan & a purpose◦ Identify what kind of data you need◦ Identify data sources, method, timing

Collect for quality, not quantity◦ Test your instruments ◦ Have interview guides◦ Be consistent and timely

Use existing indicators and data collection procedures from other orgs.

Look for opportunities to gather outcome data that you can observe (rather than asking in a survey):

◦ Students sending instructor email◦ Completed projects◦ Numbers of new partners

30

Step 8: Create your analysis plan

Use data to answer your evaluation questions◦ What happened: What did we do? What resulted? How

and why?◦ What worked, for whom, in what ways, under what

conditions, with what implications?◦ Are we doing the right things? (Testing program’s theory

of change)◦ Are we doing them right? (Assessing implementation

quality)

31

Sample from analysis planEvaluation questions:• How many people use the library computers to look for jobs?• What kinds of people use library computers for looking for jobs?• Are people using computers for job seeking different than people who use

library computers for other purposes?

Self-reported characteristics of public access technology usersWeighting? YesUnivariate or Bivariate? BivariateIndependent var(s) and scale? Z_income

Z_race_catsZ_education

ContinuousCategoricalOrdinal

Dependent var(s) and scale? N/AIs dependent normal? N/AProposed test (if normal) One-sample t-test

Point or interval estimate of population mean, Margins of errorProposed test (if not normal) Chi-square goodness-of-fit test (grouped scores)

Point or interval estimate of population mean, Margins of errorNotes Check CIs and SEs, as well as skew/kurtosis.

If non-normal, use quartiles.

32

Step 9: Employ reflective practice

Telling your story

Wise program decisionsAccountability

33

Questions & Discussion

34

Resources to help you Taking Stock: A Practical Guide to Evaluating Your Own Programs (Horizon Research)

http://www.horizon-research.com/reports/1997/taking_stock.php

Hatry, H. P. (2008). Performance measurement: Getting results. Washington, DC: The Urban Inst. Press.

The Urban Institute: Outcome Indicators Project http://www.urban.org/center/cnp/projects/outcomeindicators.cfm

University of Wisconsin-Extension Program Development and Evaluation Resources http://www.uwex.edu/ces/pdande/evaluation/index.html

Crandall, M. & Fisher, K.E. (2009). Digital inclusion: Measuring the impact of information and community technology. Medford, NJ: Information Today, Inc.

Journal reflection What are some takeaways you have from this session?

How might you use these concepts, tools, and/or resources in your evaluation project?

What challenges might you experience when implementing these new concepts, tools, and/or resources? What are some strategies you can use to address these challenges?

List 1-3 action items related to what you learned in this session. When will you complete each of these items?