evaluating digital literacy programs

talks!

How to Evaluate Your Digital Literacy Program June 30, 2011Audio is only available by calling this number:Conference Call: 866-740-1260; Access Code: 6339392

Sponsored by

How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392

Using ReadyTalk

• Chat & raise hand • All lines are muted• If you lose your

internet connection, reconnect using the link emailed to you.

• If you lose your phone connection, re-dial the phone number and re-join.

• ReadyTalk support: 800-843-9166


You are being recorded…

• This seminar will be available on the TechSoup website along with past webinar presentations:– www.techsoup.org/go/webinars

• You will receive a link to this presentation, material and links.

• After the webinar, you can ask follow-up questions in our Accessible Technology and Public Computing Community Forum:– http://bit.ly/tsg-pcc

• Twitter hashtags: #techsoup

talks!

How to Evaluate Your Digital Literacy ProgramPresenters: Linda Hofschire & Erick Ruiz

Sponsored by

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.


Community Technology Network


Today’s Speakers

Linda HofschireColorado State Library

Facilitator: Kami Griffiths, Community Technology Network

Assisting with chat questions: Sarah Washburn, TechSoup GlobalStephanie Gerding, TechSoup Global Susan Burkholder, Colorado State Library

Erick RuizInstructional Design Consultant


Agenda

• Overview of evaluation fundamentals• Colorado State Library’s BTOP

program• Systems used to track impact • Long term benefit of tracking data• Tools used to track• Q&A


Poll

• Are you a BTOP grantee?– Yes– No


To measure or NOT to measure

Before you begin• Program objectives

must be developed• Stakeholders must

agree

Realities• ROI is not for every

organization or program

• Are you prepared to handle a negative ROI?

• ROI measurement can add to your program costs

Phillips, P. (2002). The Bottomline on ROI. Atlanta, GA: CEP Press


What is ROI?

• ROI is a measure of accountability• ROI is a mathematical explanation,

comparing program costs to program benefits.– Benefit Cost Ration (BCR)

•BCR = Program Benefits/Program Costs

– Return on Investment (ROI)•ROI (%) = (Net Program

Benefits/Program Costs) X 10


What is ROI?

• Tangible program benefits: expressed quantitatively – BCR and/or ROI

• Intangible program benefits: expressed qualitatively– Benefits that can not be quantified– Benefits that can not be isolated to

your program


ROI Levels

• First...– Program objectives must be developed

• Then...– Level 1: Reaction & Satisfaction– Level 2: Learning– Level 3: Application– Level 4: Business Impact– Level 5: ROI


Level 1: Reaction & Satisfaction

Key Questions• Was the program

relevant?• Was the program

important?• Did the program provide

new information?• Do participants intend to

use what was learned?• Would Participants

recommend the program to others?

Method• Online Surveys• Face-to-face

interviews• Focus Groups• Random Sampling

Phillips, P. & Phillips, J. (2005). Return On Investment Basics. Alexandria, VA: ASTD Press


Level 2: Learning

Key Questions• Did participants

acquire the knowledge and skills presented in the program?

• Do participants know how to apply what they learned?

• Are participants confident to apply what they learned?

Method• Testing

– Norm Referenced Testing

– Criterion Referenced Testing



Level 3: Application

Key Questions• How effective are

participants at applying what they learned?

• How frequently are participants applying what they learned?

• What support are participants receiving?

Method

Participant and “other” • Online Surveys• Face-to-face

interviews• Focus Groups• Random Sampling



Level 4: Business Impact

Key Questions• To what extent did

application of learning improve the measures the program was intended to improve?

• How do you know it was your program that produced these measures?

Method• Survey partners• Survey participants

– Online Surveys– Interviews– Focus Groups



Level 5: Return On Investment

Key Questions• Do the monetary

benefits of the improvement in business impact measures out-weigh the cost of the program?

Method• Financial Measures• Tangible Benefits Data• Intangible Benefits

Data

ROI (%) = (Net Program Benefits/Program Costs) x 100



Colorado Computer Centers: Bridging Colorado’s Great Digital Divide


81 Public Computer Centers (PCCs)


PCCs in Action


•450 desktops•700 laptops•70 ADA compliant workstations•10,000 trainings

By the Numbers…


Areas of Focus•Basic computer skills•Workforce/employment•Health•Resources for adult education•ESL

Partnerships9 partners, including the Gates Foundation, College in Colorado, the Workforce Development Council, and the Colorado Department of Labor


Three Components of Our BTOP Evaluation• Outputs

• Class evaluations

• Immediate outcomes


Planning the Evaluation


Data Map

Data element

Report Collection Point

Quarterly Report

Annual Report

Grant Application

Compliance Officer Website

Personnel $ X X

Broadband speed

X X X X

Type of class

X X X

Equipment $

X X

Scheduled one-to-one tutoring sessions

X

Hours open per 48-hr weekend

X X X X


Laying the Groundwork

• Pretest

• Establish baseline


Outputs


Class Evaluation


Outcomes

• For classes:– Basic technology skills– Employment – Business– Health

• For open lab users:– General


Class Outcomes—Employment

Please check all that apply:After taking this class, I am better able

to…Create or work on a resumeSearch for a job onlineLearn about a potential employerLearn skills for current or new job


General Outcomes

Please check all that apply:During my recent visit to the computer

center, I…Did schoolworkLooked for employmentCommunicated with someone (ex.,

email, online chat, instant messaging, Facebook)


Assessing Outcomes Over Time• Select a sub-sample

• Pre-/post-test

• Case study

• Tell the story behind the numbers


Benefits Beyond the Grant Period


Benefits Beyond the Grant Period• Use results for budgeting, program

planning, advocacy, etc.

• Ongoing evaluation


Tools/Resources

• Impact survey http://impactsurvey.org/

• Colorado State Library’s BTOP site http://coloradovirtuallibrary.org/btop/

• Computer reservations system http://code.google.com/p/powerline/

talks!

Questions?

Submit your questions using the chat box.


Continue the Discussion…

• Additional questions can be answered by posting in in our Community Forum:– http://bit.ly/tsg-pcc


Thank you to our Webinar Sponsor!

ReadyTalk offers dedicated product demos for TechSoup organizations 4 times per week.

For more information: pages.readytalk.com/techsoup.html

talks!

Thank you! Please complete the post event survey!

Kami Griffiths, [email protected]