some practical tips for measuring financial success dr. angela lyons university of illinois...

Some Practical Tips for Measuring Financial Success

Dr. Angela LyonsUniversity of IllinoisMeasuring Financial Success

Using Program Evaluation

Presented byDr. Angela Lyons

University of Illinois

November 2008

The Million Dollar Question

At the end of the day, does financial education

make a difference?

Lessons Learned from Current Research

Jump$tart Survey of Financial Literacy Among High School Students – Captures knowledge levels

NEFE High School Financial Planning Program – Impact

of formal financial education on confidence levels and behaviors of high school students

Bernheim, Garrett, and Maki (2001) – Effect of mandated

financial education during high school (longitudinal study)

FDICs Money Smart Program – Moving the unbanked into the financial mainstream

See “Reading List” for recent research on financial education and program evaluation.

Becoming a critical evaluator is important!

Read media stories carefully.

Look at the samples being used.

Information vs. education.

Planned behavior vs. actual behavior.

Avoid focusing only on the successes.

Think beyond participants’ finances.

Be aware of the barriers and challenges related to measuring program impact.

An Overview of the Training Session

I. Setting the Stage for Program “Success”

II. The Evaluation Process: Creating Your “Toolkit”

III. Putting It All Together: Sample Evaluations

IV. NEFE Financial Education and Program Evaluation Toolkit ®

V. Barriers and Challenges to Building Successful Programs

VI. Building Program Success: Reporting Program Impact

Part I

Setting the Stage for Program “Success”

Current evaluation efforts are still far from satisfactory.

General lack of evaluation capacity and understanding of how to conduct effective evaluations.

Evaluation is still often treated as an after thought; needs to be built into the design of the program upfront.

Lack of attention given to evaluation at all levels.

Need for “industry” standards for program evaluation.

Current State of Program Evaluation

Source: Lyons, A. C., Palmer, L., Jayaratne, K.S.U., and Scherpf, E. (2006). "Are We Making the Grade? A National Overview of Financial Education and Program Evaluation.” The Journal of Consumer Affairs, 40(2), 208-235.

One non-profit administrator commented….

“The people that typically end up [being] told that they have to do evaluation, it’s ‘dumped’ on them and it’s usually not a person that has any experience with financial education or expertise in evaluation. They’re pretty much told here’s your new hat, we’ve been told we have to do this and here’s your new hat, and they don’t know. It’s not for lack of wanting to do a good evaluation or trying to do a good evaluation. They just don’t know—it’s not the right person trying to oversee it.”

On the front lines….

“What even is an evaluation?”

“What do we mean by evaluation?

“How do we know if participants are getting better? It’s difficult to assess.”

“What are we trying to measure? There’s a lot of confusion out there.”

“What constitutes a successful, or even acceptable, evaluation?”

Take stock of who you are – What is “your vision?”

Conduct a needs assessment.

Collect baseline information from your target audience.

Identify your signature program(s).

Identify your program objectives. Be realistic!

Create an evaluation action plan.

What do you want to accomplish? At the end of the day, what do you want to show?

Getting Started: Thinking like an evaluator….(Program Planning Guide)

What is an outcome-based evaluation?

Outcomes are benefits to clients from participating in the program.

What do you want your participants to know or be able to do when they have finished the program?

Outcomes are usually in terms of enhanced learning and improved behaviors.

Outcomes are often confused with program outputs or units of service (e.g., number of clients who went through the program).

The Logic Model

• A picture of the program.

• Simple representation of the program “theory” or “action” which explains the program and what it is to accomplish.

• Shows relationships between inputs, outputs, and outcomes.

The Logic Model (conti.)

INPUTS

OUTPUTS OUTCOMES

Resources used to develop the program are called inputs. Time and money are the most common inputs needed to implement educational programs.

If inputs are invested into the financial education program, then learning opportunities will be created for the target audience. The created educational materials, services, and opportunities are called the program outputs.

Changes in participants’ perceptions, knowledge, and behavior that represent real impact in their lives. The benefits derived by the participants from the program are called outcomes.

University of Wisconsin - Extensionhttp://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

Impact Hierarchy of Outcomes

Changes in Perceptions and Levels of Satisfaction

Changes in Knowledge, Attitudes, Skills, and Aspirations (KASA)

Changes in Behavior/Practices

Planned End-Result or Improved Socio-Economic Condition

Another useful framework….Transtheoretical Model of Behavior Change (TTM)

TTM integrates major psychological theories into a theory of behavior change.

Used to identify the state at which individuals are ready and able to change their financial behaviors.

Appropriate educational interventions are then tailored to meet individual’s specific needs at that particular stage.

5 Stages of Change

Precontemplation • Individual not ready to take action and change

behavior in the immediate future.• Rarely seeks help and rarely uses information.

Contemplation• Individual is getting ready to take action and intends

to change behavior in next 6 months.• Open to education.

Preparation• Individual is ready to take action and intends to change

behavior in next 30 days.• Practices behavior by taking small steps towards the goal.• Seeks information and support, but often concerned that

changing behavior may be too difficult and they may not succeed.

5 Stages of Change (conti.)

Action• Individual changes behavior and maintains behavior for at

least 6 months.• Believes they can change.• Can control “triggers” that cause them to relapse into old

behaviors.• Has a support system to get them through challenging times.

Maintenance• Individual has changed behavior and it has lasted for more

than 6 months.• May relapse into old behaviors, but can overcome temptations

so that behavior becomes permanent.• Can assess the conditions under which relapse might occur.• Can establish successful coping strategies.

Example:

Financial Practice

Do not plan to

do

Plan to do in the

next month

Plan to do in the next 2-6 months

Have been

doing for 1-6

months

Have been

doing for more

than 6 months

Set short and long-term financial goals.

Save money regularly.

Use a spending plan to track income and expenses.

Maintain sufficient balances in bank account(s).

Pay bills on time each month.

Review bills each month for accuracy.

Comparison shop before making purchases.

Pay off credit card balances in full each month.

Identifying Program Objectives

Objectives should be:SpecificMeasurableAchievable and observableReasonableTime specific

S.M.A.R.T. objective statements should clearly define what you want to achieve with your program.

They should list the end outcomes the program intends to affect or change.

Writing objective statements

First-time home buyer education program

The objectives of this program are to: Develop first-time home buyers’ ability to shop for the lowest mortgage

interest rate. Teach first-time home buyers how to save money for closing costs. Teach first-time home buyers how to assess affordable housing.

Debt reduction education program

The objectives of this program are to: Develop participants’ ability to identify needs and wants separately. Develop participants’ ability to control “wants” to reduce expenditures. Develop participants’ ability to avoid impulse and emotional spending.

Achieving your objectives:Selecting appropriate indicators

General Indicators (objective and subjective):

Number of programs, participants, etc. Knowledge gains Changes in attitudes and satisfaction Changes in skills and confidence Changes in intended and actual behaviors

Specific Indicators (objective):

Actual dollar changes (reduce debt, increase savings) Development of financial plans Changes in spending habits Building or rebuilding credit reports and credit scores

Signature Program

Program Objective 1

Program Objective 2

Program Objective 3

Indicator 1a Indicator 2a Indicator 3a

Indicator 1b

Indicator 1c

Indicator 2b

Indicator 2c

Indicator 3b

Indicator 3c

Final program outcomes

ACTIVITY: Your Evaluation Road Map

Signature Program

Objectives Evaluation

Questions

Indicators Final Outcomes

ACTIVITY: Defining Your Objectives(Evaluation Action Plan – Part A)

What is your signature program (e.g., course, workshop, educational

materials, initiative, campaign)?

Signature Programs

1.

2.

3.

4.

5.

At the end of the day, what do you want to show?

1.

2.

3.

Who are your target audience(s)?

1. 5.

2. 6.

3. 7.

4. 8.

What are your delivery method(s) (e.g., in-person, telephone, Internet)?

1.

2.

3.

4.

Who will use the evaluation and how?

Who will use it? How will it be used?

Part IIThe Evaluation Planning Process:

Creating Your “Toolkit”

What evaluation method should you use to collect impact data?

Surveys

Focus groups

Interviews

Observations

Case studies

Tests of ability

Some examples from financial education….

Questions to ask yourself ….

What are the pros and cons of each method? What is the purpose of the evaluation? Who will use the information – and how? What information do you want to collect? Who is your target audience? What is your primary delivery method? What are your available resources (i.e., time, money,

and staff)? What is your timeline? What is your expertise and evaluation capacity? Who are your partners, funders, and stakeholders?

Common survey methods used to collect impact data

• Post evaluation only

• Retrospective pre-test (RPT)

• Pre and post evaluation

• Follow-up

• Stages to Change (TTM)

• Control groups and longitudinal studies

Key question to ask:

What is the length of your program?

Post evaluation only

When to use: Short programs that are less than 2 hours

Advantages: Only need to survey group once. Good for limited-resource audiences and groups that are

transient. Relatively inexpensive and less time intensive. Can document participants’ levels of knowledge, skills, and

planned behaviors at the end of the program.

Disadvantages: With no pre assessment, it’s difficult to document potential and

actual changes in knowledge, attitudes, and behavior.

Retrospective pre-tests (RPTs); The Post-Then-Pre Evaluation

Retrospective pre-test (RPTs)

When to use: Any program, but typically 2 hours or less

Advantages: Only need to survey group once. Good for limited-resource audiences and groups that are

transient. Controls for “response shift bias.” Can document “relative” change.

Disadvantages: Potential for respondent bias (social desirability factor). Self-assessment measures are subjective.

36

RPTs (continued)

Examples and more info on RPTs:

“Collecting Evaluation Data: End-of-Session Questionnaires.” University of Wisconsin-Extension. www.uwex.edu/ces/pdande/evaluation/evaldocs.html

Lyons, A. C., Y. Chang, and E. Scherpf. “Translating Financial Education into Behavior Change for Low-Income Populations.” Financial Counseling and Planning Journal, 17(2): 27-45.

Chang, Y., and A. C. Lyons. “Are Financial Education Programs Meetings the Needs Financially Disadvantaged Consumers?” Networks Financial of Institute, Indiana State University, 2007-WP-02.

Pre and post evaluations

When to use: Programs that are 2 hours or longer

Advantages: Can compare pre and post responses and document changes in

knowledge, attitudes, and behavior. Can be used to document immediate changes in knowledge, skills

and planned behaviors following the program.

Disadvantages: More time intensive. Identification numbers are needed to match pre and post surveys. May be difficult to show actual behavior change. May be difficult to show that the intervention caused the change. Doesn’t account for other possible reasons for change. Transient populations may lead to low unmatched evaluations.

Follow-ups

When to use:

• Program is comprehensive enough to potentially result in intermediate and long-term impact.

• Must have adequate resources and evaluation capacity.

• Usually administered three to six months after the program.

• Can document changes in actual financial behaviors, ability to achieve financial goals, and overall financial position.

Delivery methods for follow-ups

Face-to-face Mail (paper survey, post cards) Telephone Internet (e-mail, website) Group interviews

Stages to Change (TTM)

When to use: Programs that have multiple sessions

Advantages: Can document intermediate and long-term change. Easier to measure actual behavior change and to control for other

factors that may lead to change over time. Can identify stage at which individual is ready and able to change

behavior. Behaviors can be recorded at the beginning, middle, and end of the

program so that changes in actual behavior can be observed.

Disadvantages: Time and resource intensive. May require additional progress reporting and long-term follow-up. Can only be used with multi-session programs.

Train-the-trainer evaluations

Similar to pre and post evaluation, but more content specific.

Covers subject material in more detail to ensure that trainers have an adequate level of knowledge to teach the program to others.

Can be used to document changes in both the instructors’ teaching skills and personal financial behaviors.

Follow-ups can document how the curriculum materials are being used and identify additional programming needs.

Designing the evaluation instrument:Survey content

General reactions to the session Changes in knowledge Changes in motivation, confidence, and abilities Intended changes in behavior Actual changes in behavior Future programming needs and preferences Demographics Qualitative / open-ended responses

General reactions to the session

Please rate the instructor(s), materials, and the overall program

by checking the box that best applies.

Not Helpful

Somewhat

Helpful

Helpful Very Helpful

Instructor(s)

Educational materials

Overall program

Measuring changes in knowledge

Testing Knowledge

Please circle your answer to each of the following statements.

1. Fixed expenses are expenses that typically change from month to month such as food, clothing, and utilities.

True False Not

Sure

2. Financial experts recommend having an emergency fund

that is equal to 3-6 months worth of living expenses.

True False Not

Sure

3. Goals should only be made for long-term plans such as homeownership, college tuition, or retirement.

True False Not

Sure

4. Gross income is defined as income after taxes and other withholdings have been subtracted from net income.

True False Not

Sure

5. Credit reports can affect an individual’s ability to get a job, purchase a home, and obtain home and auto insurance.

True False Not

Sure

Measuring changes in knowledge (conti.)

Format can be True/False or multiple choice.

True/False is reliable indicator for low literacy audiences and youth.

The more questions you ask, the greater the reliability measure.

May include a “don’t know” option to control for guessing.

Post-test: 10 questions (established standard)

Pre- and post-test: 10-20 questions

Train-the-trainer: 10-25 questions

Changes in motivation, confidence, and abilities

Building Skills/Confidence Indicators

Please check the box that best describes your confidence to do

the following:

Your confidence to: Not Confident

A little confident

Somewhat confident

Confident Very confident


Save money regularly.



Changes in motivation, confidence, and abilities (conti.)

Recording Participants’ Attitudes

Please check the box that best describes how much you agree

with the following statements.

Statement Strongly Disagree

Disagree Undecided Agree Strongly

Agree

Saving money regularly is important to me.

Keeping track of spending is a good habit.

Planning my personal budget is a priority.

Intended changes in behavior

Taking Charge Indicators

Please check the box that best describes your answer.

As a result of this program,

you plan to:

No Maybe Yes Already doing this

Does not apply




Actual changes in behavior

Financial Behavior Indicators

Please indicate how often you are currently doing each of the following

financial practices. There is no “right” or “wrong” answer. (Choose only one)

Financial Practice Never Rarely Sometimes Usually Always

Setting short and long-term financial goals.

Saving money regularly.

Using a spending plan to track income and expenses.

Paying bills on time each month.


For each financial practice, please check the box that best describes

your current behavior.

Financial Practice

I am not considering

this

I am considering

this

I am doing this

sometimes

I am doing this

most of the time

I am doing this all of the time





Using TTM scale (general categories)

Using TTM scale (specific categories)


For each financial practice, please check the box that best describes

your current behavior.

Financial Practice

Do not plan to do

Plan to do in the next

month

Plan to do in the next 2-6 months

Have been doing for 1-6 months

Have been doing for

more than 6 months





Capturing behavior change with follow-ups

Since completing the program, please check the box that best describes

how often you are doing each financial practice. There is no “right” or

“wrong” answer. (Choose only one)

Financial Practice

I am not doing this

I am doing this

sometimes

I am doing this most of

the time

I am doing this all of the

time

Setting personal financial goals.


Keeping track of income and expenses.


Finding ways to decrease expenses.

Financial Progress Indicators

Please indicate how the following numbers have changed for you personally since completing the program.

Capturing behavior change with follow-ups (conti.)

DecreasedStayed

about the same

Increased

Amount of change,

if any (% or $)

Monthly income

Monthly expenses

Amount saved monthly

Current credit card debt

Capturing behavior change with follow-ups (conti.)

Progress Reporting

Please record your financial position based on your current progress in

the program.

Financial Position

At the beginning of the program

In the middle of the program

At the end of the program

How much credit card debt do you owe? ($)

How many credit cards do you have? (#)

What is the highest interest rate on your credit cards? (%)

How much do you have in savings? ($)

A few words about train-the-trainer programs….

Testing knowledge Building teaching skills Shaping personal skills Taking action for teaching Taking action for personal financial success Follow-ups

Qualitative / Open-Ended Questions (common examples)

“Post Evaluation Only” and “Pre and Post Evaluation” What did you like the most about this program? What did you like the least about this program? How could this program be improved? Would you recommend this program to others?

“Stages to Change Evaluation” What has made it easier for you to improve your financial practices? What has prevented you from improving your financial practices? With respect to the overall program, what did you like the most? What did you like the least? How could this program be improved? Have you shared what you learned with others? Would you recommend this program to others?

Qualitative / Open-Ended Questions (conti.)

“Train-the-Trainer Evaluation” What was the most helpful information you received during this

training program? How could this training program be improved? How do you plan to share this information with your target

audience(s)? What information and materials from this training do you plan to

share with your target audience(s)? Will you share what you learned with other instructors and

colleagues? Would you recommend this training program to other instructors

and colleagues?

Demographic Questions

Age Gender Race, Ethnicity, and Language Marital Status Education Employment Family Structure Health Status Income, Assets, and Debts Region/Location

Financial Experience Students/Youth Instructors/Educators

Common types of survey questions

Yes/No questions True/False Agree/Disagree Multiple choice One best answer Multiple answers Rating and ranking questions Qualitative / open-ended questions

Choosing measurement scales and scoring

Example:

Resource:“Collecting Evaluation Data: End-of-Session Questionnaires.” University of Wisconsin-Extension, p. 62-64. www.uwex.edu/ces/pdande/evaluation/evaldocs.html

Strongly Disagree

Disagree Neutral Agree Strongly

Agree

Other helpful tips on survey design

Think carefully about how to write the questions given your target audience. Use plain language.

Make the evaluation form easy to complete (i.e., white space and font).

Include simple instructions.

Start with non-threatening questions.

Keep the evaluation as short as possible.

Cluster similar items to save time and space.

Protect the participant’s identity.

Measurement error and validity of indicators. Financial knowledge Confidence level Financial behavior

EXAMPLE: Which of the following are valid indicators of behavior change? Participant opened a bank account. Participant increased savings. Participant avoided bankruptcy. Participant did not default on mortgage payments.

A few words of caution when selecting indicators….

Other measurement issues

Self-reports are subject to bias. Social desirability Norms and “rules of thumb” Misperceptions and over-optimism Memory distortion and recall bias

Samples may not be representative. Non-response bias Program attrition Self-selection Low response rates (e.g., follow-ups)

Environmental factors may affect outcomes. Unexpected life events Program incentives (e.g., rewards, special benefits,

enrollment programs) Individualized financial advice or “coaching”

Psychological factors. Inherent motivation Ability Attitudes

Longitudinal data?Control groups?

Randomized experiments?

Solutions for measurement issues

There are numerous behavior indicators.Here are some examples….

Increases in savings. Decreases in debt. Maintaining a regular budget. Comparison shopping. Increases in new accounts opened. Improved credit scores. Improved communication with spouse/partner/parents

about finances.

Other common indicators?

How do these indicators change for various

target populations?

Youth? Underserved?

Adults?Members of your

organization?

Part III

Putting It All Together! Sample Evaluations

ACTIVITY: Selecting Your Evaluation Methods

(Evaluation Action Plan – Part B)

Post-test only

Retrospective pre-test

Pre and post-test

Follow-up survey

Stages-to-change

Focus groups

Interviews

Case studies

Observations

Stories/anecdotal evidence

Tests of ability

Other

Think about your signature program, what is the most appropriate evaluation method?

What types of questions will the evaluation seek to answer?

I would really like to know….

1.

2.

3.

Changes in satisfaction levels Changes in knowledge Changes in skills and confidence levels Changes in attitudes Changes in aspirations Anticipated or intended changes in behavior Actual changes in behavior Socio-economic changes Other

What types of indicators will you useto document this impact?

List some specific indicators for knowledge.

1.

2.

3.

4.

List some specific indicators for skills and confidence.

1.

2.

3.

4.

List some specific indicators for anticipated and actual changes in behavior.

1.

2.

3.

4.

Useful references for evaluation design

NEFE® Financial Education Evaluation Toolkithttp://www2.nefe.org/eval/index.php

“Collecting Evaluation Data: End-of-Session Questionnaires.” University of Wisconsin-Extension. www.uwex.edu/ces/pdande/evaluation/evaldocs.html

“A Step-by-Step Guide to Developing Effective Questionnaires and Survey Procedures for Program Evaluation & Research.”Rutgers Cooperative Research & Extension, #FS995. www.rcre.rutgers.edu/pubs/publication.asp?pid=FS995

Part VINEFE Financial Education

Evaluation Toolkit ®

http://www2.nefe.org/eval/intro.html

NEFE Financial Education Evaluation Toolkit®

Database• Post evaluation only with option for follow-up• Pre and post evaluation with option for follow-up• Stages to Change Evaluation• Train-the-Trainer

• Testing Knowledge• Building Skills• Taking Charge

Manual• How-to-guide for grass-roots level organizations• Examples (survey instruments, executive summary, reports)• Guidance on how to organize and present impact data

Manualhttp://www2.nefe.org/eval/manual.html

Part I:Financial Education Overview

Part II:Understanding Program Evaluation

Part III:The Evaluation Planning Process

Part IV:Using the Evaluation Database

Part V:Reporting Program Impact

Appendix:Sample Evaluation Instruments

Databasehttp://www2.nefe.org/eval/index.php

Step 1:Program Info and Follow-up

Step 2:Knowledge Questions

Step 2a:Selecting Questions

Step 2b:Customizing Questions

Step 3:Confidence and Behavior Indicators

Step 4a:Recommendations

Step 4b:Selecting Statements

Step 4c:Customizing Statements

Step 5:Qualitative Data

Step 6:Demographics

Step 7:Follow-Up: Financial Progress Indicators

Step 8:Follow-Up: Personal Achievements

Step 9:Follow-Up: Demographics

Step 10:Finalizing Evaluation

Part V

Implementing Your Evaluation: Putting Your “Tools” into Action

1. Identifying the “ideal” approach to evaluation.

Evaluation methods and measures vary widely across programs and academic disciplines.

Wide variation in financial outcomes across programs.

Significant differences in financial needs across consumers.

Some participants unable to implement certain financial behaviors.

5 Biggest Evaluation Challenges

2. Defining “program success.”

Setting realistic expectations for program participants.

Choosing appropriate outcomes and indicators based on participants’ financial situation or other external constraints.

Identifying participants’ individual financial needs and applying appropriate educational interventions.

Finding “the teachable moment.”

5 Biggest Challenges (conti.)

“What is driving this financial education movement? Why is it so important? What are we ultimately trying to address? Is it reducing the poverty gap in this country? Between those that have and those that don’t have. And it’s widening. And those at the bottom end of the spectrum….what we’re asking them is to build wealth. And at the same time, what we’re asking people in this country who make $20,000 or less is: ‘Absent us raising your wages in this country, we’re asking you to build wealth, to participate in IDA programs. We’re asking you to save with the little amount of money you’re making. We’re asking you to reduce your debt burden, learn how to manage your money, and clean up your credit history with the little amount of money you’re working with. And we want you to get from point A to point B with all those constraints.”

Source: Lyons, A. C., Palmer, L., Jayaratne, K.S.U., and Scherpf, E. (2006). "Are We Making the Grade? A National Overview of Financial Education and Program Evaluation.” The Journal of Consumer Affairs, 40(2), 208-235.


3. Collecting impact data from program participants.

Little incentive to complete evaluations (like “pulling teeth”).

Reluctance to divulge personal information (surveys “too personal”; lack of trust).

High drop out rates, low response rates, and difficult to track.

Literacy levels (i.e., ESL, reading level).

Collecting sensitive data and information.

Tradeoff between participation and evaluation rigor.


4. Designing and implementing effective program evaluations.

Evaluation process is cumbersome.

Lack of time, staff, and financial resources.

The “PUSH” for increased rigor and “the rush to the finish line.”

Rigor vs. Reality (e.g., measurement issues)

The limitations of “one-shot” evaluations.(pre- and post-tests; intended vs. actual behavior change)

The reality of conducting longitudinal studies with control groups. (follow-ups and tracking of program participants)


5. Conducting more rigorous, theory-based evaluation research.

We need to back up and spend more time trying to understand financial behavior and why people do what they do.

Until then, financial education will only serve as a “band-aid” rather than a long-term solution, and we will continue to struggle with how to define financial success.


Increase rigor by planning more strategically.

Focus on signature programs and on multi-session programs.

Partner and pool resources.

“We’re jumping into evaluating everything, instead of…taking a couple of projected outcomes or a subset of all that we work with and trying to do evaluations with those.”

Simple Steps to Overcoming the Barriers

Identify available resources – financial and non-financial.

Understand funders’ needs and how they fit into your evaluation plan.

Take into consideration program delivery methods.

Overcoming the Barriers (conti.)

Establish a consistent and workable set of standards for measuring program impact.

Create evaluation tools that are flexible to account for the wide range in programs (i.e., one-stop shop with survey instruments, best practices, online training workshops, etc.)

Reality of program evaluation at all levels (disconnect; need better awareness of resource constraints; continued recognition of traditional evaluation methods).

Overcoming the Barriers (conti.)

If resources were not a constraint,

what would your ideal program evaluation

look like?

You first need to ask….

What challenges do you face in trying to implement

your “ideal” evaluation?

Then, the reality check….

How can you overcome these challenges?

What financial and non-financial resources are available (e.g., time, money, staff, expertise)?

Are there others who can help (e.g., partners, stakeholders, funders, volunteers)?

What financial and non-financial resources do they have available?

Given constraints, what can you realistically do?

Thinking outside of the box….

ACTIVITY: Overcoming Your Challenges(Evaluation Action Plan - Part C)

Part VI

Building Program Success:Reporting Program Impact

The common fear of evaluation

It will show what we’re doing wrong!

Learning from the successes and the failures.

Putting it all together

Look for themes.

Work with what you’ve got.

Learn as you go and be flexible.

“Tell the story,” which can be the most powerful depiction of the benefits and services of your program.

Use the findings to improve your program.

Tips for “telling your story”

Know your audience.

Use simple descriptive statistics (i.e., counts, percentages, and averages) when analyzing and interpreting data.

Don’t use jargon. Be straightforward and clearly state major findings.

Use language that is suggestive rather than decisive (i.e., “the data suggest” rather than “the data show”). Be careful not to overstate your findings.

Blend the presentation with quantitative and qualitative data.

Do not generalize the findings to the entire group. Report the results in terms of the “program participants” rather than “all U.S. families” or “all New York residents.”

Clearly describe who the results represent. Provide information and demographics on the sample of program participants.

Be honest about your program’s strengths and weaknesses, while highlighting the positive.

Writing Impact Statements - Examples

Statements that reflect intentions:

As a result of participating in this financial education program, [X] percent reported that they….

plan to do/use/adoptare more knowledgeableare more confident in their ability to doare more likely to do/use/adoptwill do/use/adopt

….a particular attitude, piece of information, or behavior.

Statements that reflect actual actions:

As a result of participating in this financial education program, [X] percent reported that they….

are now doingdidusedincreased knowledge ofadopted

….a particular attitude, piece of information, or behavior.

Analyzing the findings

How will you use the findings for program improvement and internal reporting?

How will the evaluation findings be communicated and shared with others?

Disseminating the findings

Written reports Short summary statements Media releases Internet postings Graphs and visuals Presentations Displays, posters, etc.

How will you analyze the data?

And, how will you use the findings?

ACTIVITY: Analyzing and Reporting Your Findings(Evaluation Action Plan - Part D)

What do you hope to learn from the findings?What are the potential impacts?

As a result of participating in this program….

How will you disseminate the findings?

Who will you share the findings with?

How? With Who?

Useful references for reporting impact

“Collecting Evaluation Data: Surveys.”University of Wisconsin-Extension. www.uwex.edu/ces/pdande/evaluation/evaldocs.html

“Taking Stock: A Practical Guide to Evaluating Your OwnPrograms.” Horizon Research, Inc.www.horizon-research.com/reports/1997/stock.pdf

Tipsheets #66, #80, #81. Penn State Cooperative Extension.www.extension.psu.edu/evaluation/titles.html

Where do we go from here?Online resources at your fingertips

University of Wisconsin-Extensionhttp://www.uwex.edu/ces/pdande/evaluation/index.html

Cornell University Extensionhttp://staff.cce.cornell.edu/administration/program/evaluation/evalrefs.htm

Penn State Extensionhttp://www.extension.psu.edu/evaluation/

Reading List

Lyons, A. C., Palmer, L., Jayaratne, K.S.U., and Scherpf, E. (2006). "Are We Making the Grade? A National Overview of Financial Education and Program Evaluation.” The Journal of Consumer Affairs, 40(2), 208-235.

Lyons, A. C. (2005). “Financial Education and Program Evaluation: The Challenges and Potentials for Financial Professionals.” Journal of Personal Finance, 4(4), 56-68.

US Government Accountability Office. (2004). The Federal Government’s Role in Improving Financial Literacy, #GAO-05-93SP.

Financial Literacy & Education Commission. (2006). Taking Ownership of the Future: The National Strategy for Financial Literacy. www.mymoney.gov

Contact Information

Dr. Angela Lyons

Associate Professor

Department of Agricultural and Consumer Economics

University of Illinois

Phone: 217-244-2612

E-mail: [email protected]

Questions

some practical tips for measuring financial success dr. angela lyons university of illinois...

Documents

good evaluation

program success slide

evaluation process

program evaluation toolkit

financial success

nefe financial education

current evaluation efforts

building program success