Some Practical Tips for Measuring Financial Success

Dr. Angela LyonsUniversity of Illinois

Program Evaluation II: Creating Your Evaluation Toolkit

Presented byDr. Angela Lyons

University of Illinois

October 2009

Evaluation methods and measures vary widely across programs and academic disciplines.

Wide variation in financial outcomes across programs.

Significant differences in financial needs across consumers.

Some participants unable to implement certain financial behaviors.

Identifying the “ideal” approach to program evaluation

Then you need to decide….

“To evaluate or not to evaluate, that is the question?”“To evaluate or not to

evaluate, that is the question?”

Questions to ask yourself …. At the end of the day, what are you trying to show? What is the purpose of the evaluation? Who will use the information – and how? What information do you want to collect? Who is your target audience? What is your primary delivery method? What are your available resources (i.e., time, money,

and staff)? What is your timeline? What is your expertise and evaluation capacity? Who are your partners, funders, and stakeholders?

Common survey methods used to collect impact data

• Post evaluation only • Retrospective pre-test (RPT)• Pre and post evaluation• Follow-up• Stages to Change (TTM)• Control groups and longitudinal studies

Key question to ask: What is the length of your program?

Post evaluation only

When to use: Short programs that are less than 2 hours

Advantages: Only need to survey group once. Good for limited-resource audiences and groups that are

transient. Relatively inexpensive and less time intensive. Can document participants’ levels of knowledge, skills, and

planned behaviors at the end of the program.

Disadvantages: With no pre assessment, it’s difficult to document potential and

actual changes in knowledge, attitudes, and behavior.

Retrospective pre-tests (RPTs); The Post-Then-Pre Evaluation

Retrospective pre-test (RPTs)

When to use: Any program, but typically 2 hours or less

Advantages: Only need to survey group once. Good for limited-resource audiences and groups that are

transient. Controls for “response shift bias.” Can document “relative” change.

Disadvantages: Potential for respondent bias (social desirability factor). Self-assessment measures are subjective.

7

Pre and post evaluations

When to use: Programs that are 2 hours or longer

Advantages: Can compare pre and post responses and document changes in

knowledge, attitudes, and behavior. Can be used to document immediate changes in knowledge, skills

and planned behaviors following the program.

Disadvantages: More time intensive. Identification numbers are needed to match pre and post surveys. May be difficult to show actual behavior change. May be difficult to show that the intervention caused the change. Doesn’t account for other possible reasons for change. Transient populations may lead to low unmatched evaluations.

Follow-ups

When to use:

• Program is comprehensive enough to potentially result in intermediate and long-term impact.

• Must have adequate resources and evaluation capacity.

• Usually administered three to six months after the program.

• Can document changes in actual financial behaviors, ability to achieve financial goals, and overall financial position.

Stages to Change (TTM)

When to use: Programs that have multiple sessions

Advantages: Can document intermediate and long-term change. Easier to measure actual behavior change and to control for other

factors that may lead to change over time. Can identify stage at which individual is ready and able to change

behavior. Behaviors can be recorded at the beginning, middle, and end of the

program so that changes in actual behavior can be observed.

Disadvantages: Time and resource intensive. May require additional progress reporting and long-term follow-up. Can only be used with multi-session programs.

Train-the-trainer evaluations Similar to pre and post evaluation,

but more content specific.

Covers subject material in more detail to ensure that trainers have an adequate level of knowledge to teach the program to others.

Can be used to document changes in both the instructors’ teaching skills and personal financial behaviors.

Follow-ups can document how the curriculum materials are being used and identify additional programming needs.

Designing the evaluation instrument:Key survey content

1. General reactions to the session2. Changes in knowledge3. Changes in motivation, confidence, and abilities4. Intended changes in behavior5. Actual changes in behavior6. Future programming needs and preferences7. Demographics8. Qualitative / open-ended responses

General reactions to the session

Please rate the instructor(s), materials, and the overall program by checking the box that best applies.

Not Helpful

SomewhatHelpful

Helpful Very Helpful

Instructor(s)

Educational materials

Overall program

Measuring changes in knowledge

Testing Knowledge Please circle your answer to each of the following statements.

1. Fixed expenses are expenses that typically change from month to month such as food, clothing, and utilities.

True False Not Sure

2. Financial experts recommend having an emergency fund that is equal to 3-6 months worth of living expenses.

True False Not Sure

3. Goals should only be made for long-term plans such as homeownership, college tuition, or retirement.

True False Not Sure

4. Gross income is defined as income after taxes and other withholdings have been subtracted from net income.

True False Not Sure

5. Credit reports can affect an individual’s ability to get a job, purchase a home, and obtain home and auto insurance.

True False Not Sure

Measuring changes in knowledge (conti.)

Format can be True/False or multiple choice.

True/False is reliable indicator for low literacy audiences and youth.

The more questions you ask, the greater the reliability measure.

May include a “don’t know” option to control for guessing.

Post-test: 10 questions (established standard)

Pre- and post-test: 10-20 questions

Train-the-trainer: 10-25 questions

Changes in motivation, confidence, and abilities

Building Skills/Confidence Indicators Please check the box that best describes your confidence to do the following:

Your confidence to: Not Confident

A little confident

Somewhat confident

Confident Very confident

Set short and long-term financial goals.

Save money regularly.

Use a spending plan to track income and expenses.

Pay bills on time each month.

Changes in motivation, confidence, and abilities (conti.)

Recording Participants’ Attitudes Please check the box that best describes how much you agree with the following statements.

Statement Strongly Disagree

Disagree Undecided Agree StronglyAgree

Saving money regularly is important to me.

Keeping track of spending is a good habit.

Planning my personal budget is a priority.

Intended changes in behavior

Please indicate how often you plan to do each of the followingfinancial practices. There is no “right” or “wrong” answer. (Choose only one)

As a result of this program, you plan to:

No Maybe Yes Already doing this

Does not apply

Set short and long-term financial goals.

Use a spending plan to track income and expenses.

Pay bills on time each month.

Financial Practice Never Rarely Sometimes Usually Always

Set short and long-term financial goals.

Save money regularly.

Use a spending plan to track income and expenses.

Pay bills on time each month.

Actual changes in behavior

Please indicate how often you are currently doing each of the following financial practices. There is no “right” or “wrong” answer. (Choose only one)

Financial Practice Never Rarely Sometimes Usually Always

Setting short and long-term financial goals.

Saving money regularly.

Using a spending plan to track income and expenses.

Paying bills on time each month.

Financial Progress IndicatorsPlease check the box that best describes how your financial position has changed since completing the program. Then, indicate specifically how your financial position has changed.

Capturing behavior change in follow-ups and “stages”

DecreasedStayed

about the same

Increased

Amount of change,

if any (% or $)

Monthly income

Monthly expenses

Amount saved monthly

Current credit card debt

Capturing behavior change in follow-ups and “stages”

Progress ReportingPlease record your financial position based on your current progress in the program.

Financial PositionAt the

beginning of the program

In the middle of the program

At the end of the program

How much credit card debt do you owe? ($)

How many credit cards do you have? (#)

What is the highest interest rate on your credit cards? (%)

How much do you have in savings? ($)

A few words about train-the-trainer programs….

Testing knowledge Building teaching skills Shaping personal skills Taking action for teaching Taking action for personal financial success Follow-ups

Qualitative / Open-Ended Questions (common examples)

“Post Evaluation Only” and “Pre and Post Evaluation” What did you like the most about this program? What did you like the least about this program? How could this program be improved? Would you recommend this program to others?

“Stages to Change Evaluation” What has made it easier for you to improve your financial practices? What has prevented you from improving your financial practices? With respect to the overall program, what did you like the most? What did you like the least? How could this program be improved? Have you shared what you learned with others? Would you recommend this program to others?

Qualitative / Open-Ended Questions (conti.)

“Train-the-Trainer Evaluation” What was the most helpful information you received during this

training program? How could this training program be improved? How do you plan to share this information with your target

audience(s)? What information and materials from this training do you plan to

share with your target audience(s)? Will you share what you learned with other instructors and

colleagues? Would you recommend this training program to other instructors

and colleagues?

Demographic Questions

Age Gender Race, Ethnicity, and Language Marital Status Education Employment Family Structure Health Status Income, Assets, and Debts Region/Location

Financial Experience Students/Youth Instructors/Educators

Longitudinal data?Control groups?

Randomized experiments?

Rigor vs. Reality

NEFE Financial Education Evaluation Toolkit ®

http://www2.nefe.org/eval/intro.html

NEFE Financial Education Evaluation Toolkit®

Database• Post evaluation only with option for follow-up• Pre and post evaluation with option for follow-up• Stages to Change Evaluation• Train-the-Trainer

• Testing Knowledge• Building Skills• Taking Charge

Manual• How-to-guide for grass-roots level organizations• Examples (survey instruments, executive summary, reports)• Guidance on how to organize and present impact data

Manualhttp://www2.nefe.org/eval/manual.html

Manualhttp://www2.nefe.org/eval/manual.html

Part I: Financial Education OverviewPart II: Understanding Program EvaluationPart III: The Evaluation Planning ProcessPart IV: Using the Evaluation Database

Part V: Reporting Program Impact

Part I:Financial Education Overview

Part II:Understanding Program Evaluation

Part III:The Evaluation Planning Process

Part IV:Using the Evaluation Database

Part V:Reporting Program Impact

Appendix:Sample Evaluation Instruments

Databasehttp://www2.nefe.org/eval/index.php

Step 1:Program Info and Follow-up

Step 2a:Knowledge Indicators

Step 2b:Customizing Questions

Step 3:Confidence and Behavior Indicators

Step 4a:Selecting Statements

Step 4b:Customizing Statements

Step 5:Qualitative Info

Step 6:Demographics

Step 7:Follow-Up: Financial Progress Indicators

Step 8:Follow-Up: Personal Achievements

Step 9:Follow-Up: Demographics

Step 10:Finalizing Evaluation

ACTIVITY: Putting It All Together

(Evaluation Action Plan – Part B)

Post-test only Retrospective pre-test Pre and post-test Follow-up survey Stages-to-change Focus groups

Interviews Case studies Observations Stories/anecdotal evidence Tests of ability Other

Think about your signature program, what is the most appropriate evaluation method?

What types of questions will your evaluation seek to answer?

I would really like to know….

1.

2.

3.

Changes in satisfaction levels Changes in knowledge Changes in skills and confidence levels Changes in attitudes Changes in aspirations Anticipated or intended changes in behavior Actual changes in behavior Socio-economic changes Other

What types of indicators will you useto document this impact?

List some specific indicators.

1.

2.

3.

4.

U of I Center for Economic and Financial Educationhttp://www.cefe.illinois.edu/

Educational Tools for Evaluationhttp://www.cefe.illinois.edu/tools/evaluation.html

Checklist of evaluation resources

Program Planning GuideEvaluation Action PlanEvaluation Road MapEvaluation Reading ListSample EvaluationsNEFE® Financial Education Evaluation Toolkit

http://www.cefe.illinois.edu/tools/evaluation.html

University of Wisconsin-Extensionhttp://www.uwex.edu/ces/pdande/evaluation/index.html

Cornell University Extensionhttp://staff.cce.cornell.edu/administration/program/evaluation/evalrefs.htm

Penn State Extensionhttp://www.extension.psu.edu/evaluation/

Contact Information

Dr. Angela Lyons Associate ProfessorDirector, U of I Center for Economic and Financial EducationUniversity of Illinois

Phone: 217-244-2612 E-mail: anglyons@illinois.edu

Questions