citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11
TRANSCRIPT
![Page 1: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/1.jpg)
Measuring Financial Education Success: Evaluation Methods and
Assessment Tools
Barbara O’Neill, Ph.D, CFP®, CRPC®Professor II, Rutgers University
![Page 2: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/2.jpg)
Are You Evaluating Your Financial Education Programs?
If You Are, How?
![Page 3: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/3.jpg)
The Million Dollar “So What?” Question…
At the end of the day…did your financial education program make a difference?
How do you know?
![Page 4: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/4.jpg)
The Current State of Financial Education Program Evaluation
• Current evaluation efforts are still far from satisfactory
• General lack of evaluation capacity
• Evaluation is often treated as an after-thought
• Outcomes (e.g., measures of changed behavior) are often confused with program outputs (e.g., number of participants)
![Page 5: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/5.jpg)
• What gets measured gets funded
• With evaluation results, you can
– assess impact of programs on learners
– see if you accomplished what you planned
– know if a program was “worth it”
– celebrate success and learn from failure
– make informed decisions to improve, hold, or fold programs
– promote your program and win public support
We are in an “Accountability Era”
![Page 6: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/6.jpg)
• See http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html• Begins with the end in mind• Explains what a program is and what it will accomplish• Shows relationships between inputs, outputs, and outcomes
Introducing the Logic Model
![Page 7: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/7.jpg)
HUNGRY
Feel betterFeel betterGet foodGet food Eat food Eat food
Everyday Situation
![Page 8: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/8.jpg)
Building a Strong Financial Education Program
INPUTS OUTPUTS OUTCOMES
Program investments
Activities Participation Short Medium
What We
Invest
What We Do
Who We Reach
What Results
SO WHAT??
What is the VALUE?
Long- term
![Page 9: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/9.jpg)
Staff
Money
Time
Volunteers
Partners
Equipment/Technology
Policies
Research
INPUTS
![Page 10: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/10.jpg)
OUTPUTSWhat We Do Who We Reach
ACTIVITIESAssess needs and assets
Design curriculum
Educate students
Conduct workshops
Facilitate learning groups
Sponsor conferences
Work with the media
Partner – collaborate
PARTICIPATIONParticipants
Clients
Customers
Users
Groups
Reactions - Satisfaction
![Page 11: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/11.jpg)
OUTCOMESWhat Results for Individuals, Organizations, Communities..…
SHORTLearning
Awareness
Knowledge
Attitudes
Skills
Opinion
Aspirations
Motivation
MEDIUMAction
Behavior
Practice
Decisions
Policies
Social action
LONG-TERMConditions
Human
Economic
Civic
Environment
![Page 12: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/12.jpg)
OUTCOMES
Commonly Measured Items That Are Not Outcomes
• Participant satisfaction
• Number of people taught
• Units of education completed
• Number of events held
• Time and money spent
• Level of effort
![Page 13: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/13.jpg)
Impact Evaluation Data Collection Methods
• Surveys
– Post-evaluation only (short programs)
– Pre- and post-evaluation
– Follow-up (e.g., 3 months later)
• Focus groups
• Interviews
• Observations
• Tests of knowledge/ability
• RARE: Control groups and longitudinal studies
![Page 14: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/14.jpg)
Typical Survey Questions
• General reactions to the program
• Changes in knowledge
• Changes in motivation, confidence, and abilities
• Intended changes in behavior
• Actual changes in behavior
• Future programming needs and preferences
• Demographics
• Qualitative/open-ended responses
![Page 15: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/15.jpg)
Post-Then-Pre Evaluation Surveys• Also known as a “Retrospective Pre-Test”• Helps identify changes in knowledge, attitudes, and
behavior
![Page 16: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/16.jpg)
Another Post-Then-Pre Example
Big Advantage: Avoids the problem of learners under- estimating what they don’t know before a program
![Page 18: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/18.jpg)
Downloadable Evaluation Manual
![Page 19: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/19.jpg)
Build Your Own Evaluations Online
![Page 20: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/20.jpg)
One Last Step: Share Your Program Evaluation Results
WHAT?
SO WHAT?
NOW WHAT?
![Page 21: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/21.jpg)
Tips for “Telling Your Story”
• Use simple descriptive statistics (e.g., counts, percentages, and averages)
• Don’t use jargon
• Don’t overstate your results (e.g., causality)
• Blend quantitative and qualitative data
• Clearly describe who the results represent (i.e., demographic characteristics of participants)
• Be honest about your program’s strengths and weaknesses, while highlighting the positive
![Page 22: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/22.jpg)
Impact Statements: Intentions
As a result of participating in this financial education program, X% of participants reported that they…
• plan to do/use/adopt…• are more knowledgeable about…• are more confident in their ability to…• are more likely than before to do/use/adopt…• will do/use/adopt…
…a particular attitude, piece of information, or behavior.
![Page 23: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/23.jpg)
Impact Statements: Actions
As a result of participating in this financial education program, X% of participants reported that they…
• are now doing…• did…• used…• increased their knowledge of…• adopted…• changed…
… a particular attitude, piece of information, or behavior.
![Page 24: Citi grant teacher training-fin ed evaluation-assessment tools-o'neill-06-11](https://reader033.vdocuments.net/reader033/viewer/2022052823/5550cd7ab4c905e8318b4a42/html5/thumbnails/24.jpg)
Questions and Comments?
Barbara O'Neill, Ph.D., CFP®, CRPC
Extension Specialist in Financial Resource Management and Professor II Rutgers University
Phone: 732-932-9155 Extension 250
E-mail: [email protected]
Internet: http://njaes.rutgers.edu/money2000/
Twitter: http://twitter.com/moneytalk1