introduction to utilization focused evaluation · 4.evaluation is part of initial...
TRANSCRIPT
![Page 1: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/1.jpg)
INTRODUCTION TOUTILIZATION FOCUSED
EVALUATION
DREAMIT and ISIF
APRIL , 2010
Facilitator:
Dr. Sonal Zaveri,
IDRC Consultant
![Page 2: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/2.jpg)
Agenda
1. Introduction to UFE Curriculum &DECI
2. UFE – What is it? How is it different? Why?
3. First steps – practical example
4. Stakeholder identification
5. Roles definition
6. Readiness for UFE?
Recess
![Page 3: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/3.jpg)
UFE Curriculum
![Page 4: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/4.jpg)
DECI PROJECTSDeveloping evaluation capacity for ICT4D
![Page 5: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/5.jpg)
Utilization focused evaluation is…
A decision-making framework forenhancing the utility and actualuse of evaluations. (Patton, 2008a: slide 9)
![Page 6: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/6.jpg)
Evaluation Use – Why is it important?
Many evaluations land on
dusty bookshelves
BUT…..
• Monies have been spent $$$$$
• Do not know what is effective or not
SO …
How do we do evaluations that can beused?
![Page 7: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/7.jpg)
Reflect
What has been yourexperience?
![Page 8: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/8.jpg)
What is evaluation?What do we want to know?
• What is the worth, value, significance?
• How effective was program?
• Was it implemented as planned?
• Were goals achieved?
• Did participants benefit?
• Any unexpected consequences ?
• What worked? How to improve?
• Do next?
![Page 9: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/9.jpg)
3 Questions for evaluation
WHAT?
SO WHAT?
NOW WHAT?
What to know?
What to act on?
How to use findings?
Challenge is not getting information, it is USING it! Whatshould I know and keep? What should I let go? Example
![Page 10: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/10.jpg)
Utilization focused evaluation is…
A decision-making framework forenhancing the utility and actualuse of evaluations. (Patton, 2008a: slide 9)
![Page 11: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/11.jpg)
Premises of UFE
1. Evaluations should be JUDGED by their utility and ACTUAL USE.
2. No evaluation should go forward unless and until there are primaryintended users who will use the information that can be produced.
3. Primary intended users are involved in the process - commitment.
4. Evaluation is part of initial program/evaluation design. The primaryintended users want information to help answer a question or questions.
5. Evaluator’s role is to help intended users clarify their purpose andobjectives.
6. Implications for use are part of every decision throughout the evaluation– it is the driving force of the evaluation process. (Patton, 2008a)
![Page 12: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/12.jpg)
Personal LEARNING among them.
More CONFIDENCE and direct APPLICATION ofevaluation findings to program practices.
A reduced POWER DIFFERENTIAL betweenevaluators and program practitioners.
More NEGOTIATED DECISION MAKING and learning.
Engaging users contributes to
![Page 13: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/13.jpg)
A PROCESS for helping primary intended users select themost appropriate content, model, methods, theory, and usesfor their particular situation. Use any evaluation content,model, method or theory.
Evaluation done for and with specific, intended primaryUSERS for specific, intended USES.
“USES” refers to the way real people in the real worldAPPLY evaluation findings and experience the evaluationPROCESS. (Patton, 2008: 37 - Ch.2)
Utilization focused evaluation is…
![Page 14: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/14.jpg)
REFLECT
• How have you done evaluations inthe past – were they like this?
• If you had thought like this – do youthink it would have made adifference? HOW?
![Page 15: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/15.jpg)
Comments / Questions
![Page 16: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/16.jpg)
Group Work
In my project, I would very much like toknow ……………
Individually, think of 5 questions you want toknow
In a group discuss and agree to 5 to 10questions
Write on a flipchart
![Page 17: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/17.jpg)
BREAK
![Page 18: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/18.jpg)
¿Who are the stakeholders that areor need to be involved in this
project?
![Page 19: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/19.jpg)
ROLES (1 of 3)
EVALUATOR: Person or organization responsible forfacilitating / leading the design, the implementationand the utilization of the evaluation. Tasks:organizational development agent, educator,coach/mentor, strategic planner, etc.
![Page 20: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/20.jpg)
ROLES (2 of 3)
PRIMARY USER: People who will use and apply thefindings of the evaluation. Patton (2008)suggests the following profiler:
1. Interested.2. Knowledgeable.3. Open minded.4. Represents an important interest group.5. Has credibility.6. Teachable.7. Available for ongoing interaction throughout the evaluation process.
![Page 21: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/21.jpg)
ROLES (3 of 3)
AUDIENCE INTERESTED IN THE REPORT: Actorsinterested in the unfolding and findings of theevaluation.
![Page 22: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/22.jpg)
¿From the stakeholders thatwere identified, who plays
what role?
![Page 23: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/23.jpg)
Identified stakeholders (from discussion)
Group Stakeholder Assigned role
Stakeholder Group 1 Stakeholder 1
Stakeholder 2
Stakeholder 3
Stakeholder Group 2 Stakeholder 4
Stakeholder 5
Stakeholder 6
Stakeholder Group 3 Stakeholder 7 Primary
intended user
Stakeholder 8
![Page 24: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/24.jpg)
¿Who is missing?
![Page 25: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/25.jpg)
We don’t want to end up in this situation,
so…
![Page 26: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/26.jpg)
• Evaluators make themselves the primary
decision makers.
• Identify vague, passive audiences as users.
• Targeting organizations as users.
• Focusing on decisions instead of decision
makers.
• Assuming the evaluation’s funder is theprimary stakeholder.
UFE traps or temptations (1 of 2)
![Page 27: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/27.jpg)
• Waiting until the findings are in to identify intended
users and intended uses.
• Taking a stance of standing above the messiness of
people and politics.
• Being co-opted by powerful stakeholders.
• Identifying primary intended users but not involving
them meaningfully.
(Patton, 2008, adapted from p. 90 - Ch.3)
UFE traps or temptations (2 of 2)
![Page 28: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/28.jpg)
¿How well prepared do wefeel we are for adopting
UFE as the methodology toevaluate this project?
![Page 29: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/29.jpg)
What resources do we require in order toimplement a UFE plan ?
Are all involved parties supportive ofadopting a UFE approach?
What could the main challenges of thisevaluation initiative be?
![Page 30: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/30.jpg)
1. Project / network readiness assessment.
2. Evaluator readiness and capability assessment.
3. Identification of primary intended users.
4. Situational analysis.
5. Identification of primary intended uses.
6. Focusing on evaluation.
7. Evaluation design.
8. Simulation of use.
9. Data collection.
10. Data analysis.
11. Facilitate use.
12. Meta-evaluation.
UFE in 12 steps
![Page 31: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/31.jpg)
Conclusions and next steps
![Page 32: INTRODUCTION TO UTILIZATION FOCUSED EVALUATION · 4.Evaluation is part of initial program/evaluation design. The primary intended users want information to help answer a question](https://reader034.vdocuments.net/reader034/viewer/2022042606/5f94aa25375cea0122201f62/html5/thumbnails/32.jpg)
BREAK