day 1: introduction 4 facilitators 5 expectations for the workshop
TRANSCRIPT
Day 1:
IntroductionIntroduction
4
Facilitators
5
Expectations for the Workshop
6
Workshop Objective
• To engage all participants in a lively learning process on outcome mapping so that you have the knowledge and skills to use it in ways that are appropriate to your specific context.
• The workshop is broken down into 3 modules
7
Module 1. Group Facilitation
• 1 Day
• Learning and practising group facilitation skills helpful in the practice of outcome mapping.
8
Module 2. Conceptual
• 3 Days:
• Learning about the tools and methods of outcome mapping.
9
Module 3. Practicum
• 2 days
• Reflecting, planning, and practicing how to take outcome mapping forward in your specific context.
10
Workshop Process
• Start & end on time
• Give everyone a chance to speak & ask questions
• « Issues and Concerns»
• Questions & comments at any time
• Questions et commentaires en français
11
Agenda for Day 1
12
Introductory Exercise
At your table, introduce yourself by describing 3 events that contributed to you being at this workshop.
Talk about one event long ago, one more recent, and one very recent event
Overview (Sunset Image)
14
Facilitation: Core Values
• Participation
• Capacity-building
• Diversity
• Equity
• Transparency
• Content-neutral
15
Role of Facilitator
Manages process for the group Advocates fair & open procedures Encourages full participation Responds to group dynamics Promotes clarity & understanding Models respect and inclusion
Active Listening Skill 1:
Paraphrasing & Drawing People Out
17
What is Paraphrasing?
• Using your own words to explain what the speaker in saying.
• In order to indicate you’re trying to understand the speaker, start with:– What I’m hearing you say is…– Let me see if I’m understanding what you’re
saying…– It sounds like what you’re saying is…– Am I correct that what you are saying is…
18
What is Drawing People Out?
• Asking open-ended questions to the speaker to get more detail from them
• Probing with questions like:– Can you say more about that?– What do you mean by?– In what way?
• Best used with paraphrasing
Active Listening Skill 2:
Stacking & Tracking
Conclusion & Evaluation of Day 1
Day 2:
Review of Day 1 and Evaluation Feedback
23
Agenda for Day 2
Conceptual Conceptual OverviewOverview
25
Evaluation Challenges
1. Establishing cause & effect in an open system
2. Timing
3. Encouraging iterative learning
4. Clarifying values
26
Challenge 1: Establishing Cause/Effect
in an Open System
• Multiple actors & factors contribute
• Causality is mutual, therefore usually attribution is not possible
• Unintended results often ignored
27
Shifting Influence Over Time
Low
High
ProgramBeneficiaries/Indigenous Actors
Time
Influ
ence
28
Challenge 2: Timing
• Tracking incremental, cumulative change
• Not assessing results as though final
Are we having an
impact yet ?
30
Challenge 3: Encouraging Iterative Learning
• Emphasis on development results
• Easy to measure vs. important to know
• Keeping results within their context
• Making time for reflection
31
Accountability & Learning: A Balancing Act
32
Accountability & Learning: A Balancing Act
33
Challenge 4: Clarifying Values
• Assessment is not value neutral
• Making explicit the implicit
• Building the capacity to engage
34
Challenges and Outcome MappingChallenges
1. Cause & effect in open system
2. Timing
3. Iterative Learning
4. Clarifying Values
OM Tools
1. Vision & Outcome Challenges
2. Outcome challenge & Progress Markers
3. Org. Practices &Monitoring Journals
4. Mission, Boundary Partners, Progress Markers, Org. Practices
35
What is Outcome Mapping?
• an integrated PM&E tool
• an approach that focuses on changes in the behaviour, relationships,or actions of partners (as outcomes)
• a methodology that characterizes and assesses the program’s contributions to the achievement of outcomes
• an approach for designing in relation to the broader development context but assessing within your sphere of influence
37
At What Level?
• Course is focusing at the program level• Definition of Program: A group of related
projects and activities with a specific set of resources (human, financial, capital) directed to the achievement of a set of goals within a specified period of time
• Can be used by projects, organizations, and communities too
38
Key Concepts of Outcome Mapping
• Sphere of influence
• Outcomes as behavioural change
39
Sphere of Influence
Assessing
Planning & Design
Sphere of Influence
Context
40
The Focus of Outcome Mapping
Inputs Activities Outputs Outcomes Impacts
Behavioural Changes
41
What is an « Outcome » in OM?
• Change(s) in the behaviour, relationships, activities, and/or actions of an individual, group, or organization that the program was helpful in bringing about.
• These changes are aimed at encouraging human and ecological well-being.
42
Changes Sought
Ecosystem
Human System Changes in
Behaviour
Changes in State
43
Why Behavioural Change?
• To stress that development is done by, and for, people
• To illustrate that although a program can influence the achievement of outcomes, it cannot control them because ultimate responsibility rests with the people affected
44
How Can Outcome Mapping Be Used?
For a program to tell its performance story in outcome terms by:
• articulating its goals and designing its activities
• designing a monitoring system for assessing internal performance and outcomes of partners
• setting a use-oriented evaluation plan
45
Why Use Outcome Mapping?
Focussing on changes in partners’ behaviour, relationships, or actions allows a program to:
• measure results within its sphere of influence
• obtain feedback about its efforts in order to improve its performance
• take credit for its contributions to the achievement of outcomes
• show progress towards outcomes
Stage 1: Stage 1:
Intentional Intentional DesignDesign
« I have a dream! »
Martin Luther King
50
Step 1 : Vision Statement
• Reflects the broad human, social, & environmental betterment in which the program is engaged and to which it is contributing
Example Vision Statement
Local authorities, communities, and international organizations in developing countries in Africa recognize the value of HIV/AIDS intervention as an integral part of social & economic development. Municipal, regional, and national governments actively support HIV/AIDS prevention activities by formulating and implementing effective public health policies. Using research findings, they have developed a comprehensive public health strategy to slow down the infection rate. Formerly marginalized groups (e.g. women and youth) are organized into advocacy groups that can effectively formulate their needs to policy makers. All groups have access to reliable and relevant technical information about HIV/AIDS prevention and are able to make informed choices. In essence, there are healthier, happier, and wealthier communities.
52
Facilitation Questions
“Imagine that in 3-5 years the program has been extremely successful. What changes will you have helped bring about? What are your partners doing differently? What have they achieved? In essence, what would total success look like?”
VISION STATEMENT
Mission
The Mission is that “bite” of the Vision Statement on
which the program is going to focus
55
Step 2: Mission Statement
Describes how the program intends to:
• Operationalize its role in support of the vision by determining the areas in which it will work
• Support the achievement of outcomes by its boundary partners
56
Facilitation Questions
“How can the program best support or contribute to the achievement of the vision? What areas do you need to work in? What do you need to do? ”
Example Mission Statement
In support of this vision and on behalf of its donors, the program will work in the areas of research, dissemination, capacity building, & coordination. It will contribute to the production, synthesis, & dissemination of research data, position papers, & other information that will sensitize local & international actors to HIV/AIDS prevention. The program will seek to expand the range of disciplines involved in HIV/AIDS research. It will enhance HIV/AIDS research capacity in order to produce credible information for local, national, & international policy-making & program development. It will promote an interest in HIV/AIDS research among new researchers by providing research fellowships, mentorship, & training opportunities. The program will contribute to the development of linkages between Northern & Southern researchers & encourage partnerships between research organizations, advocates, & decision makers. It will increase its visibility & credibility among the donor community & will convince them of the utility of supporting HIV/AIDS prevention.
58
Developing a Vision or Mission
• Warm-up – general conversation• Individual write up• Discuss & Post• Facilitator and/or participant draft• Present & discuss• Purge the jargon• Verify with partners (if required) &
Finalize
59
Developing a Vision with the Community in Nagaland
60
Suggestions and Tips!
If time is a limiting factor you can carry out an email survey in advance to create a draft vision & mission before the workshop starts. Use the draft version as a starting point for discussion in order to save time.
Exercise 1:
Correct Errorsin Vision & Mission
Statements
63
Step 3: Boundary Partners
Those individuals, groups, & organizations with whom the program interacts directly to effect change & with whom the program can anticipate some opportunities for influence.
64
Program`s Sphere of Influence
Program
= Program`s Boundary Partners
The Real
World
65
Boundary Partners (have boundary partners)
Program Program’s Boundary Partners
Boundary Partners’ Boundary Partners
66
Typology of IDRC Boundary Partners
IDRC Program
Applied Research Community
e.g. researchers, institutions, NGOs, networks
System Gatekeepers / Dev`t Agents e.g. business, policy
makers, NGOs, gov`t
Beneficiaries
e.g. communities, farmers, women, youth
67
Facilitation Questions
« In which individuals, groups, or organizations is your program trying to encourage change so that they can contribute to the vision? With whom will you work directly? »
68
Suggestions and Tips!
• A program can choose its boundary partners because it wants to influence them or because they will influence others
• Example: a rural development NGO chooses to work directly with farmer organizations who in turn, lobby the Minister of Agriculture to effect change.
Exercise 2:
Identify the Program`s Boundary Partners
Exercise 2a
Working on Your Own Case
71
Guiding Questions
In your context:
• How would the ideas of Outcome Mapping resonate?
• Is a focus on behavioural change appropriate?
• How would you foster the imagination of your group to come up with a vision?
• Are you clear about your contributions to the vision? (i.e., your mission)
• Who are your boundary partners?
• How could you include them in the process?
Conclusion & Evaluation of Day 2
Plenary Discussion:What’s Working? What’s Not?
How Can We Improve Our Workshop?
Day 3:
Review of Day 2 & Feedback from Evaluation
75
Agenda for Day 3
77
Step 4: Outcome Challenge
Description of:
• The changed behaviours (relationships, activities, and/or actions) of a boundary partner
• How they could contribute ideally to the vision
Example Outcome Challenge Statement
The program intends to see researchers in developing countries that have gained the expertise required to carry out HIV/AIDS research in a multidisciplinary fashion using quantitative, qualitative, and/or participatory methodologies. They are conducting research on the legislative, economic, social, and health aspects of HIV/AIDS with special emphasis on vulnerable groups including youth and women. They are providing credible evidence that allows research findings to be translated into effective HIV/AIDS policies and programs. They are skillfully disseminating research results in the appropriate format for the audience (e.g., to the scientific community, the general public, NGOs, government officials). They are successfully obtaining funding from multiple sources to conduct HIV/AIDS research. More experienced researchers are mentoring those new to the field and HIV/AIDS researchers globally are engaged in productive communications using various means especially electronic networks.
« Those Who Dream Make a Difference! »
Ms. Kalpana Pant,Chaitanya
80
Phrasing of an Outcome Challenge
“The program intends to see [boundary partner] who [description of behaviours in the active present tense]”
81
Facilitation Questions
“Ideally, in order to contribute to the vision, how would the boundary partner be behaving?”
“What would your boundary partner be doing to contribute maximally to the vision?”
82
Suggestions and Tips!
• Similarity of behaviours sought among boundary partners (policy makers and community members) in international model forests was unexpected by program staff
83
Outcome Challenge Checklist
Outcome Challenge Statements:
Describe behavioural changesAbout a single boundary partnerSet out the ideal behavioural changesDescribe the boundary partner`s
contributions to the vision
Exercise 3:
Identify Errors in Outcome Challenge Statements
85
Program Framework
Vision:
Mission:
Boundary Partner:
Boundary Partner:
Boundary Partner:
Boundary Partner:
Outcome Challenge:
Outcome Challenge:
Outcome Challenge:
Outcome Challenge:
Overview Exercise:
Put Together a Program Framework
88
Step 5: Progress Markers
• A graduated set of statements describing a progression of changed behaviours in the boundary partner
• Changes in actions, activities, & relationships leading up to the ideal outcome challenge statement
89
Levels of Progress Markers
The program sets out what it would:
• EExpect to seexpect to see its boundary partners doing?
• LLike to seeike to see its boundary partners doing?
• LLove to seeove to see its boundary partners doing?
90
How Many Progress Markers?
Limit to a manageable amount of data to be collected:
4 “expect to see”
8 “like to see”
3 “love to see”
91
Why Graduated Progress Markers?
• Articulate the complexity of the change process• Permit on-going assessment of partners’ progress
(including unintended results)• Encourages the program to think about how it can
intentionally contribute to the most profound transformation possible
• Make mid-course corrections & improvement easier
92
Example Progress Markers
The program Expects to See local communities:
1. Participating in regular model forest (MF) partnership meetings
2. Establishing a structure for cooperation in the partnership
3. Acquiring new skills for involvement in the MF4. Contributing resources necessary to get the
MF operational (minimum)
93
Example Progress Markers
The program would Like to See local communities:
5. Articulating a vision for the MF that is locally relevant6. Promoting their involvement with the MF nationally7. Expanding the partnership8. Calling upon external experts when necessary9. Requesting new opportunities for training10.Disseminating concrete examples of benefits arising
from MF activities11.Seeking out new partners for the MF12.Obtaining funding from different national sources
94
Example Progress Markers
The program would Love to See
local communities:
13. Helping other MFs establish themselves
14. Sharing lessons-learned internationally
15. Influencing national policy debates & formulation on resource use and management
95
Facilitation Questions
« How can the program know the boundary partner is moving toward the outcome? What would they be doing? »
« What milestones would be reached as the boundary partner moves towards their intended role in contributing to the vision? »
96
Suggestions and Tips!
• If a program has multiple boundary partners but the primary result to be achieved is changing their relationships with one another then it is possible to create a single outcome challenge & set of progress markers for the whole group
97
Progress Marker Checklist
Each Progress Marker: Describes a changed behaviour by the boundary
partner Can be monitored & observed
As a set, Progress Markers: Are graduated from easier to more difficult to achieve
changes in behaviour Describe the change process of a single boundary
partner
Exercise 4:
Identify Correct Progress Markers & Categorize as
Expect, Like, or Love to See
100
Step 6: Strategy Map
• Outlines the program`s approach in working with the boundary partner
• Indicates the relative influence the program is likely to have on boundary partner
• Helps pinpoint strategic gaps in the approach or if the program is overextended
• Suggests the type of evaluation method appropriate to track and assess the performance of the program
101
6 Types of Strategies
Aimed at the
Boundary Parnter
Aimed at the Boundary Partner`s
Environment
Strategy SupportivePersuasiveCausal
I-1
• Direct Output
E-1
•Alter physical or regulatory
environment
I-2
• Arouse New Skills/ Thinking
I-3
•Supporter who guides change
over time
E-2
•Modify the information system
E-3
•Create / Strengthen a Peer Network
102
6 Types of Strategies
I-1
E-3E-2E-1
I-3I-2
103
Facilitation Questions
I-3I-2I-1
E-1 E-3E-2
What will bedone to producean “immediate”
output?
What networks/relationships willbe established
or utilized?
How willsustainedsupport,
guidance, ormentoring be
provided?
How will you usethe media orpublications?
What will bedone to build
capacity?
What will bedone to changethe physical or
policyenvironment?
104
Suggestions and Tips!
• Facilitator asks the group questions about their strategies
• While they are discussing them (s)he quietly plots them on a strategy map
• After being shown the strategy map, the group is asked to analyze it for gaps, overextension, and the influence that can be expected
Exercise 5:
Plot Activities on a Strategy Map
Exercise 5a:
Work on Your Own Case
107
Guiding Questions
In your own context:
• How well accepted would the idea of planning broadly but measuring within your sphere of influence be?
• What types of strategies do you use – causal, persuasive, supportive?
Conclusion and Evaluation of Day 3
Day 4:
Review of Day 3 and Evaluation Feedback
111
Agenda for day 4
« The oft repeated and echoing question in Outcome Mapping « what or who needs to
change? » raised us from being providers of development, achieving outputs, to actually
believing we were agents of change. »Raj Verma, NEPED
114
Step 7: Organizational Practices
The things that you do as an organization to:
• foster creativity & innovation
• seek the best ways to assist your partners
• maintain your niche
115
Why Organizational Practices?
• Important to how the program is going to function to effectively fulfill its mission
• Supporting change in its boundary partners requires that the program be able to change and adapt as well
116
8 Organizational Practices
1. Prospecting for new ideas, opportunities, & resources
2. Seeking feedback from key informants
3. Obtaining the support of your next highest power
4. Assessing & (re)designing products, services, systems, and procedures
117
8 Organizational Practices
5. Checking up on those already served to add value
6. Sharing your best wisdom with the world
7. Experimenting to remain innovative
8. Engaging in organizational reflection
118
Facilitation Question
“What do you have to be doing in order to remain effective?”
119
Suggestions and Tips!
• This can be a difficult set of concepts to get across; frame it in terms of changes in behaviour internal to the organization
• Have the group think of the program as an organization in its own right
Exercise 6:
Identify the Organizational Practices
Exercise 6a:
Draw the 7 steps of Outcome Mapping
122
Overview of Stage 1
Outcome Mapping: Main Elements
Boundary Partners
Vision
Mission
O.C. OutcomeChallenge
O.C.
Progress Markers
Strategies
Organizational Practices
Stage 2: Stage 2:
Outcome & Outcome & Performance Performance MonitoringMonitoring
124
Outcome Mapping Monitoring
• Systematic collection of data
• A regular learning & improvement cycle
• Encourages the program to challenge itself
• Collects & organizes, does not analyze data
« Being busy creates a mindset that is not conducive to innovation and creativity.
Without interaction, there is no innovation. Time to discuss, reflect, and generate new ideas is the ransom that outcome
mapping demands for innovation. »
Thierry Barreto-Fernandez
West Africa Rural Foundation, Senegal
127
Step 8: Monitoring Priorities
• Tempting to gather information on a whole range of issues but this is not practical
• Base priorities on intended use of intended users:– Use(s) for findings and from process– Improve performance through learning – Help meet accountability/reporting requirements– Supply information for planned evaluation – Inform publicity documents, communication activities, or
case-study materials
128
Monitoring Plan
Monitoring Priority
Who will use
Purpose When is it
needed
Who will
collect
How often
Proposed Tool
Strategies Strategy Journal
Org’l Practices
Performance Journal
Boundary Partner Outcomes
Outcome Journal
Boundary Partner
Program
Outcome Challenge & Progress Markers
Boundary Partner
Program
Strategies & Activities
Boundary Partner
Program
Organizational Practices
Boundary Partner
Program
Outcome Journal
Strategy Journal
PerformanceJournal
Boundary Partner
Program
134
Organizing Monitoring Meetings
• Regular face-to-face meetings to record observed changes
• Filling out electronic data sheets on an on-going basis, then aggregating changes for entire program
• Interviews and/or focus groups with boundary partners
136
Step 9: Outcome Journal
• Uses progress markers to chart boundary partner`s progress
• Sets starting point or benchmark
• Captures details on changes in the boundary partner, contributing factors & actors, and sources of evidence
137
Outcome Journal
Work Dating From/To:
Contributors to Monitoring Update:
Low =
Medium =
High =
Outcome Challenge:
LMH
Expect to see: Who?
3
1
2
138
Set Values for Low, Medium & High
• Can capture either quantity or quality of change:
– Quantity (number of boundary partners)– Quality (depth of change)
139
Baseline
• multiple individuals, groups or organizations in each boundary partner set
• identify each boundary partner’s starting point in terms of progress markers
• boundary partners may be at different stages of development – individual change process can be captured
over time
141
Step 10: Strategy Journal
• Captures data on the strategies being employed to encourage change in the boundary partner
• Helps the program gauge its contributions to the achievement of outcomes and modify its actions accordingly
• Feeds into work planning & (modifications) or new activities
142
Strategy Journal
Working Dating From/To:
Contributors to Monitoring Update:
Strategy to be Monitored:
Effectiveness?
(How did it help the boundary
partner?)
Outputs
144
Step 11: Performance Journal
• Records data on how the program is operating as an organization to fulfill its mission
• Includes information on the program’s practices
• Feeds into work planning
145
Performance Journal
Working Dating From/To:
Contributors to Monitoring Update:
Practice 1: Prospecting for New Ideas, Opportunities, & Resources
Practice 2: Seeking Feedback from Key Informants
Example or Indicators:
Sources of Evidence:
Lessons:
Example or Indicators:
Sources of Evidence:
Lessons:
Exercise 7
Identify your monitoring priorities (e.g. work on your
real case)
Stage 3: Stage 3:
Evaluation Evaluation PlanningPlanning
« Evaluative thinking is a learned behaviour that you
have to practice and improve at on an on-going basis! »
Michael Quinn Patton
150
Step 12: Evaluation Plan
• A short description of the main planning elements of an evaluation study
• For Whom? What? Where? How? By Whom? When? How Much?
151
Elements of an Evaluation Plan
Who Will UseEvaluation,
How, & When?
Questions to be
Answered
Info. Sources
EvaluationMethods
Who WillConduct &
Manage theEvaluation?
Date(Start &Finish)
Cost
Evaluation Issue:
152
Setting Evaluation Priorities
Criteria with which to select include:
• Learning needs
• Reporting requirements
• Partners’ information needs
153
Guiding Principles for Evaluation
• Balancing learning & accountability
• Action oriented
• Each study is unique
• Process not only product
• Build capacity at all levels through participation
« Not everything that counts can be counted. And not
everything that can be counted, counts. »
Albert Einstein
155
An Evaluation of IMFNS
• Assessment of 1st phase
• Wanted to measure human, community, & partnership elements of model forests
• Client acceptance of methodology essential
• OM provided a framework for the study
156
Evaluation Questions
1. Who has IMFNS reached?
2. How have the behaviours of IMFNS`boundary partners changed?
3. What has been the nature of IMFNS` interactions with its partners?
157
Process of the IMFNS Evaluation
• Recreated planning framework in 1 day with IMFNS staff
• Developed progress markers for communities & goverment officials across the 5 sites
• Conducted a document review & key informant interviews
• Comparison of boundary partners across MF sites for analysis
158
Comparing Boundary Partners
ProgressMarker
Chile -Chiloe
Russia -Gassinsky
Mexico -Chihuahua
Mexico -Calakmul
Mexico -Monarch
Expect to see local communities:
1 ! ! ! ! !
2 ! ! ! ! !
3 ! ! ! ! !
4 ! ! ! ! !
Like to see local communities:
5 ! ! ! ! !
6 ! ! ! ! !
7 ! ! ! ! !
8 !
9 ! ! ! !
10 Potential !
11
12
Love to see local communities:
13
14 ! !
15 !
159
Lessons from the IMFNS Study
• Difficult to link what the Secretariat did with outcomes achieved without using cause-effect
• Boundary partners should have been involved in the development of the framework & analysis of data
• Should have included more analysis of organizational practices
• Did not study the relevance of boundary partners within their environment
160
A Self-Assessment of NEPED
• Preparation for external evaluation• Process
– Fred knew people and context prior to workshop– Retrospectively built the framework (took 1.5 days
including HML values)– « Ah-ha » moment happened at end for some
participants– Organizing data collection (preliminary
identification, collection and verification, analysis and writing report took another 1.5 days)
161
Lessons from the NEPED Study
• Group dynamics (worked together well,willing to trust Fred, cooperative)
• Data (they had the data but it wasn`t organized)
• Preparation for external review (intended use)• Relevance of work with state government
(beyond trees, number of women involved, number of villages worked with)
• Could have had greater external participation (Gov`t of Nagaland, etc.)
ConclusionConclusion
164
Conclusion
• A system to think holistically & strategically about how to contribute maximally to development
• A method to articulate contributions to the development process through boundary partners
• Outcomes are changes in the behaviour, actions, and/or relationships of boundary partners
• An approach that encourages on-going learning, challenges, & improvement through M&E activities
Conclusion and Evaluation of Day 4
Day 5:
Review of Day 4 & Evaluation Feedback
168
Agenda for Day 5
Practicum Activity 1:
Development of a vision statement
Practicum Activity 2:
Advocating the Use of Outcome Mapping Concepts
in your Work Setting
Conclusion and Evaluation of Day 5
Day 6:
Review of Day 5 and Evaluation Feedback
174
Agenda for Day 6
Back to Work…
How are you going to put your training in Outcome Mapping
to use?
Practicum Activity 3:
Planning Adaptations to Outcome Mapping
177
Were Your Expectations Met?
Evaluation of the Workshop
In terms of the content and process, what should we
keep, add, change?
Thank you!
http://www.idrc.ca/evaluation