participatory approaches to evaluation puja myles [email protected]
TRANSCRIPT
![Page 2: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/2.jpg)
Session outline
• What is Participatory evaluation?• Philosophical underpinnings • Barriers to participatory approaches• Participatory Action Research (PAR)• Most Significant Change technique (MSC)
![Page 3: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/3.jpg)
Participatory evaluation-1
“Participatory evaluation is a partnership approach to evaluation in which stakeholders actively engage in developing the evaluation and all phases of its implementation”
-Zukoski and Luluquisen (2002)
![Page 4: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/4.jpg)
Participatory evaluation-2
“Participatory monitoring and evaluation is not just a matter of using participatory techniques within a conventional monitoring and evaluation setting. It is about radically rethinking who initiates and undertakes the process, and who learns or benefits from the findings.”
-Institute of Development Studies (1998)
![Page 5: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/5.jpg)
Participatory evaluation-3
“Participatory evaluation means more than just a way of seeing how much has been achieved or produced at what cost and with what effect…Participatory evaluation is not just to do with the development of things. It is to do particularly with the development of people.”
-Feuerstein (1986)
![Page 6: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/6.jpg)
Historical and Philosophical basis
“The basic ideology of PAR is that self-conscious people, those who are currently poor and oppressed, will progressively transform their environment by their own praxis. In this process others may play a catalytic and supportive role but will not dominate"
-Fals-Borda, 1991
![Page 7: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/7.jpg)
Freirean approach to community development
• Paulo Freire (Pedagogy of the Oppressed)• An educational philosophy that actively
involves the poor in critically analysing their social situation (conscientisation), thus creating the potential for them to transform their environment.
• Praxis: reflection-action dynamic
![Page 8: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/8.jpg)
Group reflection
• How participatory should participatory evaluation be?
• What are the advantages of using participatory approaches to evaluation?
• Can a participatory evaluation replace traditional evaluation methods?
• Are participatory approaches to evaluation valid?
• What are the barriers to participatory approaches?
![Page 9: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/9.jpg)
Participatory Action Research (PAR)
“Put simply, action research is ‘learning by doing’ - a group of people identify a problem, do something to resolve it, see how successful their efforts were, and if not satisfied, try again.”
-Rory O’Brien (1998)
![Page 10: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/10.jpg)
The action research process
• Identify problem• Collect data for detailed diagnosis• Collective postulation of possible solutions• Agree and implement an action plan• Collect data on results of intervention• Analyse data• Interpret findings in relation to success of
action• Re-assess problem• Begin another PAR cycle
![Page 11: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/11.jpg)
Principles of action research (Winter 1989)
• Reflexive critique• Dialectical critique• Collaborative resource• Risk• Plural structure• Theory, practice, transformation:
theory informs practice and practice refines theory
![Page 12: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/12.jpg)
Application of PAR in health
• Health Needs Assessment• Planning health care services• Service evaluation• As a process of empowerment• Can employ both qualitative and
quantitative methods• Qualitative tools include: journals,
interviews, focus groups, filming, use of pictures or drawings
• Quantitative tools: surveys, monitoring data
![Page 13: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/13.jpg)
RAP, RRA, PRA, PLA
• Rapid Assessment Procedure (RAP)• Rapid Rural Appraisal (RRA)• Participatory Rural Appraisal (PRA)• Participatory Learning and Action
(PLA)
![Page 14: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/14.jpg)
RAP techniques for data collection
• Proportional piling: visual form of estimating proportions.
• Community maps: Drawn in groups; Useful to learn about an area and how different groups use it
• Transect walk: observation technique• Diagramming: Venn (chapati) diagrams
to represent power structures (power represented by size; relationships by overlap; exclusion by distance)
![Page 15: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/15.jpg)
Most Significant Change
Aka ‘Monitoring-without-indicators’, ‘the story approach’, ‘evolutionary approach
to organisational learning’…
![Page 16: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/16.jpg)
The kernel of the MSC process
• A question:“Looking back over the last month, what do you think was the most significant change in [particular domain of change]?”
• A second question:“From among all the significant changes, what do you think was the most significant change of all?”
![Page 17: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/17.jpg)
Steps
• Start and raise interest• Define domains of change (e.g. quality of life, health,
participation/empowerment)• Define reporting period• Collect significant change (SC) stories• Select the most significant of the stories (hierarchical
level)• Feed back results of the selection process• Verification of stories• Quantification• Secondary analysis and meta-monitoring• Revising the system
![Page 18: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/18.jpg)
When to use MSC
• Complex programmes with diverse and unexpected outcomes
• Focussed on social change• Participatory in ethos• Highly customised services• Focus on learning rather than just
accountability• Enablers: organisational culture of ‘no-
blame’; willingness to try something different; commitment by senior managers
![Page 19: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/19.jpg)
Defining domains of change
• Domains are broad and often fuzzy categories of possible SC stories
• A domain of change is not an indicator (deliberately fuzzy as opposed to SMART)
• Include an ‘open window’ domain• Could include a domain for negative
changes ‘lessons learned’ or ‘changes that reflect an area to improve’
• Could be based on programme objectives• Between 3-5 domains are best
![Page 20: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/20.jpg)
Collecting the SC stories
• Information to be documented includes:• Who collected the story and where the
events occurred• Description of the story itself- what
happened (sequence of events)• Significance (to the storyteller) of the
events described in the story• Stories could be 1-2 pages long
![Page 21: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/21.jpg)
Selecting the most significant of stories
• Hierarchy of selection processes• Criteria for selecting stories: Majority
vote, iterative voting, scoring• Documenting the results of the selection
process: the reasons for selecting an SC story as the most significant should be documented and attached to the story
![Page 22: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/22.jpg)
Secondary analysis and meta-monitoring
For all collected stories:• Thematic coding• Look for positive and negative changes• Analyse changes mentioned in MSC
stories against a logic model• Analyse genre• Analyse differences between selected
stories and those not selected
![Page 23: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/23.jpg)
What can MSC offer as an evaluation technique?
• Uses an inductive approach to help identify unexpected outcomes
• Helps monitor messy impacts, intangible and indirect consequences
• The hierarchical selection process of sorting through diverse SC stories becomes an opportunity for making implicit values explicit, communicating shared values and agreeing a direction for the project
![Page 24: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/24.jpg)
Validity and voice in MSC
Thick description: richness of detail and context
Systematic process of selection (compare to case studies)
Transparency (systematically records interpretation)
VerificationParticipation in keeping with the broader
programme philosophy of empowermentPurposeful sampling
![Page 25: Participatory approaches to evaluation Puja Myles Puja.myles@nottingham.ac.uk](https://reader033.vdocuments.net/reader033/viewer/2022052522/5515d136550346cf6f8b4636/html5/thumbnails/25.jpg)
Further reading
1. Davies, R. and Dart, J. (2005).The ‘Most Significant Change’ (MSC) Technique: A guide to its use
(http://www.mande.co.uk/docs/MSCGuide.pdf)
2. Action research resources
(http://www.scu.edu.au/schools/gcm/ar/arhome.html)