partners in evaluation

Partners in EvaluationCo-constructingeducational evaluation at the local level

Ruth Pritchard and Carol McSporranPresentation to ANZEA 2013

Nāu te rourou, nāku te rourou, ka ora te iwi.

With your food basket and my food basket,

people will thrive.‘Nga Kite Whai Matauranga’ by Mihi Williams and Roimata Emery

The Notion of PartnershipTe Tiriti o Waitangi• - Partnership• - Protection• - Participation

embedded in evaluationprinciples, standardsand competencies

•Serious engagement with differenceengagement with diversity and difference in evaluation is both substantive and a moral commitment (Hood, Hopson & Frierson, 2005, p.9)

•Values-engaged approachevaluators must assume responsibility for explicating and justifying the values being advanced in their work, in ways that respect other values (Hall, Ahn & Greene, 2012, p. 206)

Tukutuku: partnership and participation

From national library and te papa archive collections

Te Whare Runanga, Waitangi House

Co-constructing the big pictureERO’s complementary evaluation

Weaving on a 2-dimensional framework • Ko te tamaiti te putake o te kaupapa

The child is the heart of the matter• Evaluation – complementary review

Co-constructionLincoln and Guba (1985); Stake (1975; 1983)Morris (2002)• Multiple perspectives build towards ‘a

commonly accepted reality’• “Stakeholder involvement using the principles of

social constructivism should lead to a more valid evaluation that empowers stakeholders and increases the likelihood of the utilization of results as stakeholder investment is greater than would otherwise be the case” (p. 51)

Participatory evaluation as partnership(e.g. Patton, 2008, p. 224; Cousins & Earl, 1992; Cousins & Whitmore, 1998; Greene, 1988; Ayers, 1987).

• Complex dynamic operating at three levels: individual,group, organisational(Smits & Champagne, 2008)

• Comprising five interactive processes (Weaver & Cousins, 2004)

Cousins & Whitmore, 1998; Cousins & Chouinard, 2012

“this shifting responsibility is a prerequisite for the success of an evaluation” Themessl-Huber & Grutsch, 2003, p. 98.

Complementary evaluation(Mutch, 2012; Nevo, 1994, 2002)

•Appreciative Inquiry approach (Cooperrider, 1982)

•Deeper participation, fewer/primary users (Taut & Alkin, 2010)

•Transparent processes and values•Complementary relationships

Hopkins & West (2002)

ERO’s approachEnacting partnership at the local level•contextually responsive design

•negotiated values

•dialogic basis for engagement

•space for shared sense-making

ERO (2013)

Opportunities for weaving threads• Reciprocity of learning

bridge the gap between theory and practice…and create opportunities for mutual learning” (MacLellan-Wright. et al., 2007, p. 102).

• Enhancing validitythe experiences and expertise of stake-holders enlarge and enrich the scope of the evaluation (Abma, 2000)acknowledged as ‘natural experts in their field’ (Themessl-Huber & Grutsch, 2003, p. 94).

• Evaluation capability building (Compton, Baizerman & Stockdill, 2002)

•Utilization process use (Greene, 1988; Patton, 1998) contributes to a learning phase that reinforces understandings and the ownership of results and, eventually, a greater sense of obligation to follow through on the results (Smits & Champagne, 2008, p. 429).

Position of the evaluator

• a shift from the primary investigator and participant observer to becoming a facilitator (Garaway, 1995)

• a “high tolerance for ambiguity” (Patton, 2011, p. 26)

• cultural responsiveness with “careful attention to one’s place and perspective” (Letiecq & Bailey, 2004, p. 32)

putting ourselves into practice (Collins 1994)it is as much about who we are, and where we position ourselves in relation to others, as it is about what we do (ANZEA, 2011, p. 10)

•use diverse methods of inquiry•suspend judgement and sense-making•create time and space for participants to:

- articulate- explore- co-construct knowledge - develop their evaluative capacity

Repositioning evaluators to:

Challenges•Complex contexts and competing agendas- growth in reliance on evaluation- diversity in the sector

- political implications

•Competence and understanding of practice/theory- moving from apprenticeship to a professional model- variability

• Evolving, dynamic process and expectations - high expectations for creativity, innovation

and cultural responsiveness

- we can be deliberate and responsive in the evaluation approaches we use and they can co-exist :participatory, empowerment, appreciative, transformative, collaborative, deliberately democratic, developmental, responsive, and utilization focused

Enabling partnership - what seems to work?

•Communication and information sharing

•Inviting participation at different levels•Shared team and context understandings

•Professional Practice Model (PPM) initiation

Enabling partnership - what seems to work? (contd.)•Review Officers’ understanding of expected

practices and principles for complementary practices:

- clarity of roles and responsibilities- agreed evaluation framework- recognition of the contribution each party makes- respect for different perspectives (Mutch, 2012)

•Synergy between internal and external evaluation – making connections at the local level

Pūawaitanga o te Ringa - Fruits of our busy hands, 2001Christchurch Central Library

ReferencesAbma, T. A. (2000) .‘Stakeholder Conflict: A Case Study’, Evaluation and Program Planning, 23(2): 199–210.ANZEA (2011). Evaluator Competencies 2011 Retrieved from

www.anzea.org.nz/.../05/110801_anzea_evaluator_competencies_final.pdf ·Ayers, T. D. (1987). Stakeholders as partners in evaluation: A stakeholder-collaborative approach. Evaluation and Program

Planning, 10, 263-211.Collins, M. (1994). Putting ourselves into practice: New prospects for program planning and evaluation. In M. Brown (Ed.), A

collection of original essays on curriculum for the workplace, 94–131. Victoria, Australia: Deakin University Press.Compton, D. W., Baizerman, M., & Stockdill, S. H. (2002). The art, craft, and science of evaluation capacity building. New

Directions for Program Evaluation, 93. Cooperrider, D & Srivastva, S. (1987). Appreciative Inquiry in organizational life. Research in Organizational Change and

Development 1, 129-169. Cousins, J.B. & Whitmore, E. (1998). Framing participatory evaluation. In E. Whitmore (Ed.), Understanding and practicing

participatory evaluation. New Directions for Evaluation, 80, 2-23.Cousins, J.B. & Chouinard, J.A. (2012). Participatory evaluation up close: An integration of research-based knowledge.

Charlotte, NC: Information Age Publishing. Cousins, J. B., & Earl, L. M. (1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14,

397–418. ERO (2013). He Pou Tātaki:How ERO reviews early childhood services. Wellington, New Zealand: Education Review Office.ERO (2012). Statement of Intent 2012-2013. Wellington, New Zealand: Education Review Office.Garaway, G. B. (1995). Participatory practice. Studies in Educational Evaluation, 21,. 85-102,Greene, J.G. (1988). Stakeholder participation and utilization in program evaluation. Evaluation Review, 12, 91–116.Hall, J. N., Ahn, J. & Greene, J. C. (2012). Values engagement in evaluation: Ideas, illustrations, and implications. American

Journal of Evaluation, 33, 195-207.Hawkins, P. (2012). Successful evaluation management: Engaging mind and spirit. The Canadian Journal of Program

Evaluation, 25 (3) 27–36.

Hood, S., Hopson, R. & Frierson, H. (2005). Introduction : this is where we stand S. Hood, R. Hopson, and H. Frierson (Eds.). The role of culture and cultural context: a mandate for inclusion, the discovery of truth and understanding in evaluative theory and practice . Greenwich, CT: IAP

Hopkins, D., & West, M. (2002). Evaluation as school improvement. In D. Nevo (Ed.) School-based Evaluation: An International Perspective. Oxford, England: Elsevier Science.

Letiecq, B. L. & Bailey, S. J. (2004). Evaluating from the outside. Evaluation Review, 28(4), 342-357Lincoln, Y. & Guba, E. (1985). Naturalistic Inquiry. Thousand Oaks, CA.: Sage MacLellan-Wright, M.F., Patten, S., de la Cruz, A.M. & Flaherty, A. (2007). A participatory approach to the development of an

evaluation framework: process, pitfalls, and payoffs. The Canadian Journal of Program Evaluation 22(1), 99–124.Morris, D.B. (2002). The inclusion of stakeholders in evaluation: benefits and drawbacks. The Canadian Journal of Program

Evaluation, (17)2, 49-58.Mutch, C. (2012). Complementary evaluation: the development of a conceptual framework to integrate external and internal

evaluation in the New Zealand school context. Policy Futures in Education, 10(5), 569-586.Nevo, D. (1994). Combining internal and external evaluation: A case for school-based evaluation. Studies in Educational

Evaluation, 20(1), 87-98.Nevo, D. (2002). Dialogue evaluation: combining internal and external evaluation. In D. Nevo (Ed.) School-based Evaluation: An

International Perspective. Oxford, England: Elsevier Science.Patton, M. Q. (1998) ‘Discovering Process Use’, Evaluation 4(2): 225–33.Patton, M. Q. (2008). Utilization-Focused Evaluation (4th ed.).Thousand Oaks, CA; Sage Publications Inc.Smits, P.A. & Champagne, F. (2008). An assessment of the theoretical underpinnings of practical participatory evaluation.

American Journal of Evaluation 29, 427-442.Stake, R. E. (1975). Evaluating the Arts in Education. Colombus, OH: Merrill. Stake, R. E. (1983). Program evaluation, particularly responsive evaluation. In G. F. Madaus, M. Scriven, & D. L. Stufflebeam

(Eds.), Evaluation Models. Boston: Kluwer-Nihoff.Taut, S. & Aiken, M. C. (2010). The role of stakeholders in educational evaluation. Educational evaluation: Contexts of Educational

Evaluation, 629-635.Themessl-Huber, M.T. & Grutsch, M.A. (2003). The shifting locus of control in participatory evaluations. Evaluation 9, 92-111.Weaver, L., & Cousins, J. B. (2004). Unpacking the participatory process. Journal of MultiDisciplinary Evaluation 1, 19-40.