Title Slide Making Evaluation Judgements

A quick overview

He Oranga Poutama Provider Hui, 12-13 October 2009 - Te Manukanuka o Hoturoa Airport Marae

Nan Wehipeihana, Kataraina Pipi and Kate McKegg

Developmental evaluation- A practitioner’s introduction

Kate McKegg and Nan Wehipeihana

Our whakapapa

• Who am I?• Where am I from?• What is my history?

Introductions• Who are you?• Where are you from?• What brings you to this workshop?• What do you hope to learn today?

1. Pair up with someone you don’t know (or don’t know very well)

2. Listen and pay attention to why each other is here today3. Tell us your partner’s name and what they hope to get

out of this session

Workshop OutlineTime Focus

8.30 – 9.00 Whanaungatanga / Introductions

9.00 – 10.00 Introduction to Developmental Evaluation (DE) – what is it?

10.00 – 10.30 Morning tea

10.30 – 12.30 DE - the why, the where and the who

12.30 – 1.30 Lunch

1.30 – 3.00 DE in practice - inquiry approaches

3.00 – 3.30 Afternoon Tea

3.30 – 4.30 Final thoughts, tips and questions

Developmental Evaluation - what is it?

Developmental evaluation supports innovation development to guide adaption to emergent

and dynamic realities in complex environments.

(Michael Quinn Patton, 2010)

Primary developmental evaluation purpose

Complex system challenges Implications

1. Ongoing development Being implemented in a complex and dynamic environment

No intention to become a fixed/standardised modelIdentifies effective principles

2. Adapting effective principles to a new context

Innovative initiativeDevelop ‘own’ version based on adaption of effective principles and knowledge

Top-down—general principles knowledge disseminatedBottom-up—sensitivity to context, experience, capabilities and prioritiesAdaptation vs Adoption

3. Exploring real-time solutions and generating innovative responses in the face of sudden and turbulent major change

Exisiting initiatives and responses no longer effective as conditions change suddenly

Planning, execution and evaluation occur simultaneously

Five purposes of developmental evaluation

Primary developmental evaluation purpose

Complex system challenges Implications

4. Pre-formative development of potentially scalable innovative

Changing and dynamic situations require innovative solutions to worsening conditionsModel needs to be developed/does not exist

Models may move into formative and summative evaluation, others remain in developmental modeInform different potential scaling options

5. Major systems change and cross scale developmental evaluation

Disrupt existing systemTaking an innovation to scaleMajor systems change and changing scale will add levels of complexity, new uncertainties and disagreements

Models change as they are taken across time, space and to larger systemsAdaptive cross scale innovations assume complex, nonlinear dynamics—agility and responsivenessAdaptation -- Replication

Five purposes of developmental evaluation

Adapted from Patton (2010)

Development and innovation in complex situations

• Intractable issues e.g., poverty, homelessness, environmental degradation, family violence, obesity

• Uncertainty about options and solutions• Political and economic turbulence • Environment characterised by unpredictability and

the unexpected emergence of issues and ideas (e.g., rise of social networking)

• Rapid adaption and change occurring across many contexts

DE is informed by complexity and systems thinking (picture sourced from Patton, 2009)

Some key complexity concepts• Adaption• Emergence• Non-linearity• Uncertainty• Interdependence

Exercise One: Handout 1: match each scenario with a complexity concept

Concept ScenarioSomething phenomenal comes out of the blue – like Susan Boyle

So, the fact that people know each other is more influential than expected

Flock of birds flying in formation

Iceland volcano

The introduction of the possum to New Zealand

Concept Brief overview Application to DE

Adaption Interacting elements respond and adapt to each other so that what emerges is a function of the relationships between different agents over time

The essence of DE. An evaluation process that adapts to the needs of programs and organizations, and to the relationships among program stakeholders and the evaluators themselves. Learning by doing.

Emergence Patterns emerge that are beyond, outside of and oblivious to any notion of shared intentionality – the whole is greater than the separate parts

Watching for things to percolate up from interactions, capturing those ideas and new relationships, and placing them in front of people as options for further development

Non-linearity Sensitivity to initial conditions; small actions can stimulate large reactions – a small increase in one variable produces a large increase in another.

Being alert for tipping points or critical incidences is a key part of the DE evaluator’s role, noting forks in the road that can lead to significant changes in programs or organisations – and this often happens in unpredictable ways.

Uncertainty Unpredictable conditions, sometimes unknowable in advance – a ‘maybe’ condition

In DE, the evaluation process cannot be fixed. It needs to have flexibility built in, such as a preparedness for program processes and outcomes to change. Measurement also needs to be flexible.

Interdependence Relationships between different elements are highly interdependent

Paying attention to the interrelationships and inter connections that create feedback loops is a key part of DE. This is important because very often in dynamical situations, we can sometimes not predict what might happen, we only really know once it’s happened.

Some complexity concepts: Handout 2: practical application

DE - a distinct niche• Two distinct niches– Pre-formative– Dynamic situations

• Support exploration and innovation before there is a program model (pre-formative)

• Support the ongoing development and adaption of a programme, or innovation in emergent and complex situations (dynamic situations)

DE - a distinct nicheDifferent to formative and summative• Differs from improvement orientated evaluation (making

a programme better)• It aims to support the ongoing real-time decisions –

what to change, expand, close out further developImportant Distinction:• Differentiating ongoing strategic thinking and periodic

strategic planning • DE as a form of thinking and acting strategically as an

innovative intervention unfolds

• Increase participation in sport and recreation – ‘by Māori’

• 2007-08 review and programme ‘re-visioning’

• Focus on participation ‘as Māori’ participation– Tino rangatiratanga (self

determination)

– 20 years of Māori development

The story of He Oranga Poutama

Case example – He Oranga Poutama and complexity

• Uncertainty around the new programme concept and direction, not sure what ‘as Māori’ participation would involve

• New innovative programme and goal• Values, principles and vision driven• No formal evaluation evidence from other efforts

about what to expect, no real data available to measure key concepts and variables

• Complex environment – contracting economy, political environment, changing organisation

DE – and creative thinking• Requires critical AND creative thinking• We have to think outside the box• Ask questions about what makes sense as we

try to connect evaluation to the program development

Draw four straight lines which go through the middle of all of the dots without taking the pencil off the paper

Creative thinking

Creative thinking:Thinking outside the box

http://www.brainstorming.co.uk/puzzles/ninedotsnj.html

Start here 1

Step 2

Step 3

Step 4

MORNING TEA

When not to use DE

• Not able or willing to commit the time to actively participate in the evaluation and to build and sustain relational trust – no room for the set and forget evaluation management approach

• High levels of certainty required – no tolerance for ambiguity

• Lack of openness to experimentation and reflection – high level of design certainty required

• No adaptive capacity – no real ability to make changes

• Unwillingness to ‘fail’ or hear ‘bad news’ – fear of failure context overrides learning focus

• Poor relationships between management and staff and evaluators – respectful and trusting relationships critical in a context of uncertainty, ambiguity and turbulence

Traditional evaluation is not well suited to all contexts

Traditional evaluation paradigms Developmental evaluation

Manageable and relatively stable situations

Complex dynamic situations, rapidly changing environments

Root cause(s) of problems being addressed is known and reasonably bounded

No known solution to issues, new issues entirely, no certain way forward

Intervention reasonably well defined Multiple pathways possible, need for innovation and exploration

Key variables expected to affect outcomes are knowable, and reasonably measurable

Unknown outcomes, vision and values driven processes

Exercise 2:Teasing out complexity

Handout 3:– Read the brief description of Whānau Ora– What if anything is complex about Whānau Ora?– Share and discuss in small groups– Make notes to feedback to the larger group

Roles and relationships of the DE evaluator

• Evaluator is part of the innovation team• Facilitator and learning coach• Brings evaluative thinking to the group• Support or share innovators’ values and vision• Support the feedback process around “what is

being developed”

Context / Organisation Evaluator – ‘The pragmatic bricoleur’

High levels of awareness of context, and changes in the wider environment

Vigilance to tracking internal and external emergence

Willingness to balance development and innovation with a commitment to testing reality

High tolerance for ambiguity, as well as the ability to facilitate values based sense making, argument and decisions

Willingness to explore, dig deeper, capacity to interpret data, make sense of feedback

Methodological agility and creativity combined with a willingness and ability to change and respond – design, framework, programme theory, methods etc

Courage to face the possibility that some things may not be going so well, adapt and keep going

Courage to take on ‘messy’ journey of ups and downs, sidetracks and the unexpected all the while retaining a tolerant and critical open mindedness and commitment to truth telling

Preparedness to co-create the future, collaborate and trust

Preparedness to develop long term relationships of trust – be ‘in for the long haul’

DE - the context and the evaluator

Skills, abilities and dispositions• The ability to build and maintain trusting relationship

– Critical in a context of uncertainty, ambiguity and turbulence

• High level of facilitation skills, multiple settings, stakeholders and contexts– To generate and ‘test’ ideas, to provide feedback, to ask the ‘tough’ questions;

highly engaging, to build evaluative capability

• Deep methodological toolkit– Methodological flexibility, eclecticism, and adaptability (Patton, p27, 2010)

• Enquiring– Observant and critically reflective

• Systematic – Attention to data and making data driven decisions

• Courageous– Flexible, adaptable, responsive ‘on the run, on the fly’

LUNCH TIME

What will happen to the piece of wood when the person lets go of it?

If the person is on earth…

If the person is under water…

If the person is in space…

The block drops to the ground

The block floats to the surface

The block will not move because there are no overall forces

DE in practice - Inquiry approaches

• Focus of DE is on what is being developed?• What’s emerging?• Given what’s emerged, what’s next?

The DE evaluator:• Inquires into developments• Tracks developments• Facilitates interpretation of developments so that

judgments can be made about the what, how, impact and consequence of developments

DE: Asking questions that matter and matching questions to context• Patton (2010) suggests that questions that matter

can be thought of as “a tool for working in complex situations” (p227)

• Matching questions to particular situations is a central challenge - situational responsiveness and adaption

6 simple rules (Patton, 2010)1. Connect questions with the ideas, language and frameworks

of the people you are working with2. Less is more. Limit the number of questions within the

enquiry framework. 7+ or - 2 is a good rule of thumb3. Keep the evaluation grounded in whatever basic

developmental inquiry framework you and those you’re working with choose to guide your work

4. Distinguish overarching inquiry questions from detailed implementation questions

5. There are (no) stupid questions6. Remember that whatever inquiry framework you are using,

the focus is on What is being developed?

DE – is values based‘[DE] sits alongside, doesn’t control or dampen the core values of

innovation’ (Wehipeihana, cited in Patton, 2010).

• Values / processes / relationships become very important

• ‘How’ outcomes / results are achieved is very important• Process matters• People and relationships matter• Where the end is unpredictable and emergent, values

and process become anchors‘there will never be enough certain knowledge to guide action’ Wendell Berry cited in Patton, 2010.

DE – inquiry approaches• There is no definitive list of developmental evaluation

inquiry approaches - neither should one be constructed• Developmental evaluation creatively adapts whatever

approaches and methods fit the complexities of the situation.

• It is responsive, appropriate, and credible to those you work with, and support opening up new understandings and guiding further developmentIt’s all about persistently asking questions and pursuing credible answers in time to be used. Questioning is the ultimate method.

Questions are the quintessential tools. (Patton, 2010)

DE – uses a range of interpretive frameworks

• But applies evaluation logic and thinking in complex situations

• Enables people to engage in data based, ongoing evaluative sense-making and reasoned argument– ensuring the criteria (values) people use to decide

what’s ‘good’ and desirable are open to scrutiny and made explicit

– ensuring the decisions made about development / the ‘forks in the road’ are documented and systematically reflected upon in a critical and open minded way

Our experience: introducing DE to the uninitiated

• Small steps, getting people grounded, on the same page really matters

• The most simple of frameworks:– What, why, when, how, where and who

• What is happening and what has changed? • Why?• Who has been affected, Who is involved?• How did this happen?• Where is the situation / programme at now?• When do key people need to know what?• How do people feel about the change? Why?• etc

Or some slightly more evaluative questions:

– What’s being developed? (WHAT?)– What sense can we make of emerging issues,

evidence, data about this development? (WHAT’S EMERGING?)

– What is the value / importance /significance of what we are doing / achieving? What does it mean to us now and in the future? (SO WHAT?)

– What does this mean for how we should now act? What are our options? And into the future? (NOW WHAT)

Applying DE questionsDE Question Applied Example (HOP)

What? What’s being developed? What does / might ‘as Māori’ look like in the sport and recreation sector?

What’s emerging?

What sense can we make of emerging issues, evidence, data about this development?

What are other’s doing in the ‘as Māori space? What are the patterns of ideas and practice emerging from the provider communities about ‘as Māori’

So what? What’s the value / significance / importance of what we’re doing and achieving?

Whose values are we applying to our judgments? And what importance do different values place on what’s emerging?

Now what? What does this mean for how we should now act? Into the future?

How will be take our new understandings of ‘as Māori’ and apply these to the next phase of development?

Some useful interpretive frameworks:

• Appreciative Inquiry• Success Case Method• Most Significant Change• Systems approaches – with an emphasis on

perspectives, boundaries and interrelationships• Outcome mapping• Action research

See Handout 5

Example – ‘as Māori’ a developmental journey

The process• As Māori not prescribed in the provider selection process• Series of facilitated discussions

– 1 and 2 day provider hui (n=4)– individual providers

• Ongoing iteration– Cycle of feedback loops

What’s emerging• Shared understanding• Co-constructed But still emergent

See Handout 6

Emergent understanding of ‘as Māori’

• It’s something that is led by tikanga• In a Māori way, underpinned by Māori ways, values and

perspectives. • It may also be in Māori places… but not necessarily. • It may also be doing Māori activities… but not necessarily.• It’s not specific just to Māori contexts e.g. Māori people

participating as Māori in the wider world context.• There’s not going to be one answer• I just want some clear definitions of what non-Māori are

thinking of it

Co-construction of ‘as Māori’

• Te reo• Kanohi ki te kanohi• Guided by kawa and tikanga• Whānau, hapu, iwi and Māori • Uses Māori institutions e.g. marae, kohanga• Whanaungatanga• Traditional Māori sports• Spirit of us• Creating a feeling of belonging and empowerment

What’s the value of ‘as Māori’

• It’s our own self determination of how we do it, in ways that we want to do it.

• To do the things we want to do, in ways that we want to do them

• Opportunity to portray our uniqueness• Effective and better engagement achieved when

cultural values and aspects core and/or included– Traditional sports are just as valuable - in terms of

engaging our communities

Example – ‘as Māori’ a developmental journey• An ‘as Māori’ continuum with five dimensions emerging:

By Māori Refers to the extent of rangatiratanga control by Māori in the governance, management and delivery of an initiative

For Māori Emphasises the importance of whānau, hapu, iwi, and other Māori collectives bought together for kaupapa Māori purposes

With te reo me ona tikanga

Points to the importance of te reo me ona tikanga as central to the survival and affirmation of what it means to live ‘as Māori’

In/on places of cultural significance to Māori

From ancestral marae, awa, maunga, marae to more contemporary places founded on kaupapa Māori principles such as kōhanga reo, kura etc.

Through Those activities that can be distinguished as Māori e.g., nga taonga taakaro

AFTERNOON TEA

Some final thoughts, tips and questionsFAQs we have encountered:• How systematic is DE?• How does DE differ from Action Research?• Isn’t DE just a bunch of stories – when do you

get to outcomes? And measurement? And judgment?– What questions do you have about DE?

What we’ve learned a long the way• It’s a journey with others, this is not for the

sole trader, nor for the feint hearted!• As an evaluation team, we model the

reflection process togetherThis is hard thinking stuff, it doesn’t come easy

• Great rewards possibleThank goodness we chose and are able to use DE, it’s the right way to go, but it is hard work

Wrap up• One thing you learned today• One thing you will do as a result of today• One thing you will follow up on

ReferencesGamble J., A Developmental Evaluation Primer, J.W. McConnell Family

Foundation, 2008.

Patton, M. Q. Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. Guilford Press, June 2010.

Patton, M.Q. Utilization-Focused Evaluation, 4th ed., Sage, 2008.

Wehipeihana, N and McKegg K, Developmental Evaluation in an Indigenous Context, Reflections on the journey to date. Presentation to American Evaluation Conference, Orlando, Florida, 2009

Westley, Frances; Zimmerman, Brenda; Patton, Michael Q. Getting to Maybe: How the World Is Changed? Random House Canada, 2006