![Page 1: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/1.jpg)
1
Developmental Evaluation:
Systems Thinking and
Complexity Science
Michael Quinn Patton
MESI 2016
Michael Quinn Patton
MESI 2016
![Page 2: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/2.jpg)
2
Research on
Social Innovators’
Perceptions
of Evaluation
Michael Quinn Patton
MESI 2016
![Page 3: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/3.jpg)
Interpretive Frameworks
• May 2003 Harvard Business Review "The High Cost of Accuracy." Kathleen Sutcliffe and Klaus Weber.
They concluded that "the way senior executives interpret their business environment is more important for performance than how accurately they know their environment."
3Michael Quinn Patton
MESI 2016
![Page 4: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/4.jpg)
They further concluded that it is a waste of resources to spend a lot of money increasing the marginal accuracy of data available to senior executives compared to the value of enhancing their capacity to interpret whatever data they have.
Executives were more limited by a lack of capacity to make sense of data than by inadequate or inaccurate data.
4
In essence, they found that
interpretive capacity, or "mind-sets,"
distinguish high-performance more
than data quality and accuracy.Michael Quinn Patton
MESI 2016
![Page 5: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/5.jpg)
Ubuntu small group exercise
The spirit of interpersonal interaction that
pervades sub-Saharan Africa, ubuntu
emphasizes the idea that:
“A person is a person because of
other people.”
5Michael Quinn Patton
MESI 2016
![Page 6: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/6.jpg)
EMERGENCE
• Self-organizing group experiences and
outcomes
• Inter-relationships and interconnections
• Boundaries and levels of analysis
6Michael Quinn Patton
MESI 2016
![Page 7: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/7.jpg)
Emergence Example
Wilderness
Leadership
Program
7Michael Quinn Patton
MESI 2016
![Page 8: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/8.jpg)
Michael Quinn Patton
MESI 2016
Taking Emergence Seriously
• Beyond “unanticipated
consequences” to genuine
openness
8
![Page 9: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/9.jpg)
Uncertainty and Emergence
“No battle plan ever survives contact with
the enemy.” Field Marshall Helmuth
Carl Bernard von Moltke
“Everyone has a plan…until he gets hit.”
Former World Heavyweight
boxing champion, Mike Tyson
Tom Peters (1996) Liberation Management :
“READY. FIRE. AIM.”
Michael Quinn Patton
MESI 2016
9
![Page 10: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/10.jpg)
10
Adaptation and Learning
“Shoot bullets not cannonballs.”
Michael Quinn Patton
MESI 2016
![Page 11: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/11.jpg)
Original Primary Options
Formative
and
Summative
Evaluation
(Mid-term and End-of-Project Reviews)
11Michael Quinn Patton
MESI 2016
![Page 12: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/12.jpg)
Blandin Community
Leadership Program
12Michael Quinn Patton
MESI 2016
![Page 13: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/13.jpg)
Fundamental issue
Development
vs.
Improvement
13Michael Quinn Patton
MESI 2016
![Page 14: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/14.jpg)
Inspired by Jeff Conklin, cognexus.org
Time
Complex development situations are ones in which this…
14Michael Quinn Patton
MESI 2016
![Page 15: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/15.jpg)
15
And this…
Michael Quinn Patton
MESI 2016
![Page 16: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/16.jpg)
Turns out to be this…
Time16Michael Quinn Patton
MESI 2016
![Page 17: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/17.jpg)
17
…looks like this
Michael Quinn Patton
MESI 2016
![Page 18: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/18.jpg)
18
Formative evaluation
Summative evaluation
Developmental evaluationMichael Quinn Patton
MESI 2016
![Page 19: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/19.jpg)
Henry MintzbergStrategic
Leadership
Expert
19
Evaluation of
strategy
Implementaion
Michael Quinn Patton
MESI 2016
![Page 20: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/20.jpg)
Mintzberg on Strategy
Unrealized Strategy
Intended
Strategy
Deliberate
Strategy
Realized
Strategy
Emergent Strategy
20Michael Quinn Patton
MESI 2016
![Page 21: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/21.jpg)
21
Developmental Evaluation Defined
Purpose: Developmental evaluation (DE) informs and supports
innovative and adaptive development in
complex dynamic environments.
DE brings to innovation and adaptation the processes of
asking evaluative questions, applying evaluation logic, and
gathering and reporting evaluative data to support project,
program, product, and/or organizational development with
timely feedback.
Michael Quinn Patton
MESI 2016
![Page 22: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/22.jpg)
Key DE Characteristics
• Focus on development (versus improvement,
accountability or summative judgment)
• Takes place in complex dynamic environments
• Feedback is rapid (as real time as possible).
• The evaluator works collaboratively with social
innovators to conceptualize, design and test new
approaches in a long-term, on-going process
of adaptation, intentional change,
and development.
22Michael Quinn Patton
MESI 2016
![Page 23: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/23.jpg)
Key DE Characteristics
• The DE evaluator can be part of the
intervention team.
• The evaluator's primary functions are to
elucidate the innovation and adaptation
processes, track their implications and
results, and facilitate ongoing, real-time,
data-based decision-making in the
developmental process.
• DE becomes part of the intervention.
23Michael Quinn Patton
MESI 2016
![Page 24: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/24.jpg)
High Degree of Process Use
Process use refers to and is indicated by individual changes in thinking and behavior, and program or organizational changes in procedures and culture, that occur among those involved in evaluation as a result of the learning that occurs during the evaluation process. Evidence of process use is represented by the following kind of statement after an evaluation: "The impact on our program came not so much from the findings but from going through the thinking process that the evaluation required."
24Michael Quinn Patton
MESI 2016
![Page 25: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/25.jpg)
Small group exercise
Distinguishing
improvement
from
development
Each person describe a
project example that
distinguishes
an improvement from
a development
25Michael Quinn Patton
MESI 2016
![Page 26: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/26.jpg)
Other names
Real time evaluation
Emergent evaluation
Action evaluation
Adaptive evaluation
26Michael Quinn Patton
MESI 2016
![Page 27: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/27.jpg)
27
Development
Evaluation
Developmental
Evaluation
DD2 = Developmental evaluation used
for development evaluation
DD2
Michael Quinn Patton
MESI 2016
![Page 28: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/28.jpg)
28
Primary developmental evaluation purpose
Complex system challenges Implications
1. Ongoing development Being implemented in a complex and dynamic environment
No intention to become a fixed/standardised modelIdentifies effective principles
2. Adapting effective principles to a new context
Innovative initiativeDevelop ‘own’ version based on adaption of effective principles and knowledge
Top-down—general principles knowledge disseminatedBottom-up—sensitivity to context, experience, capabilities and prioritiesAdaptation vs Adoption
3. Developing a rapid response in turbulent crisis conditions, e.g.,natural resource orhumanitarian disaster
Existing initiatives and responses no longer effective as conditions change suddenly
Planning, execution and evaluation occur simultaneously
Five purposes of developmental
evaluation
Michael Quinn Patton
MESI 2016
![Page 29: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/29.jpg)
29
Primary developmental evaluation purpose
Complex system challenges Implications
4. Pre-formative development of potentially scalable innovative
Changing and dynamic situations require innovative solutions to worsening conditionsModel needs to be developed/does not exist
Models may move into formative and summative evaluation, others remain in developmental modeInform different potential scaling options
5. Major systems change and cross scale developmental evaluation
Disrupt existing systemTaking an innovation to scaleMajor systems change and changing scale will add levels of complexity, new uncertainties and disagreements
Models change as they are taken across time, space and to larger systemsAdaptive cross scale innovations assume complex, nonlinear dynamics—agility and responsivenessAdaptation -- Replication
Five purposes of developmental
evaluation
Michael Quinn Patton
MESI 2016
![Page 30: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/30.jpg)
Conditions that challenge traditional
model-testing evaluation
• High innovation
• Development
• High uncertainty
• Dynamic
• Emergent
• Systems Change
Adaptive
Management
and
Developmental
Evaluation
30Michael Quinn Patton
MESI 2016
![Page 31: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/31.jpg)
Complexity concepts & Evaluation
• Emergence: Self-organizing, Attractors
• Nonlinear: Small actions can have large
reactions. “The Butterfly Wings Metaphor
• Dynamical: Interactions within, between, and
among subsystems and parts within systems
can volatile, changing
• Getting to Maybe: Uncertainty; unpredictable;
uncontrollable; unanticipated consequences
• Coevolution: Process uses; interdependence
• Adaptation: Staff & Intended beneficiaries31Michael Quinn Patton
MESI 2016
![Page 32: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/32.jpg)
32
Adaptation and Learning
“Shoot bullets not cannonballs.”
Michael Quinn Patton
MESI 2016
![Page 33: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/33.jpg)
Complexity Concepts
• Nonlinearity: Small actions can
have large reactions. “The
Butterfly Wings Metaphor”
33Michael Quinn Patton
MESI 2016
![Page 34: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/34.jpg)
Damiano Soup Kitchen, Duluth, Minnesota
Michael Quinn Patton
MESI 2016
34
![Page 35: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/35.jpg)
Nonlinearity
• Vicious circle: From working to homeless
• Virtuous circle: Youth homeless center
staff story
35Michael Quinn Patton
MESI 2016
![Page 36: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/36.jpg)
Dynamical
• Youth homelessness
• Mental illness
• Chemical dependency
• Poverty
36
Journey-oriented
Michael Quinn Patton
MESI 2016
![Page 37: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/37.jpg)
Co-evolution
37
Bumblebees and the flowers they pollinate have coevolved so
that both have become dependent on each other for survival.
Michael Quinn Patton
MESI 2016
![Page 38: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/38.jpg)
Seeing Through A
Complexity Lens
38
DEFINING COMPLEXITY
Michael Quinn Patton
MESI 2016
![Page 39: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/39.jpg)
Getting to Maybe:
How the World Is
Changed?
Frances Westley,
Brenda Zimmerman,
Michael Q. Patton
Random House
Canada,2006
39Michael Quinn Patton
MESI 2016
![Page 40: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/40.jpg)
Conceptual Options
•Simple
•Complicated
•Complex
40Michael Quinn Patton
MESI 2016
![Page 41: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/41.jpg)
Situation Analysis
Matrix:
Mapping the Territory
Degree ofCertainty
Close to Far from
Clo
se t
o
41Michael Quinn Patton
MESI 2016
![Page 42: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/42.jpg)
Simple Space
CertaintyClose to Far from
Clo
se t
o SimplePlan, control
42Michael Quinn Patton
MESI 2016
![Page 43: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/43.jpg)
43
Following a Recipe A Rocket to the Moon Raising a ChildComplicated Complex
The recipe is essential
Recipes (best practices) are tested to assure replicability
Recipes produce standard products
Focus is on following the recipe (Fidelity evaluation)
The goal is certainty of same results every time
Simple METAPHORS
Michael Quinn Patton
MESI 2016
![Page 44: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/44.jpg)
44Michael Quinn Patton
MESI 2016
![Page 45: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/45.jpg)
Technically Complicated
CertaintyClose to Far from
Clo
se t
o
SimplePlan, control
Technically ComplicatedExperiment, coordinate expertise
45Michael Quinn Patton
MESI 2016
![Page 46: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/46.jpg)
Following a Recipe A Rocket to the Moon Raising a
Child• Formulae are critical and necessary
• Sending one rocket increases assurance that next will be ok
• High level of expertise in many specialized fields + coordination
• Rockets similar in critical ways
• High degree of certainty of outcome
Complicated Complex
The recipe is essential
Recipes are tested to assure replicability of later efforts
No particular expertise; knowing how to cook increases success
Recipes produce standard products
Certainty of same results every time
Simple
46Michael Quinn Patton
MESI 2016
![Page 47: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/47.jpg)
Socially Complicated
CertaintyClose to Far from
Clo
se t
o
SimplePlan, control
Technically ComplicatedExperiment, coordinate expertise
SociallyComplicatedBuild relationships, create common ground
47Michael Quinn Patton
MESI 2016
![Page 48: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/48.jpg)
48
Space Shuttle Disasters
• Challenger disaster
January 28, 1986• Columbia disaster
February 1, 2003
Michael Quinn Patton
MESI 2016
![Page 49: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/49.jpg)
Socially complicated
Implementing human rights agreements, like gender equity or outlawing child labor
Environmental Initiatives
Many different and competing stakeholders
Diverse vested interests
High stakes
49Michael Quinn Patton
MESI 2016
![Page 50: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/50.jpg)
Socially complicated
situations
pose the challenge
of coordinating and
integrating
many players
50Michael Quinn Patton
MESI 2016
![Page 51: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/51.jpg)
Know When Your Challenges Are In the
Zone of Complexity
CertaintyClose to Far from
Clo
se t
o
SimplePlan, control
Zone of Complexity
Technically ComplicatedExperiment, coordinate expertise
SociallyComplicatedBuild
relationships, create common ground
51Michael Quinn Patton
MESI 2016
![Page 52: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/52.jpg)
Following a Recipe A Rocket to the Moon Raising a
Child
Sending one rocket increases assurance that next will be ok
High level of expertise in many specialized fields + coordination
Rockets similar in critical ways
High degree of certainty of outcome
• Formulae have only a limited application
• Raising one child gives no assurance of success with the next
• Expertise can help but is not sufficient;
relationships are key
• Every child is unique
• Uncertainty of outcome remains
Complicated Complex
The recipe is essential
Recipes are tested to assure replicability of later efforts
No particular expertise; knowing how to cook increases success
Recipes produce standard products
Certainty of same results every time
Simple
52Michael Quinn Patton
MESI 2016
![Page 53: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/53.jpg)
Following a Recipe A Rocket to the Moon Raising a Child
• Formulae are critical and necessary
• Sending one rocket increases assurance that next will be ok
• High level of expertise in many specialized fields + coordination
• Separate into parts and then coordinate
• Rockets similar in critical ways
• High degree of certainty of outcome
• Formulae have only a limited application
• Raising one child gives no assurance of success with the next
• Expertise can help but is not sufficient; relationships are key
• Can’t separate parts from the whole
• Every child is unique
• Uncertainty of outcome remains
Complicated Complex
The recipe is essential
Recipes are tested to assure replicability of later efforts
No particular expertise; knowing how to cook increases success
Recipe notes the quantity and nature of “parts” needed
Recipes produce standard products
Certainty of same results every time
Simple
53Michael Quinn Patton
MESI 2016
![Page 54: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/54.jpg)
“A Leader's Framework for Decision
Making” by David J. Snowden and
Mary E. Boone, Harvard Business
Review,
November, 2007:
Wise executives tailor their approach to fit
the complexity of the circumstances
they face.
54Michael Quinn Patton
MESI 2016
![Page 55: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/55.jpg)
Wise evaluators tailor their approach
to fit the complexity of the
circumstances they face
55Michael Quinn Patton
MESI 2016
![Page 56: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/56.jpg)
Example
The McGill-McConnell Leadership
Program Example
Simple elements
Complicated elements
Complex elements
56Michael Quinn Patton
MESI 2016
![Page 57: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/57.jpg)
Simple outcomes
• Increase knowledge and skills of
participants
Evaluation: Pre-post data and
documentation of learning
57Michael Quinn Patton
MESI 2016
![Page 58: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/58.jpg)
Complicated Impacts
• Change participants’ organizations
Evaluation:
Case studies
of
organizational change
58Michael Quinn Patton
MESI 2016
![Page 59: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/59.jpg)
Complex Vision
• Infuse energy into the moribund not-
for-profit (voluntary) sector
• Make the sector more dynamic
• Create network of leaders who
actively engage in change
59Michael Quinn Patton
MESI 2016
![Page 60: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/60.jpg)
Evaluating the Complex
• Real time follow-up of network
connections and actions
• Follow-up is an intervention
• Rapid feedback of findings permits
infusion of resources in support of
emergent outcomes
60Michael Quinn Patton
MESI 2016
![Page 61: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/61.jpg)
Exercise
1. Identify a program or project at your table.
2. What elements are…
Simple
Complicated
Complex
61Michael Quinn Patton
MESI 2016
![Page 62: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/62.jpg)
62
Primary developmental evaluation purpose
Complex system challenges Implications
1. Ongoing development Being implemented in a complex and dynamic environment
No intention to become a fixed/standardised modelIdentifies effective principles
2. Adapting effective principles to a new context
Innovative initiativeDevelop ‘own’ version based on adaption of effective principles and knowledge
Top-down—general principles knowledge disseminatedBottom-up—sensitivity to context, experience, capabilities and prioritiesAdaptation vs Adoption
3. Developing a rapid response in turbulent major change context
Existing initiatives and responses no longer effective as conditions change suddenly
Planning, execution and evaluation occur simultaneously
Purposes of developmental evaluation
Michael Quinn Patton
MESI 2016
![Page 63: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/63.jpg)
Evidence-based Practice
Evaluation grew up in the “projects”
testing models under a theory of
change that pilot testing would lead to
proven models that could be
disseminated and taken to scale:
The search for best practices
and evidenced-based practices
63Michael Quinn Patton
MESI 2016
![Page 64: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/64.jpg)
Fundamental Issue:
How the World Is Changed
Top-down dissemination of
“proven models”
versus
Bottoms-up adaptive
management
64Michael Quinn Patton
MESI 2016
![Page 65: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/65.jpg)
Models vs. Principles
Identifying proven principles for
adaptive management
(bottoms-up approach)
versus
Identifying and disseminating
proven models
(top down approach)65Michael Quinn Patton
MESI 2016
![Page 66: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/66.jpg)
Some premises:• Evaluation is part of initial program design,
including conceptualizing the theory of
change
• Evaluator’s role is to help users clarify their
purpose, hoped-for results, and change
model.
• Evaluators can/should offer conceptual and
methodological options.
• Evaluators can help by questioning
assumptions.
• Evaluators can play a key role in facilitating
evaluative thinking all along the way.66Michael Quinn Patton
MESI 2016
![Page 67: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/67.jpg)
Three ways of conceptualizing
and mapping theories of change
Linear Newtonian causality
Interdependent systems relationships
Complex nonlinear dynamics
67Michael Quinn Patton
MESI 2016
![Page 68: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/68.jpg)
Linear Logic Model
INPUTS (people, materials)
ACTIVITIES (processes)
OUTPUTS
OUTCOMES
CHANGES IN PEOPLES LIVES
IMPACTS
CHANGES IN COMMUNITIES
68Michael Quinn Patton
MESI 2016
![Page 69: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/69.jpg)
Output / Product
Essential Attributes
Attributes required to meet of exceed
customer needs:
"Do the Right Thing"
Efficacy
Appropriate
Characteristics to meet or exceed
customer wants and expectations
of excellence
"Do the Right Thing Well":
Efficiency
Dignity and Respect
Effectiveness
Timeliness
Reduce Waste
Safety
Continuity
Availability
What inputs need to go into the
process to make the product
that produces the desired
result?
What steps need to be taken to
create the product that achieves
the desired result?
What features / characteristics should the product
have?
Systems Logic Model
Customer
Outcomes
&
Satisfaction
Measure
Effectiveness
Measure
Satisfaction
Inform
Improvement
needs
Effect
Inputs
Staff Resources
Financial resources
Internal Standards
External Requirements
and Information
Equipment/Materials
Key Processes & Functions
Inputs organized and utilized
Procedures
Steps
Key processes
Measure Variability
Assess Process Control
Assess fidelity to planned
procedures
Assess impact of variation
Evaluate opportunity to raise the
bar
Cause
Feedback into process
What is the desired result?
What should customer
experience?
Planning
Implementation
Structure Process OUTCOMES
Feedback
69Michael Quinn Patton
MESI 2016
![Page 70: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/70.jpg)
Division workplace that:
· Offers a healthy work environment
· Recognizes excellence
· Provides quality training and management
· Includes effective systems, procedures, and communication (Goal 5)
Increased adoption, reach, implementation, and sustainability of recommended public health strategies to achieve strategic plan goals:
· Prevent risk factors for heart disease and stroke (Goal 1)
· Increase detection and treatment of risk factors (Goal 2)
· Increase early identification and treatment of heart attacks and strokes (Goal 3)
· Prevent recurring cardiovascular events (Goal 4)
Division for Heart Disease and Stroke Prevention Evaluation Planning Logic Model
Internal
Workforce that is:
· Diverse
· Skilled
Resources that are:
· Available
· Timely
Division Leadership that provides sufficient:
· Infrastructure
· Policies
· Strategic Planning
Leadership
Disparities
Surveillance
Research
Evaluation
Program
Translation
and dissemination
of the current
knowledge base,
and identification
of ways to improve
that knowledge
base
Effective:
· Management
· Coordination
· Staff development
Enhanced
competency of public
health workforce
Enhanced ability
of programs to apply
findings to improve
public health
Enhanced external
application of Division
goals and strategies
Increased advocacy
and “activated
constituency”
Engaged network of
states and partners
Enhanced integration
among chronic
disease programs
Increased focus
on heart disease
and stroke
prevention efforts
by states and
partners,
especially with
regard to
disparities
Policy
Increased
knowledge of
signs and
symptoms
Improved
emergency
response
Improved quality
of care
Reduced risk
factors
Reduced
economic impact
of heart disease
and stroke
Eliminated
preventable
strokes and risks
Reduced levels of
disparities in heart
disease and stroke
Reduced
morbidity and
mortality of heart
disease and stroke
External
Planning Activities Translation,
Dissemination Adoption, Practice, Sustainability Impact
WHAT WHYHOW
Communication
Collaboration
70Michael Quinn Patton
MESI 2016
![Page 71: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/71.jpg)
71
Primary developmental evaluation purpose
Complex system challenges Implications
4. Pre-formative development of potentially scalable innovative
Changing and dynamic situations require innovative solutions to worsening conditionsModel needs to be developed/does not exist
Models may move into formative and summative evaluation, others remain in developmental modeInform different potential scaling options
5. Major systems change and cross scale developmental evaluation
Disrupt existing systemTaking an innovation to scaleMajor systems change and changing scale will add levels of complexity, new uncertainties and disagreements
Models change as they are taken across time, space and to larger systemsAdaptive cross scale innovations assume complex, nonlinear dynamics—agility and responsivenessAdaptation -- Replication
Purposes of developmental evaluation
Michael Quinn Patton
MESI 2016
![Page 72: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/72.jpg)
Systems
• Parts are interdependent such a
change in one part changes all parts
• The whole is greater than the sum of
the parts
• Focus on interconnected relationships
• Systems are made up of sub-systems
and function within larger systems
72Michael Quinn Patton
MESI 2016
![Page 73: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/73.jpg)
Systems Concepts in Evaluation –
An Expert Anthology. 2006.
Bob Williams and Iraj Imam AEA Monograph,
EdgePress/AEA Point Reyes CA.
73Michael Quinn Patton
MESI 2016
![Page 74: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/74.jpg)
Understanding the
Elephant
from
a Systems Perspective
74Michael Quinn Patton
MESI 2016
![Page 75: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/75.jpg)
75Michael Quinn Patton
MESI 2016
![Page 76: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/76.jpg)
The relationship between what
goes in and what comes out
What
conceptual
framework
informs
front-end
evaluation
work?76Michael Quinn Patton
MESI 2016
![Page 77: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/77.jpg)
Teen Pregnancy Program
Example
77Michael Quinn Patton
MESI 2016
![Page 78: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/78.jpg)
Logic Model for Pregnant Teens Program1. Program reaches out to pregnant teens
2. Pregnant teens enter and attend the program (participation)
3. Teens learn prenatal nutrition and self-care (increased
knowledge)
4. Teens develop commitment to take care of themselves
and their babies (attitude change)
5. Teens adopt healthy behaviors: no smoking, no drinking,
attend prenatal clinic, eat properly (behavior change)
6. Teens have healthy babies (desired outcome)
78Michael Quinn Patton
MESI 2016
![Page 79: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/79.jpg)
Systems web showing possible influence
linkages to a pregnant teenager
Teachers/
other
adults
Young
pregnant
woman's
attitudes &
behaviors
Her
parents &
other family
members
Child's
father &
peers
Prenatal
program
staff
Her peer
group
79Michael Quinn Patton
MESI 2016
![Page 80: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/80.jpg)
Program systems web showing possible institutional
influences affecting pregnant teenagers:
SCHOOL
SYSTEMYoung
pregnant
women's
attitudes &
behaviors
Prenatal
Clinic and
Hospital
Outreach
Church
Prenatal
program
Other
community-
based youth
programs
Other Systems
-- welfare
-- legal
-- nutrition
programs
-- transportation
-- child
protection
-- media messages
Context factors
-- politics
-- economic
incentives
-- social norms
-- culture
-- music
Youth
Culture
80Michael Quinn Patton
MESI 2016
![Page 81: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/81.jpg)
SYSTEMS CHANGE:
Interrelationships, Boundaries,
Perceptions, Networks
81Michael Quinn Patton
MESI 2016
![Page 82: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/82.jpg)
HIV/AIDS Example• Hits every system: health, family,
social, religious, economic, political,
community, international
• Requires multiple interventions on
multiple fronts in all subsystems
simultaneously
• Resulting reactions, interactions,
consequences dynamic, unpredictable,
emergent, and ever changing
82Michael Quinn Patton
MESI 2016
![Page 83: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/83.jpg)
Challenges:
Situation Recognition
and
Appropriate Evaluation
Designs
83Michael Quinn Patton
MESI 2016
![Page 84: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/84.jpg)
The nature of
EXPERTISE
84Michael Quinn Patton
MESI 2016
![Page 85: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/85.jpg)
Complex Situations
• Highly emergent (difficult to
plan and predict)
• Highly dynamic, rapidly
changing
• Relationships are
interdependent and non-linear
rather than simple and linear
(cause-effect)85Michael Quinn Patton
MESI 2016
![Page 86: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/86.jpg)
Some Particularly Appropriate
Applications of DE:
Examples of Innovative Arenas
• Social Movements and networks
• Advocacy Evaluation
• Large-scale, cross-sector, collaborative
initiatives
• R & D in public health, technological
innovation, science
• Prevention86Michael Quinn Patton
MESI 2016
![Page 87: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/87.jpg)
Beyond
just
Summative
and
Formative
87Michael Quinn Patton
MESI 2016
![Page 88: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/88.jpg)
Beyond
Static Accountability
Models
88Michael Quinn Patton
MESI 2016
![Page 89: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/89.jpg)
![Page 90: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/90.jpg)
![Page 91: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/91.jpg)
91
Coherence Through Shared Principles
Principles-focused
Evaluation
![Page 92: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/92.jpg)
Evaluation Principles
AEA guiding principles
Participatory evaluation principles
Utilization-Focused Evaluation principles
Empowerment Evaluation principles
Culturally competence evaluation principles
Indigenous peoples’ research and evaluation principles
92
![Page 93: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/93.jpg)
U-FE Principle
Focus on Intended Use
by
Intended Users
93
![Page 94: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/94.jpg)
Principles
“I am a man of fixed and unbending
principles, the first of which is to be flexible
at all times.”
U.S. Senator
Everett Dirksen
94
![Page 95: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/95.jpg)
"Principles are
like prayers.
Noble, of course,
but awkward at a
party."
Lady Crawley, the
Dowager Countess,
Downton Abbey
95
![Page 96: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/96.jpg)
“It is not the responsibility
of knights errant to
discover whether the
afflicted, the enchained
and the oppressed whom
they encounter on the road
are reduced to these
circumstances and suffer
this distress for their vices,
or for their virtues: the
knight's sole responsibility
is to succour them as
people in need, having
eyes only for their
sufferings, not for their
misdeeds.”
― Miguel de Cervantes
Saavedra, Don Quixote
96
![Page 97: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/97.jpg)
RECIPES vs PRINCIPLES
REPLICATION RECIPE
Add 1/4 teaspoon
of salt
ADAPTIVE PRINCIPLESeason to taste &
situation
97
![Page 98: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/98.jpg)
Managing email
98
![Page 99: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/99.jpg)
Exercise
Rule:
30 minutes of aerobic exercise each day
Principle:
Exercise regularly at a level that supports health and is sustainable given your health, life style, age, and capacity.
99
![Page 100: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/100.jpg)
100
![Page 101: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/101.jpg)
101
![Page 102: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/102.jpg)
Principles
• Provide direction but not detailed prescription
• Are grounded in values about what matters
• Are based on evidence about how to be effective
• Must be interpreted and applied contextually,
• Require judgment in application
• Inform choices at forks in the road
• Are the rudder for navigating complex dynamic systems
• Point to outcomes and impacts
• Can be evaluated for both process (implementation) -- and results
102
![Page 103: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/103.jpg)
DE Principles
1. Developmental purpose
2. Evaluation rigor
3. Utilization focus
4. Innovation niche
5. Complexity perspective
6. Systems thinking
7. Co-creation
8. Timely feedback103
![Page 104: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/104.jpg)
104
![Page 105: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/105.jpg)
What is Essential in
Developmental Evaluation?
On Integrity, Fidelity, Adultery,
Abstinence, Impotence, Long-
term Commitment, Integrity,
and Sensitivity in Implementing
Evaluation Models
![Page 106: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/106.jpg)
Creative Challenge
Situational adaptability:
Contingency-based evaluation
Appropriateness
--Using standard forms of evaluation
and
-- Going beyond standard forms when
appropriate and useful
106Michael Quinn Patton
MESI 2016
![Page 107: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/107.jpg)
Paradigms and Lenses
• The importance of interpretive
frameworks
• Complexity as an interpretive
framework
107Michael Quinn Patton
MESI 2016
![Page 108: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/108.jpg)
VALUES CONTRASTS
Traditional evaluations…
1. Testing models
Complexity-based, Developmental Evaluation…
1. Supporting innovation and adaptation
108Michael Quinn Patton
MESI 2016
![Page 109: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/109.jpg)
Traditional
Evaluation…
2. Render
definitive
judgments of
success or
failure:
Does the
program work?
Developmental
Evaluation…
2. Rendering nuanced,
disaggregated
feedback & generate
learnings for
adaptation &
development:
What works for whom
in what ways under
what conditions?109Michael Quinn Patton
MESI 2016
![Page 110: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/110.jpg)
Traditional DevelopmentalEvaluation… Evaluation…
3. INDEPENDENCE:
Evaluator external, independent, objective
3. RELATIONSHIP-FOCUSED, COLLABORATIVE
Evaluator a facilitator and learning coach bringing evaluative thinking to the table, supportive of innovator’s vision
110Michael Quinn Patton
MESI 2016
![Page 111: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/111.jpg)
Traditional Developmental
Evaluation… Evaluation…
4. CONTROL:
Evaluator determines the design based on the evaluator’s perspective about what is important. The evaluator controls the evaluation.
4. OPENNESS &
NATURALISTIC
INQUIRY
Evaluator goes
with the flow,
watches for
what emerges
111Michael Quinn Patton
MESI 2016
![Page 112: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/112.jpg)
Traditional Developmental
Evaluation… Evaluation…
5. CERTAINTY:
Predetermined
outcomes
Fix the design
upfront
Predetermind
indicators
Fixed
questions
5. FLEXIBILITY
Emergent
outcomes
Flexible design
Emergent
indicators
Dynamic
questions
112
Michael Quinn Patton AEA 2011 112
Michael Quinn Patton
MESI 2016
![Page 113: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/113.jpg)
Traditional Developmental
Evaluation… Evaluation…
6. Linear cause-
effect thinking
and logic
models
6. Systems and
complexity thinking
with attention to
dynamics,
permeable
boundaries,
interdependencies,
and emergent
interconnections113Michael Quinn Patton
MESI 2016
![Page 114: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/114.jpg)
Traditional Developmental
Evaluation… Evaluation…
7. Value top-down change based on generalizable findings across time & space
• High fidelity, prescriptive “best practices” based on summative
evaluation.
7. Value bottom-up
principles that provide
direction but have to
be adapted to context
• Aim to produce
context-specific
understandings that
inform ongoing
innovation and
adaptation.
114Michael Quinn Patton
MESI 2016
![Page 115: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/115.jpg)
Traditional Developmental
Evaluation… Evaluation…
8. Accountability
focused on and
directed to
external
authorities and
funders.
8. Accountability
centered on the
innovators’ deep
sense of
fundamental
values and
commitments –
and learning as
accountability115Michael Quinn Patton
MESI 2016
![Page 116: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/116.jpg)
9. Being outside
the action,
above the fray
9. Being part of
the action,
engaged in the
fray
Traditional Developmental
Evaluation… Evaluation…
116
Michael Quinn Patton
MESI 2016
![Page 117: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/117.jpg)
Traditional Developmental
Evaluation… Evaluation…
10. TRUTH:
Speaking truth to
power
10.PERSPECTIVES
Facilitating dialogue
and engagement
with complexity
and shifting
understandings
117Michael Quinn Patton
MESI 2016
![Page 118: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/118.jpg)
118Michael Quinn Patton
MESI 2016
![Page 119: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/119.jpg)
119
Website sample chapter:
http://www.guilford.com/excer
pts/patton.pdf
website for the book:
http://www.guilford.com/cgi-
bin/cartscript.cgi?page=pr/patt
on.htm&dir=research/res_eval
&cart_id=824067.29797
Michael Quinn Patton
MESI 2016
![Page 120: Systems Thinking and Complexity Science...Systems Thinking and Complexity Science Michael Quinn Patton MESI 2016 Michael Quinn Patton MESI 2016 2 Research on Social Innovators’ Perceptions](https://reader033.vdocuments.net/reader033/viewer/2022060307/5f09c9827e708231d4287dc3/html5/thumbnails/120.jpg)
ReferencesDevelopmental Evaluation: Applying Complexity
Concepts to Enhance Innovation and Use.
Guilford Press, 2011.
Utilization-Focused Evaluation, 4th ed.,
Michael Quinn Patton, Sage, 2008.
Essentials of Utilization-Focused Evaluation,
Michael Quinn Patton, Sage, 2012
120Michael Quinn Patton
MESI 2016