Download - Developmental Evaluation on the Ground
![Page 1: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/1.jpg)
Developmental Evaluation on the
Ground
Brigitte Scott, Ph.D.
Evaluation and Research Specialist
Military Families Learning Network
![Page 2: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/2.jpg)
Military Families Learning Network
Personal Finance
Child Care
Family Advocacy
Network Literacy
Personal Finance
Family Development
Network Literacy
Community Capacity
Building
Nutrition & Wellness
Transition Support
Early Intervention
Special Needs
Life Span Special
Needs
Military Caregiving
![Page 3: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/3.jpg)
![Page 4: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/4.jpg)
MQP: DE Defined
“DE supports innovation development to guide
adaptation to emergent and dynamic realities in
complex environments.” —MQP, p. 1
Goal: support project, program, product, and/or
organizational development with timely
feedback
![Page 5: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/5.jpg)
MQP: Core DE Question
What is getting developed and what are the
implications of what gets developed?
Complexity theory of change:
Bring people together who are knowledgeable and
committed and they will self-organize, take
action, and work together to create movement,
innovation, and change.
![Page 6: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/6.jpg)
MQP: Key Characteristics of DE
![Page 7: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/7.jpg)
Focus on development
![Page 8: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/8.jpg)
Focus on developmentMQP
• NOT the same as continuous improvement (formative eval or accountability)
– Quality improvement is helping programs meet standards that have been set.
Development is when people are changing what they are doing, and the very nature of the standards are also changing. Standards are adapting to changing conditions.
![Page 9: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/9.jpg)
Complex and dynamic environment
![Page 10: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/10.jpg)
Complex and dynamic environmentMQP
Complex environment = lack of
central control
What to do to solve problems is
uncertain and there can be conflict
about how to proceed
![Page 11: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/11.jpg)
Real time feedback
![Page 12: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/12.jpg)
DE becomes part of intervention
![Page 13: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/13.jpg)
Developmental Evaluator
![Page 14: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/14.jpg)
Developmental EvaluatorMQP
• Works collaboratively with innovators to
conceptualize, design, and test new approaches
in a long-term, on-going process of adaptation,
intentional change, and development.
• Ask eval questions
• Apply eval logic
• Gather and report eval data
![Page 15: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/15.jpg)
Developmental EvaluatorMQP
Primary Functions:
Elucidate the innovation and adaptation
processes
Track implications and results
Facilitate data-based decision-making
![Page 16: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/16.jpg)
“There are no best practices. ‘Best’ is
entirely context-free.”—MQP, AEA 2014
![Page 17: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/17.jpg)
Evaluation for MFLN
• Ongoing development
• Innovation in learning
– Internal practice
– External product
![Page 18: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/18.jpg)
DE for MFLN: Why It Fits
![Page 19: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/19.jpg)
Program development
![Page 20: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/20.jpg)
Cooperative agreement environment
+
eXtension Initiative
![Page 21: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/21.jpg)
Cooperative Agreement
“An opportunity to provide responsive and timely educational programming through being actively nimble, flexible, innovative, and creative in true partnership with our funders as they identify organizational priorities and ask us to engage in the construction of appropriate and necessary deliverables that meet the on-time needs of the target audience. The MFLN and funding partners are seen as true and honest equals in expertise, providing valued and accepted feedback bi-directionally. This often results in surpassing expectations for success in delivery.”
—Kyle Kostelecky, National Program Director, MFLN
![Page 22: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/22.jpg)
Constant evaluative feedback
![Page 23: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/23.jpg)
Constant evaluative feedback and
discussion
• Monthly reports (internal, DoD)
• Quarterly reports (NIFA)
• Annual reports (internal, DoD, NIFA)
• Webinar evaluation reports
• Quarterly webinar reports
• Weekly leadership team meetings
• Social Media Specialists meetings
• Net Lit support meetings
![Page 24: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/24.jpg)
Evaluation used for decision-making
![Page 25: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/25.jpg)
Evaluator is on leadership team
![Page 26: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/26.jpg)
DE for MFLN: What It Looks Like
Continuous adaptation
![Page 27: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/27.jpg)
DE for MFLN: What It Looks Like
Continuous adaptationSocial, political, economic, technological, and demographic
patterns
![Page 28: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/28.jpg)
DE for MFLN: What It Looks Like
Continuous adaptationSocial, political, economic, technological, and demographic
patterns
Emergent developments/needs of our target audiences
![Page 29: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/29.jpg)
DE for MFLN: What It Looks Like
Continuous adaptationSocial, political, economic, technological, and demographic
patterns
Emergent developments/needs of our target audiences
Cooperative agreement environment
![Page 30: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/30.jpg)
DE for MFLN: What It Looks Like
Continuous adaptationSocial, political, economic, technological, and demographic
patterns
Emergent developments/needs of our target audiences
Cooperative agreement environment
eXtension
![Page 31: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/31.jpg)
DE for MFLN: What It Looks Like
Continuous adaptationSocial, political, economic, technological, and demographic
patterns
Emergent developments/needs of our target audiences
Cooperative agreement environment
eXtension
DoD
![Page 32: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/32.jpg)
DE for MFLN: What It Looks Like
(not this. . . .)
![Page 33: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/33.jpg)
DE for MFLN: What It Looks Like
(but this. . . .)
![Page 34: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/34.jpg)
DE for MFLN: What It Looks Like
Inductive
![Page 35: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/35.jpg)
DE for MFLN: What It Looks Like
InductiveChaotic and messy
![Page 36: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/36.jpg)
DE for MFLN: What It Looks Like
InductiveChaotic and messy
Embrace forks in the road
![Page 37: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/37.jpg)
DE for MFLN: What It Looks Like
InductiveChaotic and messy
Embrace forks in the road
Developmental moments
Track what’s going on
Understand what’s going on
Adapt to what’s going on
![Page 38: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/38.jpg)
DE for MFLN: What It Looks Like
Everyone is an
evaluator
![Page 39: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/39.jpg)
Inspired by Jeff Conklin, cognexus.org
Time
Complex development situations are ones in which this…
39Michael Quinn Patton
AEA eStudy webinar 2014
![Page 40: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/40.jpg)
Michael Quinn Patton
AEA eStudy webinar 201440
And this…
![Page 41: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/41.jpg)
Turns out to be this…
Time41
Michael Quinn Patton
AEA eStudy webinar 2014
![Page 42: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/42.jpg)
Michael Quinn Patton
AEA eStudy webinar 201442
…looks like this
![Page 43: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/43.jpg)
DE Successes for MFLN
![Page 44: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/44.jpg)
DE Successes for MFLN
• Very responsive to our partners’ programmatic
requests and innovations
![Page 45: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/45.jpg)
DE Successes for MFLN
• Very responsive to our partners’ programmatic
requests and innovations
• Doubling in size in 2015 with competitive award
– New CAs and new opportunities for innovation in
programming
![Page 46: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/46.jpg)
DE Successes for MFLN
• Very responsive to our partners’ programmatic
requests and innovations
• Doubling in size in 2015 with competitive award
– New CAs and new opportunities for innovation in
programming
• Programming
![Page 47: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/47.jpg)
DE Successes for MFLN
• Very responsive to our partners’ programmatic requests and innovations
• Doubling in size in 2015 with competitive award
– New CAs and new opportunities for innovation in programming
• Programming
• “Walk the talk”: model use of social media, collaborative learning, personal learning networks
![Page 48: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/48.jpg)
DE Challenges for MFLN
![Page 49: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/49.jpg)
DE Challenges for MFLN
• Innovation is challenging, slow, and . . . contentious
![Page 50: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/50.jpg)
DE Challenges for MFLN
• Innovation is challenging, slow, and . . . contentious
• Moving from program improvement mode to redesign
based on evaluation findings
![Page 51: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/51.jpg)
DE Challenges for MFLN
• Innovation is challenging, slow, and . . . contentious
• Moving from program improvement mode to redesign
based on evaluation findings
• IRB challenges
![Page 52: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/52.jpg)
DE Challenges for MFLN
• Innovation is challenging, slow, and . . . contentious
• Moving from program improvement mode to redesign
based on evaluation findings
• IRB challenges
• “In your face” evaluation
![Page 53: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/53.jpg)
DE Challenges for MFLN
• Innovation is challenging, slow, and . . . contentious
• Moving from program improvement mode to redesign
based on evaluation findings
• IRB challenges
• “In your face” evaluation
• Daily, evaluative thinking
![Page 54: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/54.jpg)
DE Challenges for MFLN
• Innovation is challenging, slow, and . . . contentious
• Moving from program improvement mode to redesign
based on evaluation findings
• IRB challenges
• “In your face” evaluation
• Daily, evaluative thinking
• Platform delivery limitations
![Page 55: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/55.jpg)
DE Challenges for MFLN
• Innovation is challenging, slow, and . . . contentious
• Moving from program improvement mode to redesign
based on evaluation findings
• IRB challenges
• “In your face” evaluation
• Daily, evaluative thinking
• Platform delivery limitations
• Military culture
![Page 56: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/56.jpg)
DE Challenges for MFLN
• Innovation is challenging, slow, and . . . contentious
• Moving from program improvement mode to redesign
based on evaluation findings
• IRB challenges
• “In your face” evaluation
• Daily, evaluative thinking
• Platform delivery limitations
• Military culture
• Doubling in size in January 2015
![Page 57: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/57.jpg)
![Page 58: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/58.jpg)
DE Features for MFLN
• Reports
• Leadership
• Reflective discussion groups
• Transparency
• Collaboration
![Page 59: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/59.jpg)
DE Features for MFLN
![Page 60: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/60.jpg)
Patton, Michael Quinn. (2014).
“Intermediate Developmental Evaluation.”
AEA eStudy Webinar.
Patton, Michael Quinn. (2011). Developmental
Evaluation: Applying Complexity Concepts to
Enhance Innovation and Use. Guilford: New
York.
![Page 61: Developmental Evaluation on the Ground](https://reader031.vdocuments.net/reader031/viewer/2022032422/55a985ad1a28ab2a0a8b46ab/html5/thumbnails/61.jpg)
Brigitte Scott
@4ed_eval
www.blogs.extension.org/militaryfamilies
Icons made by Freepik:
http://www.flaticon.com. Flaticon is
licensed by Creative Commons 3.0.