![Page 1: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/1.jpg)
Improvement Science 101 Breakout Set D
Thursday, April 5, 2018 Brandon Bennett, Principal Advisor, Improvement Science Consulting & Fellow, Carnegie Foundation
Alicia Grunow, Co-Founder, Improvement Collective & Senior Fellow, Carnegie Foundation
Amanda Meyer, Director of Improvement, CORE Districts
![Page 2: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/2.jpg)
Individual Reflection
What is one inequitable outcome in your school or community or our field that keeps you up at night?
2
![Page 3: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/3.jpg)
Change begins with...
3
Dissatisfaction with the status quo and a recognition that existent outcomes are either not desirable or just not good enough
VISION
CURRENT REALITY
GAP
![Page 4: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/4.jpg)
So what do we typically do?
Innovate Roll out
Say it louder
Train Superhero
![Page 5: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/5.jpg)
Improvement Science
A disciplined approach to making changes that draws on the efforts of everyone to collectively learn their way into stronger system performance and better outcomes for students.
Theory of Management
Skills and Tools
Mindsets
“Quality Improvement”
![Page 6: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/6.jpg)
Organize as networks
Learn through disciplined
inquiry
Embrace measurement
Attend to variability
See the system
Be problem-focused and user-centered
CORE PRINCIPLES OF IMPROVEMENT SCIENCE
6
![Page 7: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/7.jpg)
Step 1: Pick a number
from 3 to 9
Step 2: Multiply your number by 9
Step 3: Add 12 to the
number from step 2
Step 7: Write down the
name of a country that begins with
your letter
Step 4: Add your 2
digits together
Step 5: Divide # from step 4
by 3 to get a 1 digit number
Step 6: Convert your number to a
letter: 1=A 2=B 3=C 4=D 5=E 6=F 7=G 8=H 9 = I
Step 8: Go to the next letter: A to B,
B to C, C to D, etc.
Step 9: Write down the name
of an animal (not bird, fish, or insect) that begins with your
letter from Step 8
Step 10: Write down the color of
your animal
Do you have a 2-digit Number?
NO
YES
Output:
Color____________
Animal___________
Country__________
![Page 8: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/8.jpg)
Spread and Scale
Understand the Problem and the
System that Produces It
Test and Build Evidence
Generate Ideas for Change
Focus Collective Efforts
Improvement Science Cheat Sheet
8
![Page 9: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/9.jpg)
Case: College-going Rates
![Page 10: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/10.jpg)
Spread and Scale
Understand the Problem and the
System that Produces It
Test and Build Evidence
Generate Ideas for Change
Focus Collective Efforts
Improvement Science Cheat Sheet
10
![Page 11: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/11.jpg)
Tool: Process Map
11
What is it? A graphical representation of the steps that come together to produce a particular outcome Why is it useful? ▪ Processes are a basic building block of systems. ▪ Creates a shared understanding of current practice
that can then be analyzed and improved.
![Page 12: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/12.jpg)
Triad Activity: Let’s map!
Beginning Boundary: ? Ending Boundary: Complete college applications submitted
College Application Process Map
Start End
Create a high-level process map consisting of at least 5 steps in the college application process.
![Page 13: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/13.jpg)
Register for 9th grade
classes
Take college entrance
exam
Write college essay
Gather supplemental
materials
Complete FAFSA
Submit complete
application
Cur
rent
Pro
cess
Select colleges to
apply to
One possible map of the process
![Page 14: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/14.jpg)
14
-Student does not register for required courses
Failu
re M
odes
-Student scores low - Student does not retake exam after low score
-Student writes low-quality essay -Student does not revise initial essay
-Student’s family financial information incomplete
Cur
rent
Pro
cess
-Student selects colleges that are poor match
Register for 9th grade
classes
Take college entrance
exam
Write college essay
Gather supplemental
materials
Complete FAFSA
Submit complete
application
Select colleges to
apply to
Cha
nge
Idea
s
![Page 15: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/15.jpg)
Ways to “See the System”
1. Empathy Exercises: Engage users to understand the system from their point of view.
2. Leverage Data: Collect & analyze data to identify important causes & understand variation in performance across the system.
3. Map the Current System: Create a shared picture of “what is.”
15
![Page 16: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/16.jpg)
![Page 17: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/17.jpg)
Spread and Scale
Understand the Problem and the
System that Produces It
Test and Build Evidence
Generate Ideas for Change
Focus Collective Efforts
Improvement Science Cheat Sheet
17
![Page 18: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/18.jpg)
Model for Improvement @2009 API
PLAN
DO STUDY
ACT
An “Engine” for Learning
What specifically are we trying to accomplish?
What change(s) might we introduce and why?
How will we know if a change is an improvement?
![Page 19: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/19.jpg)
Model for Improvement @2009 API
PLAN
DO STUDY
ACT
An “Engine” for Learning
What specifically are we trying to accomplish?
What change(s) might we introduce and why?
How will we know if a change is an improvement?
What change(s) might we introduce and why?
How will we know if a change is an improvement?
AIM STATEMENT
![Page 20: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/20.jpg)
▪ What will be improved? (clear operational definitions) ▪ How much? (measureable, specific, numerical goals) ▪ By when? (time frame) ▪ For what/whom? (target population/setting or
system/process)
20
What specifically are we trying to accomplish?
Crafting an Aim Statement
![Page 21: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/21.jpg)
Examples of Aim Statements
(A)By June 2018, increase the percentage of district students who attend a post-secondary learning institution from 45% to 65%.
(A)By June 2018, increase from 35% to 80% the
percentage of low-income students in the district who complete the FAFSA.
21
![Page 22: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/22.jpg)
Triads: Give it a try!
Imagine your triad is a district team thinking about the college-going disparities between your students. How would you revise this draft aim statement? Close the gap in college-going rates in our district. It should include: ▪ What will be improved? (clear operational definitions) ▪ How much? (measureable, specific, numerical goals) ▪ By when? (time frame) ▪ For what/whom? (target population/setting or system/process)
22
![Page 23: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/23.jpg)
Evolving Aim Statements
Theories & Ideas
Target 2:
Target 3:
AIM 1 around a specific lever
Target 4:
BIG-DOT GOAL AIM
![Page 24: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/24.jpg)
Spread and Scale
Understand the Problem and the
System that Produces it
Test and Build Evidence
Generate ideas for change
Focus collective efforts
Probably wrong, definitely incomplete
Improvement Science Cheat
Sheet
24
![Page 25: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/25.jpg)
Model for Improvement @2009 API
PLAN
DO STUDY
ACT
An “Engine” for Learning
What specifically are we trying to accomplish?
What change(s) might we introduce and why?
How will we know if a change is an improvement?
How will we know if a change is an improvement?
What specifically are we trying to accomplish?
CHANGE IDEAS
![Page 26: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/26.jpg)
Tool: (Simplified) Failure Mode Effects Analysis
26
What is it? A systematic method to identify process problems/ breakdowns that may result in the inability to achieve desired outcomes consistently. Why is it useful? Helps to identify where to target changes in order to change outcomes.
![Page 27: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/27.jpg)
Where Do Change Ideas Come From?
1. Research knowledge: What does academic literature have to say about solving this problem?
2. Practice knowledge: What have other organizations in the field done to solve this problem?
3. Design/Creative Thinking: What new solutions might we design to address this problem?
4. Analysis of the problem: What does our analysis of the problem indicate may be a helpful solution?
![Page 28: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/28.jpg)
Spread and Scale
Understand the Problem and the
System that Produces it
Test and Build Evidence
Generate ideas for change
Focus collective efforts
Improvement Science Cheat
Sheet
28
![Page 29: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/29.jpg)
Model for Improvement @2009 API
PLAN
DO STUDY
ACT
An “Engine” for Learning
What specifically are we trying to accomplish?
What change(s) might we introduce and why?
How will we know if a change is an improvement?
What change(s) might we introduce and why?
What specifically are we trying to accomplish?
MEASURES
![Page 30: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/30.jpg)
How will we know if a change is an improvement?
LAGGING OUTCOME: % students go to college
OUTCOME: % students that are accepted to at least one college
PROCESS MEASURE: % of students completing FAFSA
PROCESS MEASURE: % students taking SAT or ACT
Ultimate outcomes
Frequently-collected; embedded in daily work
![Page 31: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/31.jpg)
**Percentages calculated using student numbers for 2015 from publically accessible databases
Schools
![Page 32: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/32.jpg)
Spread and Scale
Understand the Problem and the
System that Produces it
Test and Build Evidence
Generate ideas for change
Focus collective efforts
Improvement Science Cheat
Sheet
32
![Page 33: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/33.jpg)
Tool: The PDSA Cycle
@2009 API
PLAN • What’s your change?
• What’s your prediction?
• Plan to conduct test
DO • Execute test • Collect data, document observations
STUDY • Compare
results to prediction
• What did you learn?
ACT • Next steps: Adapt, adopt, abandon
Making your theories,
assumptions and hypotheses
explicit
Revealing gaps in our
understanding
![Page 34: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/34.jpg)
1. PLAN
2. DO 3. STUDY
4. ACT
OBSERVATIONS: ~100 seniors attended, 30 advisors Fewer students successfully completed
RESULTS: 50% completed full application LEARNINGS: Sitting with them helped, but large number of students hadn’t completed personal statements yet! Full application was more time-intensive than anticipated!
NEXT CYCLE: Test a personal statement writing unit into 11th grade Extend AppFest to Full day
CHANGE IDEA: AppFest half-day with advisors and seniors
PREDICTIONS: 80% of students will complete at least one full application
![Page 35: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/35.jpg)
Triad Activity:
Go back to one of your proposed changes to the college application process. Plan a test of the change.
35
Change Idea to be Tested:
Details of the test: When, where, and with who will you enact the change idea? Predictions: • What will be the results of
enacting this change? • How might this change fail? • How might this change
negatively affect another part of your system?
![Page 36: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/36.jpg)
Spread and Scale
Understand the Problem and the
System that Produces it
Test and Build Evidence
Generate ideas for change
Focus collective efforts
Improvement Science Cheat
Sheet
P D
S A
P D
S A
![Page 37: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/37.jpg)
How Do We Make Change?
API 2013
Quality with reliability at
scale A P S D
A P S D
Very Small Scale Test
Follow-up Tests
Wide-Scale Tests of Change
Implementation of Changes
The Improvement Science Way
Ideas for change
![Page 38: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/38.jpg)
38
What are the “End” Goals?
Measureable improvement
Specified changes that led to the improvement
Improvement capacity to apply to future work
![Page 39: Improvement Science 101summit.carnegiefoundation.org/session_materials/D1. Improvement Science 101/D1...Science Cheat Sheet. 10 . Tool: Process Map. 11 What is it? ... (measureable,](https://reader034.vdocuments.net/reader034/viewer/2022042100/5e7c8ecca16fc424f576a445/html5/thumbnails/39.jpg)
Reflections on Improvement Science:
39
As you think about applying improvement science to the problematic outcome you named at the start of the session…
What is one thing that could immediately help you?
Where do you anticipate challenges with this approach?
What questions do you still have?