Anthony S. Bryk Master Class, University of Bristol

2

Triple Aims of Educational Improvement

EFFICIENCY  

EFFECTIVENESS  

ENGAGEMENT  

Be0er  Use  of  Resources  

Ambi<ous  Learning    For  All  Students  

More  Relevance  

How We Work Now: Tower of Babel Problem

3

The Educational R&D Problem

•  Accelerate Improvement Efforts

•  Aim for Quality, Reliably at Scale

4

How We Are Working on This

•  Analogical Scavengers—The Gawande Inspiration

•  Learning by Doing—Can we actually make the ideas work?

•  Engaging a Larger Community

An Inspiration: Improvement Science in Healthcare Protecting 5 Million from Harm,

Saving 100,000 Lives

7

We can accomplish more together, than

even the best of us can do alone.

Complex systems problems that we now seek to solve

Power of Networks

Networked Improvement Communities: What are they?

Integrating Two Big Ideas:

•  The discipline of Improvement Science

joined to

•  The Power of Networks

Accelera'ng  Learning  in  and  through  Prac'ce  to  Improve  

Six Principles Guide the Work (plus useful tools to scaffold the activity)

9

Taken Together:

•  Disciplined Inquiry

•  Rudiments a scientific community

•  Aim: systematic practice improvement

I.  Problem-­‐  &  User-­‐Centered  

•  What  is  the  specific  problem  we’re  trying    to  solve?  

 •  What  we  tend  to  do  now:  a  general  issue  comes  into  view  and  we  jump  on  solu<ons  

   

60-­‐70%  Students  assigned  to  developmental  math  

course.  

80%  Percent  of  these  

students  that  never  get  past  this  gate.  

500,000  students  

in  every  cohort  will  never  complete  college  math  

requirement.  

11

The  Problem  

A  Solu<on  Framework:    Integrated  Pathways  

12  

Through  college-­‐level  sta5s5cs  

“To-and-through” college-level quantitative reasoning

Two 1-year pathways “to and through college math”

1  

2  

II.  Varia5on  in  Performance  is  the  problem  to  solve  

•  “What  Works”  is  typically  the  wrong  ques<on    •  Real  Issue:  Quality  Improvement  Ques<on    

“How  to  advance  effec<veness  among  diverse    faculty  engaging  varied    popula<ons    of  students  and  working  in  different    organiza<onal  contexts?”    

•  Goal:  Achieve  efficacy  with                      reliability  at  scale  

             

 

Trad

ition

al S

eque

nce

Stat

way

Effects: Time to Complete a College Level Math Course 1  Year   2  Years    

 Triple the

success rate in half the time.

6%

51%

15%

What is Next?

•  Normal Course of Events: “It Works” – Tout success – Publish results – Hope others pick this up – Go onto our next project

Varia<on  in  Pathways  Success  Rates  by  College  (n=19)  

16  

1

2 3

4 5

6 7

8

9

11

12 13

14

15

17

18 19

-50%

0%

50%

100%

0% 50% 100%

Stat

way

Stu

dent

s

Non-Statway Matched Comparisons

No improvement line

We also have a failure, why? What can we learn?

Triple success rate line

III.  See  the  System  to  Improve  it  

 

•  Put  simply:  It  is  hard  to  improve  what  we  do  not              fully  understand.  

How Do We Heal Medicine? Atul Gawande April, 2012

Gawande’s Closing Observation

Making systems work is the great task of my generation of physicians and scientists. But I would go further and say that making systems work — whether in healthcare, education, climate change, making a pathway out of poverty — is the great task of our generation as a whole.

The  Invisible  Complexity  Schooling  

21  

The  Invisible  Complexity  of  Schooling  

60-­‐70%  Students  assigned  to  developmental  math  

course.  

80%  Percent  of  these  

students  that  never  get  past  this  gate.  

500,000  students  

in  every  cohort  will  never  complete  college  math  

requirement.  

22

Returning  to  The  Presen<ng  Problem  

The  Orien<ng  Problem  

     Extraordinarily  high  failure  rates  among  students  assigned  to  developmental  math  instruc<on  

Consolidate  the  courses  into  a  1-­‐year  pathway  

Real  world  problems  from  sta<s<cs  as  the  organizer  

Psycho-­‐social  interven<ons  aimed  at  “produc<ve  persistence”  

Rapid  analy<cs  capacity  

Faculty  development  

Causal  Systems  Analysis:  Why  do  we  con<nue  to  get  the  outcomes  observed?  

Primary  Causes    for    High  Failure  Rates    

Organizing  Improvement  Hypotheses  

event  

???  

Ins$tu$onal*structures*don’t*support*student*success*

Students*are*not*engaged*or*mo$vated*

The*course*material/content*is*problema$c*

Instructors*lack*skills*and*beliefs*that*students*can*succeed*

State*policy*does*not*support*student*success*

Low*success*rates*in*development*

math*func$on*as*a*gatekeeper*to*opportunity*

Students*lack*the*skills*to*succeed*

Ineffec$ve*advising*system*

Ineffec$ve*learning*support*services*

Lack*of*social*$es*to*each*other*and*to*their*instructors*

Inaccurate*placement*

High*dropAout*rates*between*courses*

Don’t*believe*they*can*learn*

Math*and*tes$ng*anxiety*

Poorly*prepared*mathema$cally*

They*don’t*know*how*to*study*math*

They*don’t*know*how*to*“navigate”*the*college*world*

Not*interes$ng*or*relevant**

Not*seen*as*useful*

Text*is*inaccessible*

Does*not*leverage*what*we*know*about*how*students*learn*

Too*many*courses*in*the*developmental*sequence*

Few*opportuni$es*to*learn*from*others*

Lack*knowledge*of*learning*theory*

Weak*pedagogy*

Don’t*believe*suppor$ng**student*success*is*their*job*

Lower*reimbursement*for*developmental*math*

Tradi$onal*transfer*requirements*impede*innova$on*

Arcane*curricular*topics*create*needless*hurdles*

Funding*based*on*enrollment*rather*than*outcomes*

OOen*taught*by*adjunct*faculty**with*liPle*professional*support*

The  Orien<ng  Problem  

Embedded  literacy  and  language  barriers    

     Extraordinarily  high  failure  rates  among  students  assigned  to  developmental  math  instruc<on  

Lose  large  #  of  students  at  the  transi<ons  

Consolidate  the  courses  into  a  1-­‐year  pathway  

Students  mindsets  undermine  success  

Real  world  problems  from  sta<s<cs  as  the  organizer  

Students  “gone”  before  we  know  it  

Psycho-­‐social  interven<ons  aimed  at  “produc<ve  persistence”  

Rapid  analy<cs  capacity  

Course  material  and  instruc<on  are    not  engaging  

Faculty  development  

Analy<c  Summary  of  Causal  Systems  Analysis  

Primary  Causes    for    High  Failure  Rates    

Organizing  Improvement  Hypotheses  

Eventually  leads  to  a    “Pathways  Strategy”  

Pathways Driver Diagram: Organizing a Networked Improvement Community

Aim: increase from 5% to 50%, students achieving college math credit within one year of continuous enrollment

Instructional System: Organized around productive struggle, explicit connections, and

deliberate practice.

Productive Persistence:

Students develop skills and maintain positive mindsets

Language and Literacy: Students

use language in understanding

problems, reason mathematically, and communicate results

Advancing Teaching: Effective

teaching within 2 years of

implementation

Reduce transitions + assure enrollment across semesters

Deliberate focus on “Starting Strong”

Promote students’ ties to peers, faculty,

pathway

Math that matters: students see material interesting, relevant

Enhance faculty’s beliefs and relational

practices

Opening lessons engage interest, assure early

success

Direct interventions to influence student mindsets

Real-time data tracking on student engagement

Detail supportive classroom norms and social

connections

Professional development on “Starting Strong”

A  Community  Explicates  its  Causal  Thinking:    

A  Community  Explicates    its  Causal  Thinking:  

 A  Driver  Diagram  

 to  Organize  Its  Major    Improvement  Hypotheses  

Pathways Driver Diagram: Organizing a Networked Improvement Community

Aim: increase from 5% to 50%, students achieving college math credit within one year of continuous enrollment

Instructional System: Organized around productive struggle, explicit connections, and

deliberate practice.

Productive Persistence:

Students develop and maintain

positive mindsets

Language and Literacy: Students

use language in understanding

problems, reason mathematically, and communicate results

Advancing Teaching: Effective teaching within 2

years of implementation

Reduce transitions + assure enrollment across semesters

Deliberate focus on “Starting Strong”

Promote students’ ties to peers, faculty,

pathway

Math that matters: students see material interesting, relevant

Enhance faculty’s beliefs and relational

practices

Opening lessons engage interest, assure early

success

Direct interventions to influence student mindsets

Real-time data tracking on student engagement

Detail supportive classroom norms and social

connections

Professional development on “Starting Strong”

Elabora<ng  Out    The  Driver  Diagram  

Produc<ve  Persistence  

IV.  You  cannot  improve  at  scale  what  you  cannot  measure  

•  Measureable  targets:  “Some  is  not  a  number;  soon  is  not  a  <me”-­‐-­‐Valued  outcome  measures  – But,  you  just  can  not  stand  at  the  end  of  the  line.    

•  We  need  process  measures  <ed  to  intermediate  targets.  

Produc<ve  Persistence  

Suppor<ve  social  rela<onships  

Target:  How  do  we                  measure  it?  

Mindsets  about  the  value  of  math  

Mindsets  about  poten<al  to  learn  

math  

Anxiety  Regula<on  

Study  Skills          Conceptual  Task:        reduce  to  5  core  ideas      focus  on  underlying          malleable  causes  +            change  evidence      

 Prac5cal          Measurement:  

reduce  900  items  to  26          “you  have  3  minutes”  

V.  Accelerate  Improvement:      Embrace  Disciplined  Inquiry  

•  Policy  Romance  of  the  Silver  Bullet  – Move  quickly  to  large  scale  implementa<on,  but…    

•  We  typically  don’t  know  whether:  –   We  can  make  these  ideas  work  at  all;    –   We  have  capacity  and  will  to  execute  with  efficacy  at      

 scale.    

•  Instead,  a  DEED  orienta<on  – Quick,  minimally  intrusive,  an  empirical  warrant  –   Mantra:  Learn  Fast,  Fail  Fast,  Improve  Fast!  

A System of Social Learning to Improve

Transla5onal  Research  

Interven5ons    (Alpha  Labs)  

 Will  they  work  for  community  college  students,  and  if  so,  

how?    

Expert    Prac55oner  Knowledge  (Subnet)  

 Building  robust  clinical  

knowledge  about    effec<ve  materials  and    instruc<onal  prac<ces.  

Learning  from    Network  Data  (Hub  Analy5cs)  

   

Learning  from  observed  variability.  

Discerning  the  unseen.  

32

Transla5onal  Research  

Interven5ons    (Alpha  Labs)  

•  Will  they  work  for  community  college  students,  and  if  so,  

how?    

Initial Alpha Lab: Mindset Intervention

•  A carefully designed experimental intervention has changed student mindsets.

•  But just because an intervention can work in one setting does not mean it will work in another.

•  Need to engineer it to “fit” in instructional contexts. –  Conduct rapid R&D using DEED methodology. –  “Smell testing” –  4 months from small-scale test to larger scale use.

Rapid Iterative DEED cycles

•  Research-Practitioner Team

•  Testing –  Small double-blind randomized

trial in Algebra course (n = 26)

–  Larger double-blind experiment (n = 288)

•  Introduce to faculty network, carefully study emerging results, continue to revise, refine, and extend.

34

Roberta Carew, Statway faculty

Valencia College

35

Learning  from    Network  Data  (Hub  Analy5cs)  

 •  Learning  from  observed  

variability.  Discerning  the  “unseen.”    

36

!

1. Assessing Change: Initial Evidence of Efficacy of Starting Strong Package

2. Predictive Analytics—targeting support

(a simple at-risk indicator scoring 5 key items/item clusters-day 1)

37

%  of  who  failed  the  end-­‐of-­‐term  common  assessment    

Connections to Stereotype Threat

12%   13%   14%  

28%  

40%  

7%   11%   14%  

50%  

71%  

0%  

10%  

20%  

30%  

40%  

50%  

60%  

70%  

80%  

Never   Hardly  Ever   Some<mes   Frequently   Always  

Pathways    Drop

out  

All  students   Black  students  

“How  oqen,  if  ever,  do  you  wonder:  ‘Maybe  I  don't  belong  here?’”

N  =  714  math  students    

39

Expert    Prac55oner  Knowledge  (Subnet)  

 Building  robust  clinical  

knowledge  about  effec<ve  instruc<onal  materials  and  prac<ces.  

   

PDSA Cycle: Rapid, Small Experimental Trials

PLAN   DO  

ACT   STUDY  

The  Three  Ques5ons:    •  What  specifically  are  we  trying  to  accomplish?  

 •  What  change  might  we  introduce?  

 •  How  will  we  know  that  the  changes  are  an  improvement?  

Improving Instructional Routines in Support of Productive Persistence: PDSA Cycles 

•  Faculty routines and email scripts re: absent students

•  Student group noticing routine

•  Effective scaffolding for group roles (rich problems)

41

Sample Run Chart for a PDSA Cycle (Student Group Noticing Routine)

0%#

20%#

40%#

60%#

80%#

100%#

120%#

1/14/1

3#

1/21/1

3#

1/28/1

3#

2/4/13

#

2/11/1

3#

2/18/1

3#

2/25/1

3#

3/4/13

#

3/11/1

3#

3/18/1

3#

3/25/1

3#

4/1/13

#

4/8/13

#

4/15/1

3#

4/22/1

3#

4/29/1

3#

Perce

nt'of

'Stud

ent'in'A

/end

ance'

A/endance'(By'Day)'

Typical#a4endance#

Observed#a4endance#

n=44#Median:#0.85#

Median#

Developing a Quality Process Reliably at Scale

Develop  A  Change  

Test  under  mul<ple  condi<ons  

Test  under  increasingly  varied  

condi<ons  

Make  the  change  permanent  

Ini5al  Hunches  

System  Changes  

1  school  1  administrator  

5  schools  Many  administrators  

En<re  ver<cal  team    A  more  diverse  group  of  administrators  

District  Wide  All  administrators  

Seeing  Task  Complexity  

Seeing  Organiza<onal  Complexity  

Learning  to  improve  feedback  conversa<ons  between  

 principals  and  new  teachers  PLAN   DO  

ACT   STUDY  

A  Developmental  Dynamic  

Hunches  Theories  Ideas  

Ini<a<ng  Resources  

P D

S A

P D

S A

P D

S A P D

S A

Moving  out  toward  More  diverse  condi<ons:  “factor  of  5  rule  of  thumb”    

Aiming  for  Efficacy  with  Reliability  at  

Scale  

VI.  Accelerate  Improvement:    Tap  the  Power  of  Networks  

•  A  source  of  innova<on  – Dig  into  the  details:  what  worked,  how,  for  whom?  – Can  we  adap<vely  integrate  this  into  other  contexts?    

•  Mul<ple  fast  replica<on  – Can  we  make  this  happen  with  efficacy,  reliably  at  scale?    

•  Innova<on  diffusion—it  is  largely  about  who  is  connected  to  whom  and  what  they  think  and  do  

A  Learning  Educa'onal  System  

A   A  

Improvement  Networks:  Accelerate  Learning  in  Prac<ce  for  Improvement  

A  

B  

A  A   A  B  

A  A   A  B  

A  A   A  B  

C  

(Englebart,1994)  

It is all about accelerating how we learn in and through practice to improve.