Continuous Decision Improvement (CDI):Decisive Leadership for Complex Environments

Tomas J. Aragon, MD, DrPHPrincipal Investigator, Cal PREPAREUC Berkeley School of Public Health

Health Officer, City and County of San FranciscoDirector, Population Health Division

San Francisco Department of Public Health

January 30, 2015

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 1 / 34

Outline

1 Background and introduction

2 Continuous Decision Improvement

3 Selected public health examples

4 Summary and review

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 2 / 34

Acknowledgments to CDI team!

Tomás J. Aragón, MD, DrPHHarvey Kayman, MD, MPHCindy Lambdin, RN, MSWayne Enanoria, Ph.D, MPHChristine Siador, MPHJeannie BalidoJulia Dysart, MBALaura GerhardtDavid Krauth

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 3 / 34

Background and introduction

What kind of decisive leader are you? should you be?

DECISIVE LEADERSHIP

SOLUTION-ORIENTEDWhat should we do?

PROCESS-ORIENTEDHow should we decide?VERSUS

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 4 / 34

Background and introduction

What kind of decisive leader are you? should you be?

DECISIVE LEADERSHIP

SOLUTION-ORIENTEDWhat should we do?

PROCESS-ORIENTEDHow should we decide?VERSUS

Professor Michael Roberto:“Many leaders focus on finding the right solutions to problems rather thanthinking carefully about what process they should employ to make key decisions.When confronted with a tough issue, we focus on the question, what decisionshould I make? We should first ask, how I should I go about making thisdecision?” (Source: The Art of Critical Decision Making—Course Guidebook)

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 5 / 34

Background and introduction

Overview—What is good decision making?

Good decision makingis having a reliable processto choose the right combination of actions (alternatives)to achieve the desired combination of results (objectives)

Decision making occurs in increasingly complex environments . . .%PXIVREXMZIW������'SRWIUYIRGIW�����������������3FNIGXMZIW

9RGIVXEMRXMIW

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 6 / 34

Background and introduction

Complexity1 and why it matters

What is a complex system?1 A population of diverse agents, all of which are2 connected, with behaviors and actions that are3 interdependent, and that exhibit4 adaptation and learning.

Why do we care? Complex systems . . .are ambiguous, deceptive, unpredictableare difficult to direct and control (e.g., adaptive resistance)can evolve along divergent pathways (path dependance)can produce “tipping points” (e.g., epidemics)

1Miller JH & Page SE. Complex Adaptive Systems: An Introduction toComputational Models of Social Life Princeton University Press 2007

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 7 / 34

Background and introduction

Mitigating and harnessing complexity

Mitigating complexityExpect the unexpected and unintended consequencesExpect and prepare to fail (avoid overconfidence)

Harnessing complexityPractice humility / humble inquirya

Strengthen decision making processesSee every failure as a learning opportunityDesign for agility, adaptability, and responsivenessBalance exploration (learning) and exploitation (execution)

aSchein EH. Humble Inquiry: The Gentle Art of Asking Instead of Telling.Berrett-Koehler Publishers, 2013

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 8 / 34

Background and introduction

What is the Dual-Process model?

DeliberationIntuition

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 9 / 34

Background and introduction

Intuition (naturalistic decision making)

Size-up Imagine Do Size-up

DefinitionFramework for studying how people make decisions and perform cognitivelycomplex functions in demanding, real-world situations. These includesituations marked by limited time, uncertainty, high stakes, environmentalconstraints, unstable conditions, and varying amounts of experience.

ExamplesPanhandler approaches you on the street asking for money.Driving on city street when suddenly you hear a siren.Fighter pilots in heat of aerial battle: Observe-Orient-Decide-Act(OODA) Loop

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 10 / 34

Background and introduction

Deliberation (rational decision making)

Analyze Plan Do Re-analyse

Example: Systems Decision Processa—a best practice decision processSystems Decision Process 4D Decision Process

1 Problem definition Define problem2 Solution design Design alternatives3 Decision making Decide alternatives4 Solution implementation Do (plan & execute)

a Parnell GS, et al. Handbook of Decision Analysis. Wiley, 2013

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 11 / 34

Background and introduction

Systems Decision Process in full detail—not focus of talk

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 12 / 34

Continuous Decision Improvement

The 4D Decision Process—An Overview

(MZIVKIRGI���������������'SRZIVKIRGI

(IJMRI�������������(IWMKR����������������(IGMHI������������(S

463���4VSFPIQ��6IJVEQI��3FNIGXMZIW

-:)���-RXIKVEXI�ZEPYIW��IZMHIRGI��STXMQM^I�I\TIGXIH�:EPYI��)\TPSVI��)ZEPYEXI

%'8���%PXIVREXMZIW��'SRWIUYIRGIW��8VEHI�SJJW

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 13 / 34

Continuous Decision Improvement

The 4D Decision Process—An Overview

(MZIVKIRGI���������������'SRZIVKIRGI

(IJMRI�������������(IWMKR����������������(IGMHI������������(S

463���4VSFPIQ��6IJVEQI��3FNIGXMZIW

-:)���-RXIKVEXI�ZEPYIW��IZMHIRGI��:EPYI�I\TIGXEXMSR��YXMPMX] ��)\TPSVI��)ZEPYEXI

%'8���%PXIVREXMZIW��'SRWIUYIRGIW��8VEHI�SJJW

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 14 / 34

Continuous Decision Improvement

PROACTIVE—Reframe using (HELP)2 acronym

Use (HELP)2 for reframing, objectives, and consequences

H Health outcomesH Health equityE Ethical issuesE Efficiency issuesL Legal issuesL Logistical issuesP Public trustP Political support

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 15 / 34

Continuous Decision Improvement

What is Continuous Decision Improvement (CDI)? (1/2)

The 4D Decision Process improves decision making, but we aspire tocontinuously improve decision making. That is CDI!

Quality improvement (QI) in public healtha

A continuous effort to achieve measurable improvements in processperformance to improve the health of the community.

Continuous decision improvement (CDI) in public healthA continuous effort to achieve measurable improvements in the planningand execution of decision-making processes to achieve organization goalsand to improve the health of the community.

a Riley W, et al. Defining Quality Improvement in Public Health. J Publ HealthManagement and Pract, Jan/Feb 2010

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 16 / 34

Continuous Decision Improvement

What is Continuous Decision Improvement? (2/2)

4PER�(IGMWMSR���'SQTSWMXMSR��'SRXI\X��'SQQYRMGEXMSR��'SRXVSP

(IGMWMSR�4VSGIWW���(IJMRI���(IWMKR���(IGMHI���(S

-QTVSZIQIRX

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 17 / 34

Continuous Decision Improvement

Effects of conflict and consensus on decision makinga

a Roberto MA. Why Great Leaders Don’t Take Yes for an Answer: Managing forConflict and Consensus. FTPress, 2013 (2nd Ed.)

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 18 / 34

Continuous Decision Improvement

Cognitive and affective conflicta

a Roberto MA. Why Great Leaders Don’t Take Yes for an Answer: Managing forConflict and Consensus. FTPress, 2013 (2nd Ed.)

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 19 / 34

Continuous Decision Improvement

The path to decision successa

a Roberto MA. Why Great Leaders Don’t Take Yes for an Answer: Managing forConflict and Consensus. FTPress, 2013 (2nd Ed.)

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 20 / 34

Continuous Decision Improvement

Continuous Decision Improvement = PDSA + 4C + 4D

4D #1 (4C)

Communication

Context

Composition

Control

4D #2 (PRO-ACT-IVE)

A Cognitive Conflict

B Affective Conflict

C Shared Understanding

D Commitment

Quality of Decision Process

- Decision Quality*

- Constructive Conflict (A, B)

- Comprehensive Consensus (C, D)

Plan:Decision Process

Do:Decision Process

Study:Decision Process

Act:Improve Decision Process

*Frame, Alternatives, Information, Measurements, Logical reasoningTomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 21 / 34

Continuous Decision Improvement

The CDI Choice-Mobile—All aboard!

463��%'8��-:)

(IJMRI���(IWMKR���(IGMHI���(S��'SRWXVYGXMZI�'SRJPMGX��'SQTVILIRWMZI�'SRWIRWYW

��'SQTSWMXMSR��'SRXI\X��'SQQYRMGEXMSR��'SRXVSP

'334)6%8-32�!�86978���,91-0-8=

(IGMWMSR�7YGGIWW����(IGMWMSR�5YEPMX]����-QTPIQIRXEXMSR�����IJJIGXMZIRIWW

'SKRMXMZI���)QSXMSREP���&ILEZMSVEP

�,)04 B�

'��������(�������-

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 22 / 34

Continuous Decision Improvement

Aside: What is a consequence table?a

A consequence table organizes your data:1 Objectives (column 1)2 Alternatives (Options A, B, and, C)3 Measures (cells = Consequences of the alternatives on the objectives)

a Hammond JS, et al. Smart Choices: A Practical Guide to Making Better Decisions,Crown Business, 2002

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 23 / 34

Selected public health examples

Example 1: Trust is a decisiona

Definition of trust“Trust is the degree of confidence you have that another party can berelied on to fulfill commitments, be fair, be transparent, and not takeadvantage of your vulnerability.”

FACT: Good team decision making requires cooperation.Cooperation requires trust and humilityExtending and building trust are decision problems

RECOMMENDATION:Learn and practice individual trust-building CDITrain team members in trust-building CDI

a Hurley RF. The Decision to Trust: How Leaders Create High-Trust Organizations.Jossey-Bass, 2011

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 24 / 34

Selected public health examples

Understanding trust as a decision problem (1/2)

Trustor Trustee

PredispositionsVulnerability

Expectation

Competent*

CulturalHumility**

Character(intent/integrity)

Consequence Care(benevolence)

DECISIONto Trust?

Probability

Relational Context- Personal security- Power imbalance- Common values or interests

External Context- Situational security- Uncertainty

Cognitive Biases

INFLUENCEDecision to Trust?

* capable, consistent, continuously improving** self-awareness of power, privilege, & prejudices; redress imbalances for respectful partnerships

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 25 / 34

Selected public health examples

Understanding trust as a decision problem (2/2)

Trustor Trustee

PredispositionsVulnerability

Expectation

Competent*

CulturalHumility**

Character(intent/integrity)

Consequence Care(benevolence)

DECISIONto Trust?

Probability

Relational Context- Personal security- Power imbalance- Common values or interests

External Context- Situational security- Uncertainty

Cognitive Biases

INFLUENCEDecision to Trust?

* capable, consistent, continuously improving** self-awareness of power, privilege, & prejudices; redress imbalances for respectful partnerships

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 26 / 34

Selected public health examples

Example 2: Ebola: Monitoring at-risk returnees

Table: 4D Decision Process using PROACTIVE

Definition DescriptionP Problem definition Potential Ebola disease and transmissionR Reframe See (HELP)2

O Objectives What are we trying to accomplish?

A Alternatives What public health actions can we use?C Consequences What are the consequences (impacts)?T Trade-offs What are the trade-offs?

I Integrate V & E See (HELP)2

V Value expectation n/aE Explore & Evaluate pending

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 27 / 34

Selected public health examples

Organizing objectives using the (HELP)2 acronym

(HELP)2 ObjectivesH Health outcomes 1 Ensure effective monitoring for signs and symptoms

2 Minimize nonessential contact with uninformed persons3 Minimize exposures to new infections (false positives)

H Health equity 4 Minimize stigma and discrimination of AfricansE Ethical issues 5 Adopt least restrictive option (see ethical framework)E Efficiency issues 6 Ensure cost-effective solutionsL Legal issues 7 Ensure health officer order withstands legal challengesL Logistical issues 8 Ensure pychosocial support for adherence to orderP Public trust 9 Build trust with affected individuals and family

10 Ensure consistency of public health actions11 Mitigate uncertainties (eliminate, reduce, substitute)12 Optimize risk communication

P Political support 13 Ensure understanding and support of leadership14 Support health and safety goals of leadership

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 28 / 34

Selected public health examples

Alternatives (Ebola public health actions)

Public Health Actions RiskHigh risk Some risk Low risk No risk

Normal signs and sympstoms1. Active monitoring (21 days) Y N2. Direct active monitoring Y Y Y (HCWs) N3. Controlled movement Y R N N4. Exclusion from public places Y R N N

and congregate gatherings5. Exclusion from workplaces Y R N N6. Travel restrictions Y R N N7. Psychosocial support Y Y ? N8. Risk communication strategy Y Y Y Y

Postive signs or symptoms9. Isolation and evaluation Y Y Y N

Y = Yes (required); N = NoR = Recommended stronglyHCWs = Health care workers caring for Ebola patients in the US

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 29 / 34

Summary and review

CDI summary as a Consequence Table

Objectives Sub-objectives 4D CDIDefine PRO Y YDesign ACT Y YDecide IVE Y YDo . . . Y YMethodologic criteria Frame Y

Alternatives YInformation YMeasurements YLogical reasoning Y

Constructive conflict ↑ Cognitive conflict Y↓ Affective conflict Y

Comprehensive consensus ↑ Shared understanding Y↑ Commitment Y

Improve decision planning & process PDSA cycles YIncludes (HELP)2 considerations HH-EE-LL-PP Y

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 30 / 34

Summary and review

The CDI Choice-Mobile—All aboard!

463��%'8��-:)

(IJMRI���(IWMKR���(IGMHI���(S��'SRWXVYGXMZI�'SRJPMGX��'SQTVILIRWMZI�'SRWIRWYW

��'SQTSWMXMSR��'SRXI\X��'SQQYRMGEXMSR��'SRXVSP

'334)6%8-32�!�86978���,91-0-8=

(IGMWMSR�7YGGIWW����(IGMWMSR�5YEPMX]����-QTPIQIRXEXMSR�����IJJIGXMZIRIWW

'SKRMXMZI���)QSXMSREP���&ILEZMSVEP

�,)04 B�

'��������(�������-

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 31 / 34

Summary and review

What kind of decisive leader are you? should you be?

DECISIVE LEADERSHIP

SOLUTION-ORIENTEDWhat should we do?

PROCESS-ORIENTEDHow should we decide?VERSUS

Professor Michael Roberto:“Many leaders focus on finding the right solutions to problems rather thanthinking carefully about what process they should employ to make key decisions.When confronted with a tough issue, we focus on the question, what decisionshould I make? We should first ask, how I should I go about making thisdecision?” (Source: The Art of Critical Decision Making—Course Guidebook)

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 32 / 34

Summary and review

Bibliography

1 Smart Choices: A Practical Guide to Making Better Decisions, by John S.Hammond, et al. Link: http://amzn.com/0767908864

2 Why Great Leaders Don’t Take Yes for an Answer: Managing for Conflict andConsensus (2nd Ed.), by Michael A. Roberto. Link:http://amzn.com/0133095118

3 Art of Critical Decision Making, by Michael A. Roberto. Link: http://www.thegreatcourses.com/courses/art-of-critical-decision-making.html

4 Essentials Of Public Health Ethics, by Ruth Gaare Bernheim, et al. Link:http://amzn.com/0763780464

5 Decision Making in Health and Medicine: Integrating Evidence and Values, by M.G. Myriam Hunink, et al. Link: http://amzn.com/1107690471

6 Decisive: How to Make Better Choices in Life and Work, by Chip Heath et al.Link: http://amzn.com/0307956393

7 Handbook of Decision Analysis, by Gregory S. Parnell, et al. Link:http://amzn.com/1118173139

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 33 / 34

Summary and review

Acknowledgment

CDC Cooperative Agreement 5P01TP000295This project was supported by the cooperative agreement number5P01TP000295 from the Centers for Disease Control and Prevention. Itscontents are solely the responsibility of the authors and not necessarilyrepresent the official views of the Centers for Disease Control &Prevention.

Cal PREPARE is a CDC Preparedness & Emergency Response ResearchCenter (PERRC)based at the UC Berkeley School of Public Health. For more informationabout Cal PREPARE, please visit www.calprepare.com.

Tomas J. Aragon, MD, DrPH Principal Investigator, Cal PREPARE UC Berkeley School of Public Health[10pt] Health Officer, City and County of San Francisco Director, Population Health Division San Francisco Department of Public HealthContinuous Decision Improvement (CDI): Decisive Leadership for Complex EnvironmentsJanuary 30, 2015 34 / 34