Asking better questions to build better models

Sharon Dawes

Some policy questions• Should we eradicate polio?• How can schools help third graders become good readers?• What kinds of public investments lead to urban vitality?• What is the optimal approach to immigration?• How does health insurance coverage affect health care costs?• Can a ban on large sugary drinks reduce obesity and diabetes?• Does public posting of sanitary inspections lead to cleaner

restaurants?• . . .

All models are wrong, some are useful.

Box, 1979

Why models are wrong

ComplexityBoundariesDynamicsDataTime

Complexity

Wicked problems

Tangled problems

Structured problems

Boundaries

• Organizations, jurisdictions, sectors• Groups, individuals and their interests• Existing policies and laws• Contextual factors• Processes• Practices• Geography• Behaviors

What goes in the model, what is left out, and why and how do the choices matter?

Dynamics

Feedback structure of URBAN1, Ghaffarzadegan, et al.

Data

The “best available data” (BAD) is inherently flawed

Knowledge problems• Meaning• Explicitness• Codifiability• Embeddedness• Dynamics

Data problems• Accuracy• Completeness• Bias• Timeliness• Availability• . . .

TimeProblem definitionGoal settingPolicy formulationStrategy formulation

DesignDevelopment

DeploymentImplementationOperations

Performance assessmentEvaluation

Policy modeling over timeProblem definitionGoal settingPolicy formulationStrategy formulation

DesignDevelopment

DeploymentImplementationOperations

Performance assessmentEvaluation

Understanding the problem

and the options

Exploring alternative approaches

Understanding the policy

system

Evaluating and interpreting

policy impacts

Enter stakeholders . . .

Stakeholders bring

• Different knowledge and points of view• Different goals and preferences• Different capabilities and authority to act• Different cultures (national, organizational,

professional)• Different amounts and kinds of power and

autonomy • Different levels of confidence and trust • Different amounts of tolerance for risk

An example: eradication of polio

with thanks to Dr. Kimberly Thompson & colleagues

The collaborators gratefully acknowledge financial support in the form of unrestricted gifts to the Harvard Kids Risk Project and grants from the US Centers for Disease Control and Prevention (CDC): U50/CCU300860, U01 IP000029, NVPO N37 (FY2005), 200-2010-M-33379, 200-2010-M-33679, 200-2010-M-35172, U66 IP000169, the World Health Organization (WHO) APW200179134, and the Bill & Melinda Gates Foundation: 4533-17492 , 4533-18487, 4533-21031. I thank Radboud Duintjer Tebbens for embarking and continuing on the entire journey with me. I also thank collaborators from the CDC, including James Alexander, Lorraine Alexander, Larry Anderson, Gregory Armstrong, Albert Barskey, Brenton Burkholder, Cara Burns, Victor Cáceres, Jason Cecil, Susan Chu, Steve Cochi, Kathleen Gallagher, Howard Gary, John Glasser, Steve Hadler, Karen Hennessey, Hamid Jafari, Julie Jenks, Denise Johnson, Bob Keegan, Olen Kew, Nino Khetsuriani, Robb Linkins, Naile Malakmadze, Rebecca Martin, Eric Mast, Steve McLaughlin, Steve Oberste, Mark Pallansch, Becky Prevots, Hardeep Sandhu, Nalinee Sangrujee, Jean Smith, Philip Smith, Peter Strebel, Linda Venczel, Gregory Wallace, Steve Wassilak, Margie Watkins, and Bruce Weniger, and from the WHO, including Bruce Aylward, Fred Caillette, Claire Chauvin, Philippe Duclos, Esther deGourville, Hans Everts, Marta Gacic-Dobo, Tracey Goodman, Ulla Griffiths, David Heymann, Scott Lambert, Asta Lim, Jennifer Linkins, Patrick Lydon, Chris Maher, Linda Muller, Roland Sutter, Rudi Tangermann, Chris Wolff, and David Wood. I also thank the Global Polio Laboratory Network, Harrie van der Avoort, Francois Bompart, Anthony Burton, Konstantin Chumakov, Laurent Coudeville, Walter Dowdle, Paul Fine, Michael Galway, Shanelle Hall, Neal Halsey, Tapani Hovi, Kun Hu, Dominika Kalkowska, Samuel Katz, Jong-Hoon Kim, Tracy Lieu, Marc Lipsitch, Anton van Loon, Apoorva Mallya, Phil Minor, John Modlin, Van Hung Nguyen, Walter Orenstein, Carol Pandak, Peter Patriarca, Christina Pedreira, Stanley Plotkin, Hazhir Rahmandad, Robert Scott, John Sever, Thomas Sorensen, John Sterman, Robert Weibel, Jay Wenger, and Peter Wright.

Collaborators

Thompson, K. M. (2012). The role of risk analysis in polio eradication: Modeling possibilities, probabilities, and outcomes to inform choices. Expert Review of Vaccines, 11(1), 5-7.Thompson, K. M. (2006). Poliomyelitis and the role of risk analysis in global infectious disease policy and management. Risk Analysis, 26(6), 1419-1421.Thompson, K. M., & Duintjer Tebbens, R. J. (2012). Current polio global eradication and control policy options: Perspectives from modeling and prerequisites for OPV cessation. Expert Review of Vaccines, 11(4), 449-459.Thompson, K. M., & Duintjer Tebbens, R. J. (2011). Challenges related to the economic evaluation of the direct and indirect benefits and the costs of disease elimination and eradication efforts. In S. L. Cochi & W. R. Dowdle (Eds.), Disease Eradication in the 21st Century: Implications for Global Health. Cambridge, MA: MIT Press.Thompson, K. M., & Duintjer Tebbens, R. J. (2008a). The case for cooperation in managing and maintaining the end of poliomyelitis: Stockpile needs and coordinated OPV cessation. The Medscape Journal of Medicine, 10(8),190. Retrieved from http://www.medscape.com/viewarticle/578396.Thompson, K. M., & Duintjer Tebbens, R. J. (2008b). Using system dynamics to develop policies that matter: Global management of poliomyelitis and beyond. System Dynamics Review, 24(4), 433-449.Thompson, K. M., & Duintjer Tebbens, R. J. (2007). Eradication versus control for poliomyelitis: An economic analysis. The Lancet, 369(9570),1363-71.Thompson, K. M., & Duintjer Tebbens, R. J. (2006). Retrospective cost-effectiveness analyses for polio vaccination in the United States. Risk Analysis, 26(6),1423-1440.

Selected academic results

Plain language communication of results

Some suggestions• Build relationships with policy makers and policy

advisors• Treat models as learning objects to be improved by

interaction with the people who own the policy problem.• Adopt techniques that engage stakeholders• group model building, participatory design,

value-sensitive design• Learn and speak the language of practice• Custom-make presentations for policy makers and managers

• Consider multiple methods to understand context and complexity • case studies, test beds, evaluation studies

• Update, iterate, adjust, repeat

Many questions, many answers, better models

Policy makers

Other analys

ts

Policy stakeholders

Policy model

ers

The big policy question÷

Elements of complexityBoundary questions

Data challengesDynamic interactionsTime considerations

XMultiple iterations

Plain language=

Insights, better understanding of policy options and effects

To continue the discussion• eGovPoliNet project, http://www.policy-community.eu/ • Policy Making 2.0 community on Linked In• PIN-l listserv• PIN website: pin.asu.edu • Handbook of Policy Informatics (forthcoming, MIT Press)• International Digital Government Research Conference (dgo)• Policy Informatics Panel, June 17-20, Quebec City

• International Research Society for Public Mgmt (IRSPM)• Policy Informatics Track, April 10-12, Prague

• Twitter hashtag: #policyinformatics

www.ctg.albany.edu