solving wicked social problems with socio-computational...
TRANSCRIPT
Solving wicked social problems with socio-computational systems
Joshua Introne, Robert Laubacher, Gary Olson*, Thomas W. Malone
MIT Center for Collective Intelligence
MIT Center for Collective Intelligence Working Paper No. 2013-001
January 2013
This is an earlier version of an article subsequently published in Küntsliche Intelligenz, February 2013, 27 (1): 45-52.
*Donald Bren School of Information and Computer Science, University of California at Irvine
MIT Center for Collective Intelligence Massachusetts Institute of Technology
http://cci.mit.edu
Solving wicked social problems with socio-computational systems
This is an earlier version of an article subsequently published in Küntsliche Intelligenz, February 2013, 27 (1): 45-52.
Joshua Introne, Robert Laubacher, Gary Olson*, Thomas Malone
Center for Collective Intelligence
Massachusetts Institute of Technology
[jintrone,rjl,malone]@mit.edu
*Donald Bren School of Information
and Computer Science
University of California at Irvine
ABSTRACT
Global climate change is one of the most challenging problems humanity has ever faced. Fortunately, a new way of
solving large, complex problems has become possible in just the last decade or so. Examples like Wikipedia and Linux
illustrate how it is now possible to combine the work of thousands of people in ways that would have been impossible only
a few years ago. Inspired by systems like these, we developed the Climate CoLab—a global, on-line platform in which
thousands of people around the world work together to create, analyze, and ultimately select detailed plans for what we
humans can do about global climate change.
The Climate CoLab has been operating since November 2009, and has an active community of thousands of users. In this
article, we outline some of the challenges faced in developing the system, describe our current solutions to these problems,
and report on our experiences.
KEYWORDS: Collective intelligence, collaborative planning, climate change
1. INTRODUCTION
Many important decision-making problems in the real world are so-called “wicked problems”—problems for which no
single computational formulation of the problem is sufficient, for which different stakeholders do not even agree on what
the problem really is, and for which there are no right or wrong answers, only answers that are better or worse from
different points of view (see, e.g., [1][2][3]). For example, most social problems (including the environment, health care,
poverty, education, and crime) are wicked in this sense, as are many problems in management (including strategic
decision-making and product design).
The problem of global climate change is a wicked problem. In fact, many people would say it has some characteristics that
make it especially challenging (or “super-wicked” [4]): Time is running out, there is no central authority that can
implement a solution, and it is truly universal: it affects every one of us and is affected by all of our actions.
Left to their own devices, scientists, journalists, politicians, businesses, and consumers will ultimately do something about
this problem. But the inefficiencies, delays, and distortions of traditional mass media, political decision-making, markets,
and scientific publication mean that the results will almost certainly not be as good as we might hope. Fortunately,
however, in just the last decade or so, a new way of solving global problems has become possible. Examples like
Wikipedia and Linux illustrate that it is now possible to combine the work of thousands of people in ways that would have
been impossible only a few years ago. Inspired by systems like these, we developed the Climate CoLab—a global, on-line
platform in which thousands of people around the world work together to create, analyze, and ultimately select detailed
plans for what we humans can do about global climate change.
Since the CoLab was first launched to the public in November of 2009, it has attracted a diverse community of thousands
of users from around the world who have used the platform to generate dozens of candidate solutions to different parts of
the problem. In this article, we describe the challenges we have faced in developing the system, our current solutions to
them, and report on our experiences.
2. BACKGROUND
When people involved in making group decisions can only use communication tools like face-to-face meetings,
telephones, and paper-based communications, it is very difficult to have more than a few people deeply engaged in
analysis and decision-making. Even with modestly sized groups of people, groups may experience a range of
inefficiencies and process losses [5]. Group decision support systems (GDSS) have sought to grapple with such problems
[6]. But the size, complexity and open-endedness of decision-making for large-scale social problems such as global
climate change far outstrips the capabilities of traditional technological solutions.
A new generation of social-computational systems (like Wikipedia, Linux, and InnoCentive) may be able to pick up
where decision support technologies have left off. These systems leverage the combined efforts of very large groups of
people to solve complex problems and create large-scale products. Enabled by cheap, fast access to the Internet, these are
often referred to as “collective intelligence” systems [7].
There are many examples of such systems, but there is no clear recipe for their development. Creating a collective
intelligence platform to solve global climate change requires novel solutions to a range of socio-technical design problems.
In our initial platform development, three such challenges have been paramount: predicting the impacts of proposed
solutions, selecting good solutions, and combining solutions. Each of these challenges, and our current solutions to them,
are presented below.
2.1. Designing the CoLab: Challenges and Solutions
The Climate CoLab is a publicly accessible website (http:/climatecolab.org) where members collaborate to create
proposals during contests that address aspects of climate change. Proposals are similar to wiki articles in that they can be
edited on the site and changes reverted. There is also a dedicated page for conversation about each proposal. Proposal
authors have the ability to configure whether any community member can edit the proposal or if editing privileges are
available by invitation only. For some kinds of proposals, it is possible to attach a model run that supports claims made in
the body of the proposal.
In addition to writing proposals, some members of the community play special organizational roles. A team of
Moderators monitor the site for spam and inflammatory or off-topic content. Fellows are students and concerned citizens
who encourage and moderate contest activity. Advisors are experienced professionals who frame the focus of contests,
provide input on proposals, and help to bring contest results to key stakeholders. Finally, the Expert Advisory Board
and broader Expert Council, which include some of the most respected climate change researchers in the world (see [8]
for a list of current members), provide general advice about the project and review specific content on the site.
We have faced a variety of design challenges in developing the CoLab. One challenge is to predict what will happen if a
proposal is implemented. Computer simulations that make such predictions have served as the cornerstone of policy
discussions about climate change, but access to these models is mediated by the experts who run them and interpret their
results. This arrangement puts a bottleneck between stakeholders and the results they require to make decisions. For
example, this bottleneck makes it much more difficult for a group to explore a diverse set of planning possibilities without
active participation of the modeling experts at each step of the way.
Other fields in which modeling and simulation are heavily used for policy making have begun to develop systems that
incorporate access to a model in order to eliminate this bottleneck (e.g. [9] [10]). Yet the field of integrated climate
assessment models, the primary type of tool used to inform policy deliberations about climate change, is large and notable
for divergent assumptions and often conflicting results. Providing non-technical end users with comprehensive access to
these models requires an approach for synthesizing results from multiple models and communicating effectively about
uncertainty and the sources of divergence between models. There are also social and organizational hurdles to enlisting
the help of model authors, which is required if their models are to be made publicly available.
To facilitate a solution to this problem, we have created a web-service called ROMA (Radically Open Modeling
Architecture) that wraps existing simulation models (hosted either externally or internally) in a common API and
publishes a web-accessible endpoint for running them [11]. In addition to providing a layer of abstraction around existing
models, ROMA can transform Microsoft Excel spreadsheets into runnable simulations and also connect models to create
composite models.
The CoLab generates user interfaces for models hosted by ROMA (see Figure 1). Proposal writers use these models to
validate claims about their proposals. One model is currently available to the community. This model takes as input a set
of projected changes in fossil fuel emissions, disaggregated by region, and global deforestation/aforestation, and predicts
changes in atmospheric carbon dioxide concentration, global temperature, sea level, as well as economic costs, economic
damages from climate change, and negative qualitative impacts to human and physical systems such as agriculture and
health. More information about the specific models used can be found on the Climate CoLab web site [12].
Figure 1: The CoLab Modeling Interface
For some kinds of proposals, the existing simulation model cannot provide useful projections of future impacts (e.g.
estimating the impact of geo-engineering on global temperature change). In these cases experts are asked to validate
claims about impacts. In the future, we hope to continue adding models to support a broader range of proposals, and have
engaged leaders in the modeling community to do just this.
Another challenge concerns how to focus the community’s attention on the most promising proposals being explored and,
ultimately, how to select the best ones. The simplest way of doing this is with various forms of on-line voting and rating
(e.g., Reddit, Amazon, Netflix). Sophisticated versions of this include pairwise voting [13], preferential voting [14], and
range voting [15]. But pooling the opinions of many only yeilds good results when the participants have the expertise
required to make informed judgements [16]. Thus, our challenge has been to enable collective decision making that
incorporates the expertise required to identify good, feasible solutions.
Our current approach to this problem is to engage a panel of expert judges to select a set of viable alternative proposals in
each contest that community members then vote upon. Contests unfold as follows:
1. Community members create proposals within the context of a contest, which poses a question about a specific
aspect of the climate change problem.
2. At the end of this first phase, domain experts evaluate proposals for feasibility, novelty, likely impact on climate
change, and presentation quality, and select a set of finalists to move on to the second phase. Judges also provide
comments and critiques for each of the finalists.
3. Proposers then have a period of time to improve their proposals, after which the community and judges vote to
select the most desirable ones.
Awards are given for both those proposals that receive the most votes, and those selected by the judges. This strategy has
successfully increased the feasibility and quality of proposals selected by the community[17].
Maybe list this first? since this must be done before any proposals can be generated for simulation/evaluation?
A third challenge is how to break the problem down into manageable pieces and subsequently weave the solution back
together. Others have focused upon aspects of this problem in relatively constrained, well-behaved domains (e.g. [18–20]),
yet the domain of climate change is vast, dynamic, heavily interconnected across different dimensions, and highly
uncertain. For instance, one dimension of the problem concerns energy production from fossil fuels and the resulting
emission of carbon dioxide.. Yet these concerns cut across geo-political boundaries and are also constrained by social and
economic factors, all of which will change in an uncertain manner over time.
We have sought to use existing knowledge to develop an initial approach to this problem. The climate change community
at large has over time converged upon a functional decomposition of the climate change problem, and this is reflected in
the structure of its various working groups and reports. We have sought to codify this structure in a taxonomy, and to use
this taxonomy to focus the CoLab members on sub-problems. In developing the initial draft of the taxonomy, the Climate
CoLab team relied greatly on the Fourth Assessment Report of the Intergovernmental Panel on Climate Change (IPCC),
especially the contributions of Working Group 3 on mitigation [21] and Working Group 2 on adaptation [22]. The team
then circulated the draft taxonomy to multiple experts, and revised it based on their input.
The current version of our taxonomy defines three dimensions:
• What action is being taken – e.g. improving home efficiency, decarbonizing electric power, persuading
Congress to pass climate mitigation policy, etc.
• Where the action is being focused – e.g. in developed countries, in the United States, in China, etc.
• Who is taking the action – e.g. individual citizens, utilities, any organizations or individuals, etc.
Each of these dimensions is further specified with one or more hierarchies. Figure 2 shows a partial view of the “What”
taxonomy; the complete taxonomy can be found at [23].
Figure 2: Partial view of the "What" dimension of the taxonomy. The view is expanded to the second level.
The taxonomy defines a map of the climate change problem space; one might envision the space as a three-dimensional
matrix (with who, what, and where as axes). Entries in each dimension of the taxonomy are used to specify regions of the
matrix. Not all addressable regions within this space present problems worth solving (e.g. it might not make sense to ask
how church groups can help metro regions cool the ocean to offset climate driven temperature increases), but we believe
most problems worth solving can be located within the space. Because dimensions are organized hierarchically, the
taxonomy contains initial information about how sub-solutions can be integrated to cover larger portions of the problem
space.
The taxonomy is used in two ways in the CoLab. First, each contest (and its associated proposals) is located within a
region of the problem space defined by the taxonomy (see Figure 3). As the site accumulates content, this will serve as a
useful way to index and search through solutions developed by the community.
- What is the impact of the actions on physical systems (engineering, biological, and earth systems)
+ Reduce greenhouse gas (GHG) emissions (Mitigation) + Adapt to climate change (Adaptation) + Reduce the warming effects of GHG emissions (Geoengineering)
- How does the action affect physical systems? + Physical actions: Technologies and practices that address climate
change through their direct impact on physical systems + Social actions: Political, economic, behavioral, or other actions that
influence people to take physical actions that address climate change
Figure 3: The active contest index. Note that each contest is associated with a region of the problem space using
the taxonomy.
The taxonomy can also be used to specify relationships between contests on the site and guide solution integration. For
example, in the currently active contest cycle, an initial set of contests focusing on focused regions of the problem space is
underway. After these contests end, we will launch contests that focus on larger regions of the problem space that contain
the areas of the problem space explored in the initial contests, and community members will then be invited to build
integrated proposals leveraging work done in the initial contests.
3. EXPERIENCE
We have been collecting standard web analytics data via Google Analytics since 12/07/09. As of Sep. 18, 2012, the
CoLab had 43,460 unique visitors from 175 countries. Of these, 3,895 (9%) have registered for the site. Figure 4 depicts
site growth and visits per day.
Figure 4: Membership growth and site visitors over time
Prior to 2012, contests were organized one or two at a time on an annual basis. Large outreach efforts were undertaken to
build participation for these contests, using both social media and traditional channels. Since the site launched, three such
contest cycles have been completed. Site traffic and membership growth have been heavily influenced by these cycles,
with the voting period of each contest driving a large number of visitors and new registrations. Thus contests have served
as a valuable mechanism for building interest and membership.
Closer examination of site activity demonstrates that the initial contests contributed to membership growth, but there was
relatively little substantive activity between contests (Figure 5). As the member base has grown though, we have begun to
see evidence of ongoing site activity that is not closely tied to the contest schedule. Members have continued to develop
proposals, and have had ongoing discussions about the site design, the taxonomy, and new contest topics. We interpret
this as evidence that the community is becoming self-sustaining.
Contest Contest Contest Contest
Figure 5: Categorized site activity. The y-axis reflects individual actions tracked by the system, such as posting
comments, voting, and editing proposals. The "Plan" category captures all work on proposals including text edits,
updating model runs, and managing teams. Data is binned weekly, and the y-axis is capped at 140 total actions in
order to see activity detail (activity spikes to > 2000 actions during the third contest cycle).
Maybe instead of “Plan” use the term “Proposal” in chart? 3.1. CoLab Contest Results
Table 1: Summary of major contest cycles to date.
Dates Contests Phase I proposals Finalists Votes cast
11/1/2009 – 12/9/2009 1 20 n.a. 58
10/1/2010 – 11/26/2010 1 29 4 403
5/16/2011—1/15/2011 2 84 12 2100
8/05/2012— 6 26 -‐-‐ -‐-‐
Table 1 includes a summary of the major contest cycles to date. The first contest began with the official launch of the
Climate CoLab in November 2009. This contest had a single phase and there was no expert involvement. By the end of
the contest, 162 users had registered on the site, and 58 had voted on a plan. The plan receiving the most votes specified
emissions reductions in all regions of the world by 99% by the year 2050. All the experts we consulted with said that it
Contest Contest Contest Contest
would not be feasible to make these reductions (for a range of social, economic and political reasons), and yet the majority
of CoLab voters selected it.
Because of this result, we introduced the phased contest cycle described above. Proposals generated in subsequent
contests have in general been substantially more comprehensive and of higher quality than those in the first contest. For
example, in the first contest, the popular choice plan contained no information about how the proposed emissions
commitments might be achieved. In contrast, the popular choice plan from the second contest contained ten times as many
words and provided both rationale and a range of suggested actions for implementing the plan.
Generally speaking, winning proposals have been diverse, creative, serious contributions from people all over the world.
A complete analysis is not possible here, but a summary of proposal finalists from the 2011 contests (Table 2) offers a
flavor of the kinds of solutions the CoLab community has generated.
Table 2: Proposal finalists from the 2011 Contest Cycle. The two contests focused on the economy at the global
and national levels, respectively. The “proposal pitch” is a brief description provided by authors to describe their
proposals.
Proposal Title Proposal Pitch Author Description Awards
Global Proposals (Finalists)
2010 Winners combined No pitch provided Collaboration of winners
from prior year
Popular Choice
The Planet or your Plate Mitigate climate change by a rapid reduction
of the short lived warming gases by
advocating for less meat consumption
globally
1 activist (US), 1 professor
(US), 1 scientist (Australia)
Popular Choice /
Judges’
Commendation
Rewire Plus: Behaviour
change and value change for
the emerging green
economy
A shift to a green economy will require
changes in behaviours and values all the way
down to the individual. Here's how we get
started!
Sustainability director, Univ.
of Toronto; Recent graduate
Univ. Toronto; CEO of Safara
Sustainability Solutions
National Proposals (Finalists)
Cycling Carbon A pragmatic, ambitious 7-‐point plan to renew
the U.S. economy, enrich its citizens, and lead
the world in fixing the climate.
Software engineer from
North Carolina
Popular Choice
Dream for a Green Future India and dream for a green future : It's time
to make it a reality...
2 University Students at TERI
University, India
Popular Choice
Personal Rapid Transit grids Install connected Personal Rapid Transit grids
over the urban and suburban areas that
house the densest 50% of the US population.
Research scientist (MIT)
Judges’ Choice
Climate proofing the
economics of socially
sustainable small-‐scale
agricultural systems
This proposal climate proofs the economics of
sustainable agriculture in Africa
National Centre for
Technology / Professor,
Obafemi Awolowo
University, Nigeria
there were 4 authors, and
we should indicate that
Judges’ Choice
How to Change US Energy in
One Growing Season
How to Change US Energy in One Growing
Season through practical demonstrations of
available technologies and public education in
all media.
Independent solar activist in
the Boston area
4. CONCLUSION
The Climate CoLab is a novel collective intelligence system designed to help thousands of people around the world work
together to solve the problem of global climate change. It has attracted a continuing stream of visitors and members from
around the world. As the platform and its community have evolved, members have generated proposals of substantial
novelty and increasing quality. The community is now beginning to collaborate on the site outside of our annual contest
cycles. These results are evidence that we are on our way to overcoming a key hurdle in the creation of a collective
intelligence platform—the formation of a large and diverse community collectively engaged in solving a single problem.
The community members who write proposals and engage in discussions on the site are by far the most visible
contributors to the Climate CoLab and are a critical component of the system. But the full scope of human intelligence
woven into the CoLab is far greater. Each of the design solutions described here are heavily dependent upon other human
systems. Radically open modeling makes it possible for end-users to leverage the expertise of many different model
developers, and we rely upon a vibrant community of model developers to provide the embedded technology. Expert-
mediated voting is critically dependent upon the energy and willingness of experts who volunteer their time. The domain
taxonomy is a synthesis of knowledge that many interacting human systems have already created in their efforts to grapple
with climate change.
We believe the Climate CoLab is representative of a general approach to melding human intelligence and social
technology to solve wicked social problems. It is a socio-technical system writ large, that leverages not only the
intelligence of thousands of community members, but also the knowledge and capabilities of many pre-existing human
systems. The platform itself is merely a nexus in which we hope our vast potential collective intelligence may be applied
to solve the problem of climate change.
ACKNOWLEDGEMENTS
We would especially like to thank John Sterman and Hal Abelson of MIT for their support and participation in many
phases of this project. We would also like to thank the following for financial support of this project: the National
Science Foundation, BT plc, Cisco Systems, Argosy Foundation, the MIT Energy Initiative, and the MIT Sloan
Sustainability Initiative. In addition, we are grateful to Stuart Scott and the members of the Climate Summit for their
participation in early outreach efforts, Mark Klein for his advice in developing an interface to enable on-line debates,
Janusz Parfienuik and TopCoder, Inc. for their development work, and our experts, moderators, and other advisors for
volunteering their time to this project.
REFERENCES
[1] C. W. Churchman, “Guest Editorial: Wicked Problems,” Management Science, vol. 14, no. 4, pp. B141–B142, Dec.
1967.
[2] H. W. J. Rittel and M. M. Webber, “Dilemmas in a general theory of planning,” Policy Sciences, vol. 4, no. 2, pp.
155–169, Jun. 1973.
[3] J. Conklin, Dialogue Mapping: Building Shared Understanding of Wicked Problems. John Wiley & Sons, Inc.,
2005.
[4] K. Levin, B. Cashore, S. Bernstein, G. Auld, and D. Student, “Playing it Forward: Path Dependency, Progressive
Incrementalism, and the ‘Super Wicked’ Problem of Global Climate Change,” in International Studies Association
48th Annual Convention, February, 2007, vol. 28.
[5] J. F. Nunamaker, A. R. Dennis, J. S. Valacich, D. Vogel, and J. F. George, “Electronic meeting systems,”
Communications of the ACM, vol. 34, no. 7, pp. 40–61, 1991.
[6] G. DeSanctis and R. B. Gallupe, “A Foundation for the Study of Group Decision Support Systems,” Management
Science, vol. 33, no. 5, pp. 589–609, May 1987.
[7] T. W. Malone, R. Laubacher, and C. Dellarocas, “The Collective Intelligence Genome,” Sloan Management
Review, vol. 51, no. 3, pp. 21–31, Spring 2010.
[8] “Climate CoLab Credits,” 03-Aug-2010. [Online]. Available: http://www.climatecolab.org/web/guest/resources/-
/wiki/Main/Credits. [Accessed: 04-Aug-2010].
[9] M. Matthies, C. Giupponi, and B. Ostendorf, “Environmental decision support systems: Current issues, methods
and tools,” Environmental Modelling & Software, vol. 22, no. 2, pp. 123–127, Feb. 2007.
[10] I. S. Mayer, E. Van Bueren, P. Bots, H. van der Voort, and R. Seijdel, “Collaborative decisionmaking for
sustainable urban renewal projects: a simulation-gaming approach,” Environment and Planning B: planning and
design, vol. 32, no. 3, pp. 403–423, 2005.
[11] J. Introne, R. Laubacher, and T. Malone, “Enabling Open Development Methodologies in Climate Change
Assessment Modeling,” IEEE Software, vol. 28, no. 6, pp. 56–61, Dec. 2011.
[12] “Climate CoLab model help,” 10-Aug-2010. [Online]. Available:
http://www.climatecolab.org/web/guest/resources/-/wiki/Main/Models+Help. [Accessed: 04-Aug-2010].
[13] M. J. Salganik, P. S. Dodds, and D. J. Watts, “Experimental Study of Inequality and Unpredictability in an
Artificial Cultural Market,” Science, vol. 311, no. 5762, pp. 854–856, Feb. 2006.
[14] J. Toplak, “Preferential Voting: Definition and Classification,” in Annual Meeting of the Midwest Political Science
Assication 67th Annual National Conference, Chicago, IL, 2010.
[15] W. D. Smith, “Range Voting.” Dec-2000.
[16] J. Surowiecki, The wisdom of crowds. Anchor, 2005.
[17] J. Introne, R. Laubacher, G. Olson, and T. Malone, “The Climate CoLab: Large scale model-based collaborative
planning,” in 2011 International Conference on Collaboration Technologies and Systems (CTS), 2011, pp. 40–47.
[18] A. P. Kulkarni, M. Can, and B. Hartmann, “Turkomatic: automatic recursive task and workflow design for
mechanical turk,” in Proceedings of the 2011 annual conference extended abstracts on Human factors in computing
systems, New York, NY, USA, 2011, pp. 2053–2058.
[19] A. Kittur, B. Smus, S. Khamkar, and R. E. Kraut, “CrowdForge: crowdsourcing complex work,” in Proceedings of
the 24th annual ACM symposium on User interface software and technology, New York, NY, USA, 2011, pp. 43–
52.
[20] H. Zhang, E. Law, R. Miller, K. Gajos, D. Parkes, and E. Horvitz, “Human computation tasks with global
constraints,” in Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems, New
York, NY, USA, 2012, pp. 217–226.
[21] B. Metz, O. R. Davidson, P. R. Bosch, R. Dave, and L. A. Meyer, Eds., Contribution of Working Group III to the
Fourth Assessment Report of the Intergovernmental Panel on Climate Change, 2007. Cambridge, UK: Cambridge
University Press, 2007.
[22] M. L. Parry, O. F. Canziani, J. P. Palutikof, P. J. van der Linden, and C. E. Hanson, Eds., Contribution of Working
Group II to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change, 2007. Cambridge,
UK: Cambridge University Press, 2007.
[23] R. Laubacher, “What, Where, Who taxonomy--Full view,” The Climate CoLab, 2012. [Online]. Available:
http://climatecolab.org/web/guest/resources/-/wiki/Main/What%2C+Where%2C+Who+taxonomy--Full+view.
[Accessed: 18-Sep-2012].