[ieee 2010 14th international conference on computer supported cooperative work in design (cscwd) -...

5
Large Groups Decision for the Rio 2016 Olympic Games in the MUTIRÕ Project Sergio Palma J. Medeiros (1) [email protected] Gustavo Carvalho (1) [email protected] Jano Moreira de Souza (1) [email protected] Erick Alves Rezende (1) [email protected] (1) COPPE/UFRJ - Computer Science Department, Federal University of Rio de Janeiro cp 68511, cep: 21945-970, Rio de Janeiro, RJ, Brazil Abstract—The historic decision to take the biggest sporting competition in the world to South America was announced by the International Olympic Committee in Copenhagen, Denmark and started several projects in the city of Rio de Janeiro, Brazil. The redevelopment of Praça Mauá Pier, one of a number of improvements planned for the city’s port is the first of a series of enhancements planned for the city in order to prepare it for the so called Rio 2016 event. This sequence of developments will be decided by a Local Committee and should be aided by many people eager to participate and actually collaborate in construction process. To support this process, this paper intends to use decision support systems that employ different approaches in supporting large groups in their problem and solution discussing, alternatives generation and group choosing process. This work presents the use of the LaSca system in the MUTIRÕ project, which explores the coordination in a Large Scale decision support system that sustains the argumentations which were produced by the discussion involving the members of the design team and aids them to achieve a final conclusion. The main task of this project is to help groups interested in the Rio 2016 success taking into account socioeconomics aspects. Keywords-large groups; decision support systems; CSCW; participatory design I. INTRODUCTION Rio de Janeiro will host the 2016 Olympic and Paralympics Games. The investment necessary for the event should be prioritized and concentrated in preventive actions and in minimizing the risks that a event of this magnitude offers to the city. The decisions to be taken should try to solve several problems that could have serious impact in the accomplishment of the event A form of contributing for the occasion is to increase the knowledge on the area, supplying larger support to the people in the administration of their activities. In the attempt of integrating and managing information on the Olympic Games, universities and institutes of research of Rio de Janeiro created several projects, which gathers multidisciplinary studies on the theme as a pioneering initiative in Brazil. The Federal University of Rio de Janeiro started the Mutirõ project with a database that will gather information obtained in all aspects regarding the event and the contributions of all people involved in the Olympic Games. The idea behind the use of large groups to achieve better decisions is precisely to minimize the problems that arise from the aforementioned issues: in such groups, each individual aggregates different knowledge related with the problem domain [1], thus helping to better map the domain of the problem, to diminish the degree of uncertainty, and to deal with the issue of variables in parallel, as the tendency is that individuals will focus their attention on the variables that interests them the most. The main objective of the Mutirõ project is to form a research line on managing ideas and decisions of Rio 2016 community that have different locations, objectives, skills and behavior. The multi-objective analysis realized in this environment present multi-disciplinary characteristics that involve issues not easily modeled. In this area there is also the problem that the structures are partially known, and the potential solutions are controversial. These perspectives demand synergy amongst the users, representing the different areas, competencies, political agendas and social interests. In this Large Group decision environment it is necessary to stand out that the argument of the discussion could be georeferenced. A series of problems inherent to group activities is present in group spatial decision making, as for example, the emphasis given to social activities, failure to define the problem before the choice, diminish of the creativity, feeling of alienation in the group, etc. As an addition to these problems we may also find, at the time of the meeting, disorganization, absence of objectives or agenda, non-conclusion of the solution, absence of individual leadership, lack of efficiency in the decision making and redundancy of solutions. However, with large groups, new issues appear, such as time and space constraints: reunite everyone at the same time at the same place is costly and complicated; it is difficult to find a place where really large groups can gather; these meetings often take a long time; etc. That is, it is difficult to structure 978-1-4244-6763-1/10/$26.00 ©2010 IEEE Proceedings of the 2010 14th International Conference on Computer Supported Cooperative Work in Design 585

Upload: erick-alves

Post on 11-Apr-2017

212 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: [IEEE 2010 14th International Conference on Computer Supported Cooperative Work in Design (CSCWD) - Shanghai, China (2010.04.14-2010.04.16)] The 2010 14th International Conference

Large Groups Decision for the Rio 2016 Olympic Games in the MUTIRÕ Project

Sergio Palma J. Medeiros(1)

[email protected]

Gustavo Carvalho(1)

[email protected]

Jano Moreira de Souza(1)

[email protected]

Erick Alves Rezende(1)

[email protected]

(1)COPPE/UFRJ - Computer Science Department, Federal University of Rio de Janeiro cp 68511, cep: 21945-970, Rio de Janeiro, RJ, Brazil

Abstract—The historic decision to take the biggest sporting competition in the world to South America was announced by the International Olympic Committee in Copenhagen, Denmark and started several projects in the city of Rio de Janeiro, Brazil. The redevelopment of Praça Mauá Pier, one of a number of improvements planned for the city’s port is the first of a series of enhancements planned for the city in order to prepare it for the so called Rio 2016 event. This sequence of developments will be decided by a Local Committee and should be aided by many people eager to participate and actually collaborate in construction process. To support this process, this paper intends to use decision support systems that employ different approaches in supporting large groups in their problem and solution discussing, alternatives generation and group choosing process. This work presents the use of the LaSca system in the MUTIRÕ project, which explores the coordination in a Large Scale decision support system that sustains the argumentations which were produced by the discussion involving the members of the design team and aids them to achieve a final conclusion. The main task of this project is to help groups interested in the Rio 2016 success taking into account socioeconomics aspects.

Keywords-large groups; decision support systems; CSCW; participatory design

I. INTRODUCTION Rio de Janeiro will host the 2016 Olympic and Paralympics

Games. The investment necessary for the event should be prioritized and concentrated in preventive actions and in minimizing the risks that a event of this magnitude offers to the city. The decisions to be taken should try to solve several problems that could have serious impact in the accomplishment of the event A form of contributing for the occasion is to increase the knowledge on the area, supplying larger support to the people in the administration of their activities. In the attempt of integrating and managing information on the Olympic Games, universities and institutes of research of Rio de Janeiro created several projects, which gathers multidisciplinary studies on the theme as a pioneering initiative in Brazil.

The Federal University of Rio de Janeiro started the Mutirõ project with a database that will gather information obtained in all aspects regarding the event and the contributions of all people involved in the Olympic Games. The idea behind the use of large groups to achieve better decisions is precisely to minimize the problems that arise from the aforementioned issues: in such groups, each individual aggregates different knowledge related with the problem domain [1], thus helping to better map the domain of the problem, to diminish the degree of uncertainty, and to deal with the issue of variables in parallel, as the tendency is that individuals will focus their attention on the variables that interests them the most.

The main objective of the Mutirõ project is to form a research line on managing ideas and decisions of Rio 2016 community that have different locations, objectives, skills and behavior. The multi-objective analysis realized in this environment present multi-disciplinary characteristics that involve issues not easily modeled. In this area there is also the problem that the structures are partially known, and the potential solutions are controversial. These perspectives demand synergy amongst the users, representing the different areas, competencies, political agendas and social interests.

In this Large Group decision environment it is necessary to stand out that the argument of the discussion could be georeferenced. A series of problems inherent to group activities is present in group spatial decision making, as for example, the emphasis given to social activities, failure to define the problem before the choice, diminish of the creativity, feeling of alienation in the group, etc. As an addition to these problems we may also find, at the time of the meeting, disorganization, absence of objectives or agenda, non-conclusion of the solution, absence of individual leadership, lack of efficiency in the decision making and redundancy of solutions.

However, with large groups, new issues appear, such as time and space constraints: reunite everyone at the same time at the same place is costly and complicated; it is difficult to find a place where really large groups can gather; these meetings often take a long time; etc. That is, it is difficult to structure

978-1-4244-6763-1/10/$26.00 ©2010 IEEE

Proceedings of the 2010 14th International Conference on Computer Supported Cooperative Work in Design

585

Page 2: [IEEE 2010 14th International Conference on Computer Supported Cooperative Work in Design (CSCWD) - Shanghai, China (2010.04.14-2010.04.16)] The 2010 14th International Conference

and manage such groups, and so the quality of the results obtained are not what they could be.

This paper presents the application of LaSca solution to the MUTIRÕ project which is organized as follows. The second section presents the requirements for collaboration in Large Groups; the third section presents the LaSca functionalities; the fourth section, the LaSca architecture and the enhancements made to the software and the fifth section the final conclusions of this work.

II. COLLABORATIVE OPPORTUNITIES IN THE MUTIRÕ PROJECT

The word Mutirõ comes from Tupi-Guarani language which was spoken by this native community that lived in Brazil before the Portuguese colonization. In such dialect, Mutirõ means a meeting for the community to get together for harvest or a construction. The main idea behind the Mutirõ project is to unite the people and encourage discussion and debate of the important challenges of the Rio 2016 event. The Mutirõ provides materials used by discussion groups to reach informed opinions on the issues and participate in the decision process. The design of the conversation agenda is intended to promote thoughtful dialogue, to bring people together to express their ideas and opinions, and to learn from others judgment process.

A. LaSca: a Large Scale Group Decision Support System LaSca is a large scale group decision support system

[2][3][4] that not only helps its users to structure and perform a decision-making process in large groups dealing with the large groups issues, but also allows them to decide how to decide, or by applying some of the best-known decision theories (such as the Delphi method [5][6][7]), or by combining them, or even by creating new ones. Besides that, with LaSca's "crowd of moderators" concept, the information overload issue (a problem that may occur in really large groups that is becoming more common each day, as more and more people are starting to use services such as Yahoo! Answers or applications that exhibit poll features, for example) can also be dealt with.

The attractiveness of this concept of decision in large groups is increasing a lot, so that even Internet social network services as Orkut, Twitter, or video games like Nintendo Wii show survey appearance. The Yahoo! portal even has the Yahoo! Answers, a service in which someone posts a question, and others may answer it, in replace for points. However, all this decision tools are connected to some fixed means of executing the process, seriously limiting the user's autonomy in this sense. With LaSca (Large Scale), our proposal, then, is to allow the system's users to also decide how to decide, structuring the decision-making process or according to some of the best-known theories of decision-making in large groups, or even combining them, the way they find more convenient. These disciplines above allow users to improve their communication, their coordination and consequently their cooperation.

The LaSca architecture herein proposed allows the integration of complete existent applications with new solutions built for the environment and that a dynamic model of support to the activities of the members of a group. This

architecture projected in four layers assists the necessary specialization for independence of the functionalities without losing the focus of increasing the means of the system use. The advantages of this structure can be seen through the obtained transparency when the introduction of a new entity to the data model or when a new module is included to assist a specific user or a new project need.

All of the premises used in decision making process are registered and available for user search allowing the re-utilization of the experience obtained in a project so it could be reused in similar situations. The capture of the rationale is not only to keep the used information or documents, but also to store all the discussions and the sequence of the work flow executed during the decision process.

The decision process flow flexibility obtained through the Workflow tool permits new features to be used as long as the interface is not changed or that its details of equivalence to the existing interface are entered in the LaSca tool [8][9][10][11]. An extra code level is included to assure the necessary isolation of LaSca system from new functionalities. Types of decision can be deleted and new decision types can be inserted without influence in the actual judgment, as well as all evaluation used can be affected by older ones.

The introduction in the project of concepts of collaboration allows it to enroll when possible, the thought of the members of the group and the reasoning paths that orientated the decisions, as well as the options abandoned so elect solutions could prevail and all the questionings that were generated by the taken decisions. These concepts enlarge the capacity of interaction of the members of the group increasing their synergy and consequently increasing the discussion on the approached themes.

B. Large Groups In his book, James Surowiecki [1] presents ideas that are

very similar to the concepts used in the Delphi method. To him, the group is more intelligent than the individual. However, a group must not be composed only of specialists: a certain degree of heterogeneity is desirable, in order to try to move the result away from a “common place”, which often happens in environments where the decision-makers think in similar ways. This would make it possible to reach decisions of better quality, decisions that would not be reached by homogeneous groups.

Another difference refers to the feedback provided to decision-makers. The author holds that no iterations should happen during the decision-making processes in order to refine the decision, because the exposure to previous results, or the contact with the opinions of more dominant individuals, could cause less secure ones to change their opinions if they are different of the group's, or the dominant individual’s, result. This could produce a lower quality decision.

C. The Mutirõ Project in Rio 2016 The Mutirõ project in Rio 2016 experience can be divided

in three main approaches as the pre-event, during the event phase, and after the event finishes. In each of these phases the

586

Page 3: [IEEE 2010 14th International Conference on Computer Supported Cooperative Work in Design (CSCWD) - Shanghai, China (2010.04.14-2010.04.16)] The 2010 14th International Conference

users can contribute with their collaboration for the improvement of the games. As the tool is being installed users can propose new topics to each phase and wait for other group members to enter in the tool, find the topic, discuss among all, decide in group, and help to implement the solution proposed. There is an additional synergy created by this methodology since all participants feel co responsible for the decision taken.

The topics can also be categorized in classes according to the scope of the problem. These classes can be arranged by the administrator of the LaSca with the aid of some members interested in creating taxonomy for the discussions aspects.

The Security for the participants of the games can be one of these issues since the audience, the athletes, and the coordination staff will be attending not only the competitions but also visiting the city attractions before and after the game hours. Security should be discussed because it is not only a matter of making the arrangements with police, cameras, dogs, police stations, and all kind of resources. People must feel themselves secure, which is a physiological aspect, which can be improved if the elements are exhaustively discussed among all attendees of the competitions.

The Police have information about the places and hours that robbery occurs. This data indicates the position for cameras and Police cars to implement a vigilance system. However the affected population can have a different perspective about where to position these cameras, so they could feel safer. Sa the resources are fixed, where should the cameras be installed? At the places the police proposes based in past statistics or by the indication of the people with their crowd experience. It is important to remark that a people feeling safer leaves their homes more often and goes to shopping, competitions, and all kind of events.

The Transportation means is another essential theme for discussion. Many aspects can be discussed including where should the city build new Underground stations, what should be the buses routes, how should cars be used, should the use of bikes be encouraged, and many other important issues that could be addressed by the group. In this class or in another one the problem of car traffic should be addressed, since additional rented cars and owned ones will be circulating not only in the games routes but all over the town during the tourism activities.

Another important class is the Games itself, since attendees can want to discuss the competition, their favorite athletes, the results, and any aspect regarding to the competition day. These discussions can be used by the administrators to anticipate audience and hours of peak of attendees for means of marketing or even for commercial use. The software could allow users to project interactive sceneries, explore a set of temporal-space data stored in the environment and the related documentation, identifying the properties, classifying the data and projecting sceneries in group. It can also allow users to evaluate the impact of the decisions, specifying its extension, to evaluate categories, to apply simulation models, to consider the impact of a chosen solution, to analyze results, and to generate maps, texts or graphs. The system was built in order to guarantee means of facilitating the negotiation of the group

decision, allowing the development of the decision process, through a participative form among the members of the group.

Several other topics can be explored in discussions like accommodations, hotels, family houses, and any topic related to sleeping and showers. Food is also an important issue, since people will eat not only during the event but also before and after. Places for shopping, tourism spots, environmental impacts of the event, and endless topics can be brought up by group discussions.

III. COMPONENTS OF THE LASCA IMPLEMENTATION The structure described for this system is centered in the

problem of the discussing processes. The architecture implemented consists in adopting tools for a Large Scale Decision environment characteristic [12][13][14]. Within this situation, the discussion shall then meet three main requirements: i) the interaction process coordination, ii) the rationale capture by recording the decision-making and iii) the moderation structure existing within the scope of the activities performed. To meet these requirements the LaSca architecture employs several tools in such a way as to assure the work of the group and the collaboration within the discussion environment.

A. Discussion Management in LaSca The LaSca system allows its users to decide themselves

which is the best way to make a decision, choosing between the features available, or even their combination, and then executing the decision process, once it is started. For example, when revealing an issue to be decided upon, the "Creator" of this "Problem" will choose if it will or will not be possible to track user participation in a particular problem.

In order to cope with the discussion structure, the user can assume one of three independent roles:

1) Participant;

2) Moderator;

3) Creator.

While acting as a Participant, the user can see the Problems already exposed in the system, and click on the "Participate" button next to the Problem the user wants to help with. At the design phase, first, the participant will expose the Attributes he or she thinks that could help to characterize the Problem. Then, the Participant will propose possible Solutions to the Problem. As the number of Attributes and Solutions can be enormous, they have to be moderated. After this happens, the Participant will give weights to the attributes that help describe the Problem. The weights can be relative; absolute, as a grade given to each Attribute; or absolute as a quantity of points the user can distribute among the Attributes. The way of weighting the Attributes is decided by the Creator when the Problem is exposed. Now the Participant can also give an opinion regarding any aspect of this particular decision-making process (that is, if opinions are allowed by the Creator). When all this is done, the design phase is finished.

The Moderator receives the Attributes and Solutions given by the Participants regarding the Problem to be solved, in order

587

Page 4: [IEEE 2010 14th International Conference on Computer Supported Cooperative Work in Design (CSCWD) - Shanghai, China (2010.04.14-2010.04.16)] The 2010 14th International Conference

to filter and eliminate the ones not suitable which is passed to the next level of Moderators (until the Problem's Creator is reached). The first level will receive the Attributes and Solutions directly from the Participants. Otherwise, he will receive the ones that were already analyzed by Moderators from a level immediately bellow.

Finally, to be a Creator, the user has to formulate a new Problem, and publish it, so that users can become Participants of this Problem, if they want to.

B. Hierarchical Moderation The system has to deal with large crowds, and this will

cause a enormous amount of information to be treated not only by the system, but also by Participants, Moderators and Creators. This model of "crowds of moderators" is essential to achieving the best possible decision in the fastest possible way. More Moderators mean faster decision flow, since more information can be categorized, thus accelerating the decision process.

The hierarchy created is also extremely important to speed up the process. The system must be able to balance the requests among all Moderators in order to gain agility provided by the amount of personnel involved. Thus, considering many available Moderators, and disposing them is a tree structure, the amount of work each one would have to do would be better balanced, and this would make the moderating process go on faster.

IV. THE LASCA SYSTEM The LaSca implementation described in this project has one

of its screens shown in the next figure and contemplates the necessary functionality for the user needs. A detailed description of the system screens can be found in the system literature, and in this paper only some architecture aspects are described. All of the stages of the decision process were implemented, and can be used since the users are created in the system. The main components of the physical structure of the environment are detailed in Fig. 1.

The objects used as a base for the system are the Theme, the Creator, the Participant, the Moderator and the Discussion, stored in three main databases called Structure, Problem and Discussion. These objects are structurally linked to form a consistent group of types to represent the current scenario where the decisions shall take place. Each part loses much of its base when isolated from the rest.

A great deal of effort was spent with the system's interface, to make it intuitive to a non specialize d user. At the other side more dedicated users has to feel comfortable with the system, so that they would not get boring. The user interface of LaSca was worked and reworked many times, until a group of people, representing potential users, approved what they used, all this resulting in the interface being now available.

Figure 1. Components of LaSca System

A. Functional Levels of the System The LaSca system assists to several requirements and it

intends to supply the User with tools for Decision Support, Collaboration and Knowledge in a Discussion environment. These disciplines, so heterogeneous, have to be integrated in a single application, which generates great complexity and demands special cares in the construction of the programs and in the interface within several software layers. The main components of the physical structure of the atmosphere are detailed in the illustration above.

The Database composed by three main components is the basis for the system integration. The Structure base is composed by all tables that store the user’s data, the decision themes and their relationships. The database allows a hierarchy with the Moderators to be built to allow the members of the groups to an easier access to the associated components of the collaborative space. The Problem Information block allows basic data update and creation of relationships that links the Problem with other entities of the Collaboration Model. For each decision phase important relationships exist and can be used to build the information net that describes the problem and its possible solutions. Any issue can have a discussion associated and all the argumentation is kept structured and accessible inside the database. A zoom view of the collaborative panel.

B. The Prototype Performance A LaSca prototype (Fig. 2) is being developed using a Web

interface using technologies like Java, TomCat, PostgreSQL, and XML, which shall allow simple interaction for a not specialized user. It is important to note that LaSca follows the Multi-Attribute approach, being single objective.

588

Page 5: [IEEE 2010 14th International Conference on Computer Supported Cooperative Work in Design (CSCWD) - Shanghai, China (2010.04.14-2010.04.16)] The 2010 14th International Conference

Figure 2. Example of a LaSca's Screen

The first system prototype [2], was developed in a peer-to-peer platform, the COPPEER [15], an agent based framework for peer-to-peer applications. A more intuitive interface was then generated [4] as to allow a more consistent and scalable system. This last prototype is now being modified to cope with the Rio 2016 requirements, since the crowd expected is a performance issue to be addressed with care.

The number of users logged simultaneously can vary from a few to thousands and depending on the success of the system, maybe even more. The system has as its mais nonfunctional requirement the scalability and all the architecture has been built in the direction.

V. FINAL CONSIDERATIONS In this paper we presented the requirements and

specifications of a Large Groups Decision Support System for use in Rio 2016 Olympic Games. The LaSca tool produced supplies the collaborative work environment with several collaborative facilities for remote users and, consequently, introduces new interaction perspectives within the interaction arena.

The integration of GGDSS, workflow and Internet concepts [8][9][10][11] into a common architecture allows the interaction and exchange of information among decision-makers, without the need to shift between heterogeneous applications. The LaSca tool aim at helping to establish selection criteria to obtain the solution that will represent the consensus of a group involved in the decision making process.

This system will allow the results of the evaluations to be represented as statistical graphics, textual and/or as cartographic representations.

The topics of deciding in large groups in an Olympic Games event were presented, as well as the difficulties of managing the process. Then were exposed some classes of the issues important for the event and for discussions. At last an initial implementation of our solution, a system oriented to support those processes was presented. With this system, LaSca, it becomes possible to not only build and manage a decision-making process in large groups, but also to decide the way to manage the process itself, following or not the theories presented.

REFERENCES

[1] J. Surowiecki, The Wisdom of Crowds, 2006. [2] G. Carvalho, A. S. Vivacqua, J. M. Souza, and S. P. J. Medeiros, "Large

Scale Decision Making in Participatory Environmental Design," 11th International Conference on Computer Supported Cooperative Work in Design, Melbourne, Australia, April 2007.

[3] G. Carvalho, A. S. Vivacqua, J. M. Souza, and S. P. J. Medeiros, “LaSca: a Large Scale Group Decision Support System,” 12th International Conference on Computer Supported Cooperative Work in Design, Xian, China, April 2008.

[4] G. Carvalho, J. M. Souza, and S. P. J. Medeiros, “Collaboration Engineering, Philosophy, and Democracy with LaSca,” 13th International Conference on Computer Supported Cooperative Work in Design, Santiago, Chile, April 2009.

[5] N. C. Dalkey, "The Delphi Method: An Experimental Study of Group Opinion," Rand Corporation, Santa Monica, CA, USA, 1969.

[6] N. C. Dalkey, and B. Brown, S. Cochran, "The Delphi Method, III: Use of Self Ratings to Improve Group Estimates," Rand Corporation, Santa Monica, CA, USA, 1969.

[7] M. Turoff, and S. R. Hiltz, “Computer Based Delphi Processes,” in Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health, M. Adler, and E. Ziglio, Eds. 1st ed London, England, Jessica Kingsley Publishers, 1996.

[8] J. Barthès, “Computer-Supported Cooperative Work And Knowledge Management,” Proceedings Of Second International Workshop On CSCW In Design, Bangkok, Thailand, November 1997.

[9] M. Klein, "Capturing Geometry Rationale For Collaborative Design", Proceedings Of IEEE 6th International Workshop On Enabling Technologies: Infrastructure For Collaborative Enterprises, MIT, Cambridge, Massachusetts, USA, June 1997.

[10] S. P. J. Medeiros, J. M. Souza, and G. Xexeo, "Fuzzy Petri Nets For Dynamic Workflow In Gis Environment," 20th International Conference On Application And Theory Of Petri Nets, Williamsburg, Virginia, USA, 1999.

[11] J. M. Souza, M.Vaz, and S. P. J. Medeiros, “Workflow Model And Architecture In Gis Environment,” Proceedings Of Second International Workshop On CSCW In Design, Bangkok, Thailand, November 1997.

[12] G. M. Marakas, Decision Support Systems in the 21st Century, 1st ed., Upper Saddle River, NJ, USA. Prentice-Hall, 1999.

[13] E. Turban, and J. E. Aronson, Decision Support Systems and Intelligent Systems, 6th ed., Upper Saddle River, NJ, USA. Prentice Hall, 2001.

[14] H. A. Simon, The New Science of Management Decision, Revised ed., Englewood Cliffs, NJ, USA. Prentice Hall College Div, 1977.

[15] M. G. Miranda, G. Xexéo and J. M. Souza. "Building Tools for Emergent Design with COPPEER," 10th International Conference on Computer Supported Cooperative Work in Design, Nanjing, China, May 2006.

589