1. Lu Xiao Assistant Professor The University of Western Ontario 2013 Fostering reflection in About me small group ideation activities online crowdsourcing activities small group deliberation large-scale deliberation 18/09/2013 1

2. Chemical Engineering Information Sciences and Technology Computer Engineering Analytical Problem solving Statistics OOP Algorithm Theory of Computation User-Centered Design Participatory Design User Experience Research Project Management Name: Lu (solar term: , a sign of Autumn) 18/09/2013 2 3. 18/09/2013 4 4. Reflection (Schn, 1985; Boud et al, 1985; Van Manen, 1977) Design to support reflection (Baker & Land, 1997; Kriplean et al., 2012) Reflection in group setting (Yukawa, 2006; Mamykina et al., 2008; Gagnirea, Betrancourta, Dtienneb, 2012) 18/09/2013 5 5. 18/09/2013 6 Articulating and sharing rationales small group ideation activities Articulating and sharing rationales in ideation activities in online crowdsourcing Promoting awareness of group information processing in small group deliberation Extracting and visualizing important information large-scale deliberation 6. Group Project: How to manage a distributed team project for each project phase? Execution ClosureControllingPlanningInitiation Five Phases of a Project Each project phase Identify Challenges Evaluate Collaborative Tools Suggest Best Practices Mini-report 7 Articulating and sharing rationales small group ideation activities 7. Virtual Teamwork (two days) Each group of students needs to answer two questions: Brainstorming Challenges: What are the challenges for the success of distributed teamwork at this phase? Evaluating Challenges: What are the three most and three least important challenges for the success of distributed teamwork at this phase? Document Rationale: Why is it a challenge? Document Rationale: Why those challenges? 18/09/2013 8 8. Execution phase: Challenge Update project management plan Rationale Statement The execution phase relies critically on good management. Therefore it is wise to update the project management plan during the execution phase in order to compensate for any unforeseen discrepancies with the scope, or interpersonal relationships with project team members. This can be challenging because it is necessary to assess all angles of the project's functionality. 18/09/2013 9 9. Execution ClosureControllingPlanningInitiation MR MRMRMRMR Virtual Teamwork (2 days) One and half week Final Presentation and Report (One and half week) Project Duration: nine weeks : Challenges assessment activity MR : The rest course activities for the phase 18/09/2013 10 10. 11 Shared Data Spreadsheet for Posting Ranking Decisions User List Group Folders Group Chat Shared Rationale Space 18/09/2013 11. 12 Survey Background survey Rationale Awareness survey Collaboration experience survey (includes feedback on design of the shared rationale space) Interview Semi-structured interviews Three selected groups Iterative coding process (open coding) 54 codes, 712 quotations Artifacts Shared rationale space Rationale statements (729 statements) Two independent evaluators (Inter-coder agreement: Scotts Pi = 0.68) 18/09/2013 12. The evaluation of the workspace design (Xiao, 2008) The students learning experiences (Xiao et al., 2008) The impact of rationale awareness in the group activities Knowledge awareness and contribution awareness (Xiao, 2011a, 2011b) Practices related to articulating and sharing rationales (Xiao, 2011c) Development of individual reflection skill (Xiao & Carroll, 2013) 18/09/2013 13 13. 18/09/2013 14 14. Rhetorical Structure Theory (RST) pragmatics in the language RST: a theory of text organization created in the 1980s (Mann & Thompson, 1988) 18/09/2013 15 Probably the most extreme case of Visitors Fever I have ever witnessed was a few summers ago when I visited relatives in the Midwest. Probably the most extreme case of Visitors Fever I have ever witnessed was a few summers ago when I visited relatives in the Midwest. relation 15. Rhetorical Structure Theory (RST) http://www.sfu.ca/rst/ Text units as nucleus and satellite Three categories of relations (based on writers intention): subject matter relations: to convey the relation presentational relations: to convince the reader multinuclear relations: do not carry a definite selection of one nucleus 18/09/2013 16 Probably the most extreme case of Visitors Fever I have ever witnessed was a few summers ago --- (nucleus) when I visited relatives in the Midwest. (satellite) relation circumstance 16. 18/09/2013 17http://www.sfu.ca/rst/ Tempting as it may be, we shouldn't embrace every popular issue that comes along. relation concession Tempting as it may be, --satellite we shouldn't embrace every popular issue that comes along. --nucleus Animals heal, but trees compartmentalize. relation contrast An example of presentational relation Animals heal, but trees compartmentalize. An example of multinuclear relation 17. 18/09/2013 18 analyzed twice Explained RST to Research Assistant A Three sets of analysis results 18. 18/09/201319 Three sets of analysis results examined by The principle investigator Identified segments of all the statements 19. 18/09/2013 20 Research Assistant A Identified segments of all the statements Reviewed by Analyzed by Research Assistant A Final result 20. 18/09/2013 21 Team Presentational RST Relations Subject Matter RST Relations Multi-nuclear RST Relations 1 4.6% 87.7% 7.7% 2 3.7% 88.9% 7.4% 3 3% 85.6% 11.4% Table 1. Percentage of Different Types of RST Relations Team No. of RST relations not appeared in the analysis RST relations that were not used in all three team 1 10 Evidence, Justify, Motivation, Restatement, Unconditional, Disjunction, Multinuclear Restatement (7 in total) 2 8 3 11 Table 2. RST relations that did NOT occur in the Rationales 21. 18/09/2013 22 Team RST Relations and their percentages in the coded RST Relations 1 Circumstance(20%) Evaluation (17%) Elaboration (14.4%) 2 Elaboration (21.8%) Evaluation (18%) Circumstance (14%) 3 Evaluation (16.4%) Elaboration (13.7%) Circumstance (13.5%) Table 3. Three Most Occurred RST Relations and Their Percentages 22. 18/09/2013 23 23. 18/09/2013 24 24. The team members shared similar reasoning styles across teams, and Team seems to be an influencing factor on the individuals strategy of using additional information to support ones main point Limitation of the study Exploratory study 18/09/2013 25 25. Research Question: In a brainstorming task performed through human computation processes, whether and how knowing the others rationales of their ideas affects the idea quality of the task? 18/09/2013 26 Articulating and sharing rationales in ideation activities in online crowdsourcing Project II 26. Classification of human computation processes (Quinn and Bederson,2009; Malone et al., 2009; Little et al., 2010) Quality measure of different approaches (Little et al., 2010; Dow et al., 2012) Project II 27. Hypotheses (Kolodner and Schank, 1987; Clark, 1986; Endsley, 1995; Xiao, 2011) 1. Making previous workers rationales of their ideas available to the current worker can improve the quality of the iterative approach 2. Making this information available to all of its evaluators reduce the variation between evaluations by multiple raters Project II 28. Project II 29. Open source software: Java/JavaScript API for running iterative tasks on Mechanical Turk. Project II 30. Brainstorming/Idea Generation Task Six company descriptions Five names for a company description in each iteration Six iterations for each company Rating/Idea Evaluation Task Each name had 10 ratings Project II 31. Three generation conditions: Parallel, Iterative with and without showing previous workers rationales The research validity of submitted HITs for a company was checked at each iteration Brainstorming tasks were posted to Mechanical Turk through several sessions: in one session, each generation condition had two companies; and the condition for a company followed Latin square order Project II 32. Rating condition: Ideas rationale not shown 33. Company No. Brainstorming Condition Parallel approach Showing previously generated names only Showing previously generated names AND rationales 1 6.6 6.2 6 2 6.6 6.9 6.5 3 6.7 6.3 6.3 4 6.6 6.2 6 5 6.8 6.3 6.5 6 6.3 6.9 6.1 Project II 34. There is a statistically significant difference between the ratings: rationale awareness in the rating task improved the names rating. There is a statistically significant difference between the standard deviations of ratings (2.58 vs. 2.41): Reduced the variations of the ratings Project II 35. In an idea generation task, the awareness of previous workers rationales may improve the average quality but NOT the best quality of the generated ideas in iterative approach In an idea evaluation task, the awareness of an ideas rationale can affect the evaluation outcome and the quality of the rationale may play a significant role on the evaluation (Xiao, CSCW, 2012; Xiao, CI, 2012) 36. Deliberation concept Habermas (1989) public sphere Halpen and Gibbs (2013) 18/09/2013 37 a communication process that involves at least two individuals; that focuses on a social or political issue where the solutions are identifiable by participants; and that values equality among participation and emphasizes rational thinking and logic instead of a power struggle. 37. Information Pooling Phenomenon (Stasser & Titus, 1985) 18/09/2013 38 The unique information which needs to be shared and discussed most remained unique (i.e., not shared) or unfamiliar to the group (i.e., shared but did not receive enough attention during the discussion). Lu, L., Yuan, Y. C. and McLeod, P. L. Twenty-Five Years of Hidden Profiles in Group Decision Making. Personality and Social Psychology Review, 16, 1 (February 1, 2012 2012), 54-75. 38. Assumption: with improved awareness of group information processing, it will help promote reflection on deliberation process and outcome help identify hidden profiles in the decision-making process 18/09/2013 39 Project III Promoting awareness of group information processing in small group deliberation 39. Version I: Message Visualization in IM (Xiao, 2012; Xiao & Mazalov, 2012) 18/09/2013 40 Project III Promoting awareness of group information processing in small group deliberation 40. Usability study (Xiao & Haase, 2012) One factor: tool condition Gtalk (no highlighting in the message) Jitsi Jitsi with visualization Two rooms for the experiment Tangram task Director Matcher Usability survey, interview, task performance, chat log 18/09/2013 41 Promoting awareness of group information processing in small group deliberation Project III 41. 18/09/2013 42 strongly disagree (-2), disagree (-1), agree (1), strongly agree (2). Promoting awareness of group information processing in small group deliberation Project III 42. Cognitive Task vs. Experiment Task Describing -- director Clarifying -- director Interpreting -- matcher Matching matcher Correctness whose credit/fault? 18/09/2013 43 Project III 43. Version II: Web-based Deliberation Tool 18/09/2013 44 Project III Promoting awareness of group information processing in small group deliberation 44. 18/09/2013 45 Project IV Wikipedias Article for Deletion (AfD) discussions Research Questions: What are the types of rationales used in the deliberation? Are there any relationships among the articles topic, the kinds of votes, the discussion situation (unanimous or non-unanimous), and the final decision? Extracting and visualizing important information large-scale deliberation 45. Qualitative Analysis Open coding process to classify rationales used in deletion debates on three selected dates Quantitative Analysis Chi Square Tests Relationships among articles topics and deliberation outcomes, discussion situations Relationship between the SOPA act event and the deliberation outcome Articles Topics: biography, definition, event, for-profit, non-profit, location, media, and other 46. Date selection for qualitative analysis Keep, 19 Delete, 6 0 Other, 1 0 01-Jun-10 (89 articles) Keep, 2 0 Delete, 50 Other, 3 01-Jun-11 (73 articles) Keep, 1 4 Delete, 31 Other, 22 15-Jan-12 (67 articles) 47. Date selection for quantitative analysis Previous sample 20 dates for chi-square tests that require larger sample size (a priori power analysis) Before the SOPA act event: Jan. 1st to Jan. 10th, 2012 and Nov. 1st to Nov. 10th, 2011 (N = 1453) After the SOPA act event: Jan. 20th, 2012 to Jan. 29th, 2012 and March 20th to March 29th, 2012 (N = 1202). 18/09/2013 48 48. Qualitative Analysis: 1 June 2010: 443 unique votes for 89 debates, 813 coded rationales Agree 6% Disagree 1% Credibility 15% Notability 54% Policy 10% Precedent 2% Procedural 4% Richness 4% Utility 3% None 1% 1 June 2010 1 June 2011 15 January 2012 Agree 50 34 29 Credibility 119 75 66 Disagree 5 5 4 None 7 6 12 Notability 436 314 305 Policy 79 71 56 Precedent 19 23 40 Procedural 36 50 46 Richness 33 43 93 Utility 29 15 11 49. Quantitative Analysis Relationship between the type of votes and the deliberation outcome In the case that the decision is delete, there tend to have more delete votes than keep votes, whereas in the case that the decision is keep, the delete votes are not more than keep votes. The votes other than keep and delete significantly affect those decisions that would change the articles status. 18/09/2013 50 50. Quantitative Analysis Relationship between the articles topic and the deliberation outcome In AfD, articles about people, for-profit organizations, and definitions are slightly more likely to be deleted than expected, while articles about locations or events are more likely to be kept than expected, and articles about non-profit organizations and media are more likely to be suggested for other options (e.g., merge, redirect, etc) than expected. 18/09/2013 51 51. Quantitative Analysis Relationship between the articles topic and the deliberation situation and outcome 18/09/2013 52 Articles Topic More likely outcome than expected People, for-profit org., definition delete Location, events Keep Not-for-profit, media, definition Other (merge, redirect, etc) Articles Topic More likely discussion situation than expected People, for-profit Unanimous Not-for-profit, location, event Non-unanimous Media, other topics Non-unanimous (slightly more likely) Definition No effect 52. Funding support: NSERC Discovery, NSERC Engage, MITACS, SSHRC, FIMS Internal Funding, SSHRC 4A assistance John M. Carroll, Mary Beth Rosson, Craig Ganoe, Shawn Clark, Anabel Quan-Haase, Tatiana Vashchilko, Trina Joyce Sajo, William Klie, Becky Ellis, Mengshuo Chen, Nicole Askin, Jill Kavanaugh, Lindsay Baker, Achchana Nadarajah, Yumo Yin, Vadim Mazalov 18/09/2013 53 53. Lu Xiao Assistant Professor in: Faculty of Information & Media Studies Computer Science (cross appointment) The University of Western Ontario Director, Human-Computer Interaction Lab: http://hci.fims.uwo.ca Research and Teaching Interests Usability Engineering Participatory Design Collaborative and Social Computing Thank You Lu Xiao lxiao24@uwo.ca the University of Western Ontario 18/09/2013 54