supporting rationale awareness in large-scale online open participative activities

Supporting Rationale Awareness in Large-Scale Online Open Participative Activities

Lu XiaoAssociate Professor

Faculty of Information & Media Studies, Department of Computer ScienceThe University of Western Ontario

London, ON, [email protected]

1

Outline

• Brief Introduction of my research background

• Rationale and Rationale Awareness – Definition and Related Studies

• Rationale Detection – Our Computational Approach

• Design for Rationale Awareness Support

• Future Work

2

Outline Introduction Rationale Automated Detection of Rationales

Design of Awareness Tools Future Work

About Me…

How to enrich computer-mediated group activities?

3

Information Science

Human-Computer Interaction

(HCI)

Data Science Social Sciences



Between-Team Communication in the Intercultural Context

Collaborative decision-making

1. Xiao, L., & Huang, D. Y. (2015). Between-Team Communication in Cross-Cultural Context, Information, Communication & Society, http://www.tandfonline.com/doi/abs/10.1080/1369118X.2015.1067709#.VcNEDPmniDg

2. Xiao, L., Buchel, O., & Huang, D. Y. (2014). The Role of Team’s Communication Practices in Between-Team Decision Making Activities, Proceedings of iConference 2014

3. Xiao, L., Buchel, O., Martin, J., & Huang, D. Y. (2014), Towards Understanding of Contextual Factors in Information Sharing Behaviors of Inter-Team Activities, 42nd Annual Conference of Canadian Association for Information Science & Inaugural Librarians' Research Institute Symposium



Design-based Parent-Child Mathematics Workshops

1. Xiao, L., Namukasa, I., & Zhang, Y. B. (2016). Design-based Mathematics Workshop, New Library World, to appear2. Xiao, L., Cai, T. T., & Eastmure, V. (2015). Parent-Child Dialogues and Artifact Control Behavior in Computer vs. Non-Computer Mediated

Parental Interactions, Proceedings of iConference 3. Xiao, L., Martin, J. (2012) Supporting Parent-Young Child Activities with Interactive Tabletops: A Conceptual Analysis, In Proceedings of the

ACM 2012 conference on Computer Supported Cooperative Work Companion (CSCW '12)



Curating Testimony: Living Archives of the

Rwandan Diaspora in Canada

1. Xiao, L., Elueze, I., & Kavanaugh, J. R. (2014) Human Rights Researchers' Data Analysis and Management Practices, Proceedings of the 77th Annual Meeting of Association for Information Science and Technology (ASIS&T)

2. Xiao, L., Yan, X. H., Miller, B., & Shrestha, A. (2014). Towards Visual Analytics for Digging into Human Rights Violations Data, Proceedings of the 77th Annual Meeting of Association for Information Science and Technology (ASIS&T)

3. Xiao, L., Elueze, I., & Kavanaugh, J. R. (2014) Human Rights Researchers' Data Analysis and Management Practices, Proceedings of the 77th Annual Meeting of Association for Information Science and Technology (ASIS&T)

4. Xiao, L., Luo, Y., & High, S. (2013). CKM: A Shared Visual Analytical Tool for Large-Scale Analysis of Audio-Video Interviews, at workshop “Big Data and the Humanities” IEEE Big Data 2013

5. Vashchilko, T., & Xiao, L. (2013). Literature review: how content analysis is used to study human rights violations in political science? Proceedings of 2013 iConference

Information Retrieval Pattern Identification

Information SharingCollaboration

Support



About Me…

How to enrich computer-mediated group activities?

7

Information Science

Human-Computer Interaction

(HCI)

Data Science Social Sciences



What is Data Science?

From the large size (Unstructured text) datasets, using scientific (Natural Language Processing + Machine Learning) methods to: 1). identify and extract (useful) data to support the online social activities2). identify language patterns to model online social behavior

Data Pattern

Heterogeneity Size, Storage and Access

SerendipityBatch, real-time

8

Science

Systematic Methods

Knowledge DiscoveryPrediction

ReliabilityValidity



9

Large-Scale Online Open Participative Activities

Opportunities & Challenges: social footprints: users' profile attributes, their social context and ties, and their published content



10

Detect Representative Rationale from Wikipedia’s Article for Deletion (AfD)

Discussions

1. Mao, W. T., Xiao, L., & Mercer, R. (2015). Toward a Knowledge Repository for Wikipedia Article for Deletion (AfD) Discussions, poster at the Third International Symposium of Chinese CHI. ACM, New York, NY, USA., to archive

2. Mao, W. T., Mercer, R., & Xiao, L. (2014a). Extracting Imperatives from Wikipedia Article for Deletion Discussions, First Workshop on Argumentation Mining at the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), http://acl2014.org/acl2014/W14-21/pdf/W14-2117.pdf

3. Mao, W. T., Xiao, L., & Mercer, R. (2014b). Using Text Similarity and Sentiment Analysis to Identify Representative Rationales in Large-Scale Online Deliberations, 5th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis at the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), http://acl2014.org/acl2014/W14-26/pdf/W14-2624.pdf

4. Xiao, L., Mao, W. T., Mercer, R., & Nickerson, J. (under review). A computational approach to recognizing imperatives in online discussions



http://acl2014.org/acl2014/W14-21/pdf/W14-2117.pdf



11

1. Khazaei, T., Xiao, L., & Mercer, R. (2015). “Change My View”: Detecting Persuasive Dialogs in Online Discussions, the 2015 Social Media & Society Conference (Jul. 27 – 29, Toronto, Canada)

2. Xiao, L., Khazaei, T., & Mercer, R. (under review), Writing to persuade: Analysis and Discovery of Persuasive Comments in Online Interactions

Detect Persuasive Texts in “Change My View” Sub-Reddit Discussions



12

Detect Users’ Concern for Privacy in Twitter Data

1. Khazaei, T., Xiao, L., Mercer, R., & Khan, A. (2016), Privacy Behaviour and Profile Configuration in Twitter, In WWW Workshop on Modelling Social Media (MSM '16) (April 11, Montreal, Canada)

2. Khazaei, T., Xiao, L., Mercer, R. (2016). Detecting Privacy Preferences from Online Social Footprints: A Literature Review, 2016 iConference (Philadelphia, PA, USA)

3. Khazaei, T., Xiao, L., Mercer, R., Khan, A. (2015). Social Computing and Intelligence: Exploring Opportunities for the Public and the Enterprise, 2015 IBM CASCON Emerging Technology Track

4. Khazaei, T., Xiao, L., Mercer, R., & Khan, A. (under review), Privacy Preference Inference via Collaborative Filtering



13


Challenge for the Participants: To identify, interpret, and evaluate the others’ ideas and opinions



Rationale: one’s justification of her ideas and opinions

Rationale awareness: one’s awareness of the others’ rationales in the activities

14

Supporting the Participants’ Rationale Awareness



Rationales and Argumentation

15

“justifying a claim is making a communicative move that involves pragmatic as well as semantic constraints” (Bermejo-Luque, 2011)

Luque, L. B. (2011). Giving reasons: A linguistic-pragmatic approach to argumentation theory (Vol. 20). Springer Science & Business Media.



16

Rationale Studies – The Individual Level Focus



Rationale Awareness in the Group Activities

17

Brainstorming activities in small groups and large online crowd contexts1. Xiao, L. (2014). Effects of rationale awareness in online ideation crowdsourcing tasks, Journal of the American Society for Information

Science and Technology (JASIST), 65, 1707-1720, doi: 10.1002/asi.230792. Xiao, L., & Carroll, J. M. (2015). Shared practices in articulating and sharing rationale: An empirical study. International Journal of e-

Collaboration, 11(4), Article 2, 11-393. Xiao, L., & Carroll, J. M. (2013) The Effects of Rationale Awareness on Individual Reflection: Processes in Virtual Group Activities,

International Journal of e-Collaboration, 9(2), 78 - 954. Xiao, L. (2012) The Effects of a Shared Free Form Rationale Space in Collaborative Learning Activities, Journal of Systems and Software ,

86(7), 1727 – 1737



18

Rationale Awareness in the Group Activities

Decision-Making Activities: Wikipedia’s Article for Deletion (AfD) and small group decision-making

1. Xiao, L., & Askin, N. (2014). What Influences Online Deliberation? A Wikipedia Study, Journal of the American Society for Information Science and Technology (JASIST), 65: 898–910

2. Xiao, L., & Askin, N. (2015). Rationale Sharing in Large-Scale Online Deliberations, Proceedings of iConference3. Xiao, L., & Witherspoon, R. (2015). The Effects of Pre-Discussion’s Note-Taking in a Hidden Profile Tasks, Proceedings of the 78th Annual

Meeting of Association for Information Science and Technology (ASIS&T)4. Xiao, L., & Witherspoon, R. (2015). Group Decision Making As Story Construction, Proceedings of the 78th Annual Meeting of Association

for Information Science and Technology (ASIS&T)



19

Knowing the others’ rationales in the group activities is beneficial

Individual• Understanding and evaluating the others’ ideas and perspectives• Knowledge awareness • Contribution awareness• Reflective thinking

Group• Quality monitoring and control



• Authority claim detection (Bender et al., 2011; Marin et al., 2011)

• Opinionated claim detection (Rosenthal & McKeown, 2012)

• Justification identification (Biran and Rambow, 2011)- detect the “common” discourse relations in the rationale texts- Rhetorical Structure Theory (RST)- 12 selected RST relations based on the RST Tree Bank corpus

20

Rationale Detection: Related Work



Biran, O., & Rambow, O. (2011). Identifying justifications in written dialogs by classifying text as argumentative. International Journal of Semantic Computing, 5(04), 363-381.

Rhetorical Structure Theory (RST): a theory of text organization created in the 1980s

[J .1] My first challenge is fairly general for the project, but still very much important.

2-4

Interpretation

Meeting remotely in groups can be very difficult

J oint 3-4

J oint

and having each member meet at the same time is important in

having group synergy. (37)

Purpose

Rhetorical Structure Theory

21



Rationale Detection: A Corpus-based Lexical Cue Graph Model (Khazaei & Xiao, 2015;

Khazaei, Xiao, & Mercer, 2015)Step 1: Identify the RST relations that are commonly present in the rationale texts

22

Three RST relations are reported to be commonly present in the rationales: CIRCUMSTANCE, EVALUATION, and ELABORATION

1. Xiao, L., & Carroll, J. M. (2015). Shared practices in articulating and sharing rationale: An empirical study. International Journal of e-Collaboration, 11(4), Article 2, 11-39

2. Xiao, L. (2013) Do Members Converge to Similar Reasoning Styles in Teamwork? A Study of Shared Rationales in Small Team Activities, Proceedings of 2013 iConference





23

Step 2: Identify the lexical cues for these RST relations based on two corpora

but it has not been sufficiently effective,

CONCESSION

EXPLANATIONEnergetic and concrete action has been taken in Colombia during the past 60 days against the mafiosi of the drug trade, because, unfortunately, it came

too late.

Corpora• RST Discourse Treebank• SFU Review Corpus



24



Step 1: Identify the RST relations that are commonly present in the rationale texts Step 2: Identify the lexical cues for these RST relations based on two corpora

25

Step 3: Use the lexical cues to detect these RST relations so as to detect the rationale texts1. Khazaei, T., & Xiao, L. (2015a), Corpus-based Analysis of Rhetorical Relations: A Study of Lexical Cues, In Proceedings of the

IEEE International Conference on Semantic Computing (ICSC), pp. 417-423, 20152. Khazaei, T., & Xiao, L. (2015). Computational Analysis of Collective Intelligence –Towards Automatic Detection of

Rationales in Online Deliberations, In Proceedings of 2015 Collective Intelligence




26

Step 3: Use the lexical cues to detect these RST relations so as to detect the rationale textsStep 4: Use the probabilistic graph model G (V, E) to disambiguate the cues

V: the set of all possible tokens that may appear in the representations (e.g., POS tags)E: the set of all possible ordered transitions between any two tokens Weight of an edge E (i,j): the probability of a transition from token i to token j

Khazaei, T., Xiao, L., Mercer, R. (2015), Identification and Disambiguation of Lexical Cues of Rhetorical Relations across Different Text Genres, Workshop "Computational Semantics: Linking Lexical, Sentential and Discourse-level Models"(LSDSem) at 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015)



when

NNP

root

advcl advmod


Our approach vs. direct usage of syntax

RST

SFU

Best performance in RST relation labeling in the literature (F-measure): 61.5% on single genre (Ji and Eisenstein, 2014) 27





28

But…

Are these three RST relations commonly present in the rationales independent of the context?

Three RST relations are reported to be commonly present in the rationales: CIRCUMSTANCE, EVALUATION, and ELABORATION (Xiao, 2013; Xiao & Carroll, 2015)



Rutgers Argument Mining Corpora (Wacholder, Muresan, Ghosh, Aakhus, 2014)

Corpora Total No. of Callouts

Total No. of Callouts that contain rationales

android_100 177 76ban_100 138 103iPad_100 234 100

Layoffs_100 200 151Twitter_100 191 97

Overview of the Rutgers Argument Mining Corpora and Our Annotations

Re: #18 You must be a shill for the RIAA and others. Truth is that file sharing in fact has been a boon to truly independent filmmakers and other non affiliated content producers, because they've become better known; it only really harms the big sharks, who have been robbing everyone blind and suppressing the little guy for decades.

Secondly, piracey is illegal. You aren't sharing anything. It's just a eufimisom for STEALING, and that's exactly what it is. It cheats the creator out of their money. 29



RST Annotation in the Rationale Texts (Xiao, under review)

Step 1: Rationale Region Annotation 1) Iterative open-coding process to identify the coding guidelines2) Inter-coder reliability check:

Cohen’s Kappa (the annotation of 347 comments): 70.3%3) Two researchers coded the rest data and examined each other’s results

Step 2: RST Relation Annotation• An experienced RST annotator • Intra-coder reliability check (relation labeling): 70% exact match and 7%

partial match 30



"Common" RST relations

Percentage in Coded RST relations

Justify 19.11%Elaboration 13.31%Conjunction 9.74%Contrast 8.81%Joint 7.73%Evaluation 5.93%Circumstance 4.75%Condition 4.61%Non Volitional Cause

3.81%

Concession 3.22%81.03%

Corpora Percentage in coded RST relations

android 77.81%ban 76.79%ipad 81.82%layoff 84.08%twitter 82.05%

Common RST relations in the callouts that have rationales

31



RST Relations from (Biran & Rambow, 2011)

ContrastNon Volitional CauseConcessionJustifyConjunctionJointEvaluationCircumstanceConditionElaboration

Common RST relations in the callouts that have rationales

ContrastCauseConcessionAntithesisAnalogyConsequenceEvidenceExampleExplanation – argumentationPurposeReasonResult

321. Xiao, L. (under review), Discourse Relations in Rationale-Contained Text Segments



33

Automatic detection of the individuals’ rationales

Data-Centered Study

When and How to present the rationales

Design •Performance measure

•Behavior change

Data-Centered Evaluation

The effects of the rationale awareness

Rationale Studies

Research Methodology: Data-Centered Approach



34

Shared Rationale Space

Shared Rationale

Space

WYSIWIS

1. Xiao, L. (2012) The Effects of a Shared Free Form Rationale Space in Collaborative Learning Activities, Journal of Systems and Software, 86(7), 1727 – 1737

2. Xiao, L. (2011) Design for articulating and sharing rationales in virtual group learning activities, The IASTED International Conference on Technology for Education and Learning (TEL)



35

Shared Rationale Space

Shared Rationale

Space

1. Xiao, L. (2012) The Effects of a Shared Free Form Rationale Space in Collaborative Learning Activities, Journal of Systems and Software, 86(7), 1727 – 1737

2. Xiao, L. (2011) Design for articulating and sharing rationales in virtual group learning activities, The IASTED International Conference on Technology for Education and Learning (TEL)



Future Work• Improvement of the discourse relation detection approach for

identifying RST relations

• Exploration of other rationale indicators

• Design guidelines for rationale awareness tools

36



37


Data Science Opportunities: Social footprints



Challenges• Common Ground• Skill Development (e.g., social

deliberative skill, reflection skill)• Knowledge Management

AcknowledgementColleagues at Penn State and the University of Western Ontario