Published on March 4, 2014
Shared Rationales in Group Activities 1 Lu Xiao Faculty of Information & Media Studies Department of Computer Science T h e U n i v e r s i t y o f We s t e r n O n t a r i o Http://hii.fims.uwo.ca 14-03-04
Shared Rationales in Group Activities 2 Rationale - the information that justifies one’s ideas, approach, and solution in group activities. Related Studies: Explanations in Knowledge-based Systems (KBS) Types of explanations, Content of explanations, Effects of KBS Explanations (explanation use behavior, learning, perceptions, and judgmental decision making) Shared Information in Group Activities How and why group members share information The factors of information sharing, information pooling phenomenon The information practices and cultures that members develop The effects of shared information and aspects of the shared information (e.g., representation strategy, the use of language) ¡ Influence the change of people’s attitudes ¡ Shared reflections Tools to support information sharing in group activities Group decision support systems for hidden profiles Tools for capturing, archiving, and reusing design rationales Tools to foster reflective thinking in group learning activities 14-03-04
Research Gaps 3 1). the effects of shared rationales in group activities 2). Design requirements to promote the processes of articulating, sharing, and managing rationales in group activities 14-03-04
The Role of Shared Rationales in Group Ideation and Deliberation Activities 4 The effects of rationale awareness in small group ideation activities Large online crowdsourcing ideation activities The effects of shared rationales in Large online deliberation activities Rationale Awareness, as part of Activity Awareness (Carroll et al., 2003, 2005, 2011; Carroll, Rosson, Farooq, & Xiao, 2009), refers to one’s awareness of the other group members’ rationales in a group activity 14-03-04
The Effects of Rationale Awareness in Small Group Ideation Activities 5 The effects of rationale sharing (Xiao, 2011a, 2012; Xiao & Carroll, 2013) • Rationale awareness can contribute to one’s • awareness of others’ knowledge and intellectual contribution; can affect the development of his/her reflection skills • Explicit rationale sharing has potential downsides such as groupthink 14-03-04
The Effects of Shared Rationales in Online Crowdsourcing Ideation Activities 6 Related Work: Quality measure of different means ¡ Providing real-time assessment ¡ Collecting multiple assessment ¡ Analyzing workers’ behavior ¡ Parallel vs. iterative approach 14-03-04
The Effects of Shared Rationales in Large Online Ideation Activities 7 In an ideation task performed through online crowdsourcing processes, whether and how sharing previous workers’ good rationales of their generated ideas affects the ideas’ quality in the task? Pros: awareness of the others’ knowledge and intellectual contributions Cons: problems with explicit rationale sharing 14-03-04
Research Design In an idea evaluation task performed through online crowdsourcing processes, whether and how showing the idea’s rationale affect its evaluation? Hypothesis: Making the ideas’ rationales available to all of its evaluators reduce the variation between evaluations by multiple raters
Research Design Two iterative conditions in the idea generation task: presence vs. absence of previous workers’ rationales Two idea evaluation conditions in the idea evaluation task: presence vs. absence of the idea’s rationale Manipulation of the rationale’s quality: Experiment 1, Experiment 2, and Experiment 3 ¡ ¡ The quality of the ideas and rationales was checked after all the iterations were completed (Experiment 1) The quality of the ideas and rationales was checked at the end of each iteration (Experiment 2 and 3) – stricter and better quality control of ideas and rationales
Research Design (Little et al.,2010) Brainstorming/Idea Generation Task • Six company descriptions • Five names for a company description in each iteration • Six iterations for each company Rating/Idea Evaluation Task Each name had 10 ratings
Manipulation Check 12 14-03-04
Results – Average Quality of the Ideas 13 14-03-04
Results – Best Quality of the Ideas 14 14-03-04
Results: Rationale Awareness in Idea Evaluation
Findings: the Shared Rationales in Online Crowdsourcing Ideation Task In an idea generation task, the awareness of previous workers’ rationales may slightly improve the average quality but NOT the best quality of the generated ideas in iterative approach In an idea evaluation task, the awareness of an idea’s rationale can affect the evaluation outcome and the quality of the rationale may play a significant role on the evaluation (Xiao, CSCW, 2012; Xiao, CI, 2012; Xiao, JASIST, to appear)
The Role of Shared Rationales in Group Ideation and Deliberation Activities 17 The effects of rationale awareness in small group ideation activities Large online crowdsourcing ideation activities The effects of shared rationales in Large online deliberation activities 14-03-04
Shared Rationales in Large Online Deliberation 18 Deliberation concept ¡ Habermas (1989) – public sphere ¡ Halpen and Gibbs (2013) a communication process that involves at least two individuals; that focuses on a social or political issue where the solutions are identifiable by participants; and that values equality among participation and emphasizes rational thinking and logic instead of a power struggle. 14-03-04
Shared Rationales in Large Online Deliberation 19 Wikipedia’s Article for Deletion (AfD) discussions Step 1 : Types of rationales; factors of deliberation outcome What are the types of rationales used in the deliberation? Are there any relationships among the kinds of votes, the article’s topic, the discussion situation (unanimous or non-unanimous), and the final decision? 14-03-04
Possible outcomes of deletion discussions Outcome Keep Withdrawn No consensus Procedural close Delete Speedy delete Userfy Incubate Merge Rename or Move Convert Transwiki Redirect Split Coded as… Explanation Article is kept. Changes may or may not be suggested as part of debate. Keep 20 Nomination is withdrawn. Debate is inconclusive. This might result from disagreement, lack of participation, or other factors. Article defaults to keep. Debate is closed because of problems with the nomination. Article is deleted. Delete Article is deleted under the “speedy” criteria outlined at WP:CSD. Article is deleted but a copy is given to a user to work on as a draft, and may Other be recreated as an article later. As with userfy, but in a communal space rather than related to a single user. Article is deleted but some or all of its content is added into one or more existing articles. Article’s title is changed. Its scope may or may not be amended. Article is converted into another type of page, usually one with a structural function. For example, a list might be changed into a category to be added to the list entries. Article is deleted from English Wikipedia but moved to another Wikimedia project as appropriate – for example, a French article to French Wikipedia or an image gallery to Wikimedia Commons. Article’s content is replaced with a pointer to another page. Article is divided into one or more new pages, or part of the article is moved to another page. 14-03-04
Research Methodology Qualitative Analysis ¡ Open coding process to classify rationales used in deletion debates on three selected dates Quantitative Analysis ¡ Chi Square Tests ¡ Relationships among articles’ topics and deliberation outcomes, discussion situations ¡ Relationship between the SOPA act event and the deliberation outcome SOPA (Stop Online Piracy Act) act event: On January 18, 2012, the English Wikipedia, Google, and an estimate of 7,000 other smaller websites coordinated a service blackout, to raise awareness.
Sample Date selection for qualitative analysis Day # of articles 1 Jun. 2010 89 1 Jun. 2011 73 15 Jan. 2012 67 Total votes for “keep” 127 Total votes for “delete” 280 Total “other” votes (merge, userfy, etc) 37 119 212 23 109 200 63 Date selection for quantitative analysis ¡ Previous sample ¡ 20 dates for chi-square tests that require larger sample size (a priori power analysis) ÷ ÷ Before the SOPA act event: Jan. 1 - 10, 2012 and Nov. 1 - 10, 2011 (N = 1453) After the SOPA act event: Jan. 20 - 29, 2012 and March 20- 29, 2012 (N = 1202)
Findings - Types of rationales; factors of deliberation outcome 23 Rationales are mainly about the articles’ notability (50%) and credibility (12%); Wikipedia policies are often referred to as well (10%) Relationship between the deliberation outcome ÷ and the type of votes: in the case that the decision is delete, there tend to have more delete votes than keep votes, whereas in the case that the decision is keep, the delete votes are not more than keep votes; the votes other than keep and delete significantly affect those decisions that would change the articles’ status. ÷ and the topic of article : articles about people, for-profit organizations, and definitions are slightly more likely to be deleted than expected; articles about locations or events are more likely to be kept than expected; and articles about non-profit organizations and media are more likely to be suggested for other options (e.g., merge, redirect, etc) than expected (Xiao & Askin, JASIST, to appear) 14-03-04
Findings - Types of rationales; factors of deliberation outcome 24 Relationship between the discussion situation ¡ and the type of rationale: more agrees in non-unanimous situations ¡ and the deliberation outcome: in non-unanimous situations, it is more likely to have final decisions as keep or other solutions (e.g., merge, redirect, etc) ¡ and the community participation: ÷ more unique Wikipedia IDs in non-unanimous discussions ÷ More participants in a non-unanimous discussion; the most involved participant was more likely to be recognized in the discussion (Xiao & Askin, JASIST, to appear) 14-03-04
Findings – the Impact of SOPA blackout event on the Deliberation 25 Before the blackout of the site in response to the proposed Stop Online Piracy Act (SOPA) law, there were slightly less keep cases than expected and after the event there were slightly more keep cases than expected. The effect was more significant on the decisions which would change the articles’ status. These articles were more likely to be deleted before the Act, whereas after that Act (or during the discussions about it) they were more likely to be offered suggestions for other options 14-03-04
Shared Rationales in Large Online Deliberation 26 Wikipedia’s Article for Deletion (AfD) discussions Step 1 : Types of rationales; factors of deliberation outcome Step 2: Computational linguistic approaches to extract the rationales 14-03-04
Rationale Extractions for Knowledge Management 27 Motivation – to benefit new editors Approach - extraction of rationales that reflect the needed knowledge on Wikipedia policies in AfD discussions Technique –Illocutionary Act (Searle, 1976) ¡ Representatives ¡ Directives ¡ Commissives ¡ Expressives ¡ Declarations 14-03-04
Rationale Extractions for Knowledge Management 28 Directives (Ervin-Tripp, 1976) ¡ ¡ ¡ ¡ ¡ ¡ Need statements Imperatives Imbedded imperatives Permission directives Question directives Hints 14-03-04
Detect Imperatives 29 1. A verb (in its base form) as the root in the phrase structure and this particular verb has no subject child in the dependency structure. (ROOT (S (INTJ (VB please)) (VP (VB refrain) (PP (IN from) (S (VP (VBG making) (NP (JJ personal) (NNS attacks)))))) (. .))) 2. A personal pronoun or noun (e.g., you, they, username) followed by a modal verb (e.g., should, must) "You must discuss the matter there, and you need to be specific” 14-03-04
Rationale Extractions for Decision-Making Support 30 Motivation – to facilitate efficient final decisionmaking Approach – elimination of redundancy by identifying representative rationales in the discussion Technique – text similarity and sentiment analysis 14-03-04
Rationale Extractions for Decision-Making Support Discussion group similar rationales classify by sentiment polarity Group A positive neutral Group B negative positive neutral Select representative rationales – similarity score, number of users, policies negative
Text Similarity • SEMILAR, a semantic similarity toolkit, was used to compute text similarity Compared the performance of similarity measure among algorithms and human evaluation: Weighted Latent Semantic Analysis (LSA) ¡ Latent Dirichlet Allocation (LDA) ¡
Sentiment Analysis Determine the sentiment polarity of a rationale in our language context (“notable”) ¡ MPQA Subjectivity Lexicon + additional words Data Input Stanford Parser Part-of-speech tagged text Dependency relations Check modified MPQA subjectivity lexicon to obtain the prior polarity (if not in MPQA, marked as ‘nonsentiment’) MPQA format: type=strongsubj len=1 word1=aberration pos1=adj stemmed1=n priorpolarity=negative
Sentiment Analysis • Local negation: A not usually modifies the sentiment word. – “The place is not notable.” • Predicate negation: using verbs with negative polarity. – “I disagree that the place is notable.” • Subject negation: a subject leads to the negation of its predicate. – “Neither one of us agrees that the place is notable.”
Sentiment Analysis Preposition negation: the polarity of the object following the preposition “of” can be changed by the word modified by the preposition. ¡ “It is a viola&on of notability.” Modifier negation: some sentiment word’s polarity can be negated by its modifier. ¡ “The place is of indeterminable notability.”
Sentiment Analysis • Modifier negation – Phrase in the following combination: Noun modified by adjective Noun modified by noun Adjective modified by adverb Adverb modified by adverb Verb modified by verb
Sentiment Analysis Using machine learning methods to determine the polarity of a phrase that has a modifier and a word – Features: • • • • • • l First word token Second word token First word polarity Second word polarity First word part-of-speech Second word part-of-speech Performance of Naïve Bayes, k-nearest neighbor (KNN) and decision tree: ÷ Data: 961 instances (phrases) ÷ Evaluation:10 folds cross validation Naïve Bayes Accuracy (%) K-nearest neighbor Decision Tree 77.94 83.77 80.65
Sentiment Analysis Bottom-up (recursive) algorithm ¡ Based on dependency structure
Sentiment Analysis Evaluation ¡ Data: 236 sentences from discussions in AfD ¡ 3 classes: positive, negative, neutral ¡ Accuracy: 58.47%
The Role of Shared Rationales in Group Ideation and Deliberation Activities 40 The effects of rationale awareness in small group ideation activities Large online crowdsourcing ideation activities The effects of shared rationales in Large online deliberation activities Current research plan To automatic detect rationales from online ideation activities and deliberation activities 14-03-04
Current Research Plan Rhetorical Structure Theory has been recently used to identify justifications in the social Web (Biran, and Rambow, 2011), where the existence of certain discourse structures has been considered argument indicators. Justification is defined as: 1. 2. 3. 4. 5. Recommendation for action, and motivation for proposed action. Statement of like or dislike or of desires and longing, and subjective reason for this like or dislike or desire or longing Statement of like or dislike or of desires and longing, and claimed objective reason for this like or dislike or desire or longing Statement of subjectively perceived fact, with a proposed objective explanation A claimed general objective statement and a more specific objective statement that justifies the more general one Presentational relations from RST Treebank were primarily considered
The Effects of Rationale Awareness in Small Group Ideation Activities 42 Discourse relations in shared rationales in the small group ideation activities (Xiao, 2013): Most used strategies in justifying one’s ideas in the activities were: providing contextual information (circumstance), additional information (elaboration), and evaluation of the information (evaluation) We are extending Biran and Rambow’s (2011) approach by conducting further analysis on these discourse relations and their potential connections to different types of reasoning. 14-03-04
Acknowledgement 43 John M. Carroll, Mary Beth Rosson, Craig Ganoe, Shawn Clark, Steven High, Anabel Quan-Haase, Yan Luo, Tatiana Vashchilko, Trina Joyce Sajo, William Klie, Becky Ellis, Mengshuo Chen, Nicole Askin, Jill Kavanaugh, Lindsay Baker, Achchana Nadarajah, Yumo Yin, Vadim Mazalov, Wanting Mao, Taraneh Khazaei Funding support: NSERC Discovery, NSERC Engage, MITACS, SSHRC, FIMS Internal Funding, SSHRC 4A assistance 14-03-04
XRCE Seminar: Lu Xiao, Assistant Professor, University of Western Ontario, London, Ont., Canada: “Shared rationales in group activities” XRCE Seminar ...
Determine what you want to achieve through the small group activities, ... last used group work in the classroom ... Share your rationale for using group ...
Students who work in groups perform ... Think-pair-share: ... and explaining rationales—all meaningful activities that promote critical thinking and ...
Implementing Shared Services Centers ... I. Rationale ... ated a shared services unit,most activities were trans-
Activities To Promote Speaking ... share ideas about an ... The group members can be either assigned by the teacher or the students may determine ...
Rationale for Five-Day Lesson Plans ... Whole group activities are designed to last 45-60 ... Shared reading occurs at least twice a week and ...
Collaborative Learning . ... Collaborative Activities. Think-pair-share Phillips ... --Create small group activities wherein the students will feel like ...
Shared Reading is an interactive reading experience that occurs when students join in or share the ... The shared reading ... how activities related ...
What is Shared Reading? Shared Reading is exactly what it sounds like. It is a time for sharing a story and reading together (as well as a time of teaching ...