if you build it? benefits and costs of creating your own online community
DESCRIPTION
If You Build It? Benefits and Costs of Creating Your Own Online Community. Loren Terveen Computer Science & Engineering The University of Minnesota August 2011. Background: ways of knowing. Theory Simulation Lab studies Surveys Qualitative studies Build and learn - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/1.jpg)
Loren TerveenComputer Science & Engineering
The University of MinnesotaAugust 2011
If You Build It? Benefits and Costs of Creating Your
Own Online Community
1
![Page 2: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/2.jpg)
TheorySimulationLab studiesSurveysQualitative studiesBuild and learn
(e.g., Google, Facebook, Wikipedia)Build To Learn
Background: ways of knowing
![Page 3: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/3.jpg)
Build to learn
GroupLens Research• Create new interaction /
social computing techniques• Do empirical, quantitative
research• Learn from what we and
others build
![Page 4: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/4.jpg)
DataExperimental Control
To answer the kinds of research questions we like to ask, we need:
![Page 5: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/5.jpg)
1. Learning from others’ data2. Learning from our own data3. Exercising experimental control
The rest of the talk
![Page 6: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/6.jpg)
Q&A systemsWikipedia
1. Learning from others’ data
![Page 7: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/7.jpg)
WP:Clubhouse? An Exploration of Wikipedia’s Gender Imbalance. Lam, S.K., Uduwage, A., Dong, Z., Sen, S., Musicant, D.R., Terveen, L., Riedl, J. WikiSym 2011.
NICE: Social translucence through UI intervention. A. Halfaker, B. Song, D. A. Stuart, A. Kittur and J. Riedl. Wikisym 2011.
Don't bite the Newbies: How Reverts Affect the Quantity and Quality of Wikipedia Work. A. Halfaker, A. Kittur and J. Riedl. Wikisym 2011.
Mentoring in Wikipedia: A Clash of Cultures. D. Musicant, Y. Ren, J. Johnson and J. Riedl. Wikisym 2011.
The Effects of Group Composition on Decision Quality in a Social Production Community, Lam, S.K., Karim, J., Riedl, J. Group 2010.
The Effects of Diversity on Group Productivity and Member Withdrawal in Online Volunteer Groups, Chen, J., Ren, Y., Riedl, J. CHI 2010.
rv you're dumb: Identifying Discarded Work in Wiki Article History, Ekstrand, M.D., Riedl, J.T. Wikisym 2009.
A Jury of Your Peers: Quality, Experience and Ownership in Wikipedia, Halfaker, A., Kittur, N., Kraut, R., Riedl, J. Wikisym 2009.
Is Wikipedia Growing a Longer Tail?, Lam, S.K., Riedl, J. Group 2009. Wikipedians are born, not made: a study of power editors on Wikipedia, Panciera, K., Halfaker, A.,
Terveen, L. Group 2009. SuggestBot: Using Intelligent Task Routing to Help People Find Work in Wikipedia, Cosley, D.,
Frankowski, D., Terveen, L., Riedl, J. IUI 2007. Creating, Destroying, and Restoring Value in Wikipedia, Priedhorsky, R., Chen, J., Lam, S.K.,
Panciera, K., Terveen, L., Riedl, J. Group 2007.
GroupLens Wikipedia Research
![Page 8: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/8.jpg)
WP:Clubhouse? An Exploration of Wikipedia’s Gender Imbalance. Lam, S.K., Uduwage, A., Dong, Z., Sen, S., Musicant, D.R., Terveen, L., Riedl, J.
www.grouplens.org/node/466
![Page 9: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/9.jpg)
9
http://www.nytimes.com/2011/01/31/business/media/31link.html?_r=1&src=busln
A topic generally restricted to teenage girls, like friendship bracelets, can seem short at four paragraphs when compared with lengthy articles on something boys might favor, like, toy soldiers or baseball cards, whose voluminous entry includes a detailed chronological history of the subject.
(BTW, it’s not about the friendship bracelets)
Trigger…
![Page 10: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/10.jpg)
Only 16% of new editors joining Wikipedia during 2009 identified themselves as women
Women made only 9% of the edits by this cohortNew women editors are more likely to stop editing
and leave Wikipedia when their edits are revertedTopics of particular interest to women appear to
get less (and poorer) coverage in Wikipedia
(Hmm… maybe Wikipedia has a low collective IQ!)
Come to Wikisym to get the details!
Findings
![Page 11: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/11.jpg)
2. Learning from our own data
![Page 12: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/12.jpg)
MovieLensCyclopath
GroupLens online communities
![Page 13: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/13.jpg)
![Page 14: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/14.jpg)
200 Union St SE
Lagoon Theatre
![Page 15: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/15.jpg)
![Page 16: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/16.jpg)
How do contributors to open content systems become contributors?
Inspired by…
Research Question
![Page 17: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/17.jpg)
Wikipedians fill different niches than non-Wikipedians
Wikipedians branch out to new areas and topics as they mature
Wikipedians take on more “community work” as they mature
Becoming WikipedianBryant, Forte, & Bruckman 2005
Qualitative study with nine participants self-reporting
![Page 18: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/18.jpg)
Evidence for “becoming”?
Our goal: test these findings quantitatively
Quantity of workQuality of workNature of work
Are Wikipedians Born or Made?
![Page 19: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/19.jpg)
A registered editor with 250+ edits over his/her lifetime
Wikipedian
If editors reach 250 edits within our data set, they are labeled Wikipedian from the
beginning
![Page 20: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/20.jpg)
DataEnglish Wikipedia dump (January 13, 2008)Edits from bots and other non-human means removed
We counted: Only registered editors Wikipedians (users with 250+ edits) - 38K Non-wikipedians - random sample of 38K
Edits per day per editor(“User days”)
(“Day 1”)
![Page 21: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/21.jpg)
Quantity
![Page 22: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/22.jpg)
![Page 23: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/23.jpg)
![Page 24: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/24.jpg)
![Page 25: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/25.jpg)
Quantity
Is a user’s fate sealed?
Born MadeWikipedians are
![Page 26: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/26.jpg)
Measure: Persistent Word Revisions (PWRs)Proportion of words added that persist five
revisions
Quality
![Page 27: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/27.jpg)
![Page 28: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/28.jpg)
![Page 29: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/29.jpg)
![Page 30: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/30.jpg)
Quality
Other quality metrics?
Born MadeWikipedians are
![Page 31: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/31.jpg)
Conjecture: Wikipedians take on community maintenance work over time
Several ways to formalizeEditing in “talk” (and other) namespaces
(Nope: still “born”)Referring to “community norms” (Wikipedia
policies) to explain edits
Nature of Work
![Page 32: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/32.jpg)
![Page 33: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/33.jpg)
![Page 34: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/34.jpg)
![Page 35: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/35.jpg)
Community
Learning norms vs. learning to appeal to the norms?
Training: effective editing
Born MadeWikipedians are
![Page 36: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/36.jpg)
Common pattern: Initial burst of activity, decline, steady state Wikipedians look different from day one Little evidence for “Becoming Wikipedian”: Wikipedians are
born, not made Can we reconcile? This is depressing!
Possible responses: Early interventions Change the culture Systemic initiatives, e.g., APS Wikipedia Initiative: http://
www.psychologicalscience.org/index.php/members/aps-wikipedia-initiative
Accept the reality of the long tail
Summary of findings
![Page 37: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/37.jpg)
We can’t ask Wikipedia users about our interpretations
What if the learning happened before users registered?
But: methodological worries
![Page 38: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/38.jpg)
As of September 2009, we identified:1172 “unambiguous” users
268 of these users made some edits440 “ambiguous” users
For unambiguous usersDay 1 = First time a user came to the site (not
the day they registered)
Cyclopath: viewing and pre-registration activities are visible
![Page 39: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/39.jpg)
Same pattern as for Wikipedia
![Page 40: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/40.jpg)
0
50
100
150
200
250
300
Do Not Edit Do Edit
# of
use
rs
And few users edited before registration
![Page 41: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/41.jpg)
Some viewing before registration
# of users
0100200300400500600700800
0 1-50 51-100 101-250 251-500 501-1000
1001+
A minute or two
<= 5 min. <= 15<= 30 <= 60
![Page 42: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/42.jpg)
But amount of viewing before registration (or before editing) does not predict subsequent behavior
“Born, Not Made” still seems true
![Page 43: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/43.jpg)
FollowupsCyclopath user surveys – Wikisym 2011 paper
Why these patterns?What ‘triggers’ initial contribution?And how might we nurture ongoing
participation?Cyclopath contextual interviews
planned
![Page 44: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/44.jpg)
3. Exercising Experimental Control
![Page 45: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/45.jpg)
Motivating participation: How can we get more work done in open content systems?
Idea: match users with tasks they’re likely to be interested in and capable of doing
Requirements:Introduce tasks matching algorithms/interfacesAssign users to different conditionsGather data necessary for evaluationSurvey users
Research Question
![Page 46: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/46.jpg)
Get work done Nurture new
usersServe community
Goals
Recommender algorithms
Interaction design
ToolsCollective Effort Model
Social Influence
Theory
Intelligent Task
Routing
![Page 47: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/47.jpg)
MovieLens
Task: Edit movie
content
![Page 48: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/48.jpg)
Four strategies to suggest movies to a user
High Pred(individual value of outcomes)
Pick movies the system thinksthe user will really like
Rare Rated(lower effort for a given performance)
Pick movies the user has ratedthat few others have
Needs Work (contribution matters to group)
Pick movies that are missingthe most information
Random(baseline)
Pick random movies
theory-based
![Page 49: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/49.jpg)
Assign ML users to four groups, one per algorithmAbout 2,000 subjects, 200 contributors
Count # editors, contributions, fields
The experiment
![Page 50: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/50.jpg)
Editing behavior by strategy
0
50
100
150
200
250
Number of editors Number of edits Fields filled inMetric
Cou
nt
HighPredRareRated
NeedsWorkRandom
Rare rated: dominantNeeds work: bang for buckRandom: not bad hereHigh prediction: lousy
![Page 51: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/51.jpg)
Task matching workedFamiliarity of user with task was most helpful
Reduces effortIncreases value
Note: we’ve tried this approach in Wikipedia and Cyclopath, tooDifferent issuesGenerality
Summary of findings
![Page 52: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/52.jpg)
MovieLens14 years of continuous developmentSeveral complete software architecture / UI
redos (and another needed!)1 full-time software engineerMuch graduate student time over the years
That’s great, but is there a catch?
![Page 53: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/53.jpg)
~140K lines of code, in multiple languages1 full-time software engineerGrad students: expectation they will spend
25-30% of their time on ‘development’ tasksLooming tasks:
UI redesign / reimplementationExpanding geographic coverage
Cyclopath
![Page 54: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/54.jpg)
Significant resources devoted to developmentBut: typically enables new experiments and/or
builds the user communityAnd: funding for these resources often came only
due to the success of the system/community
Adding it up
Fewer papersBut: papers of a type that would be
impossible otherwiseWe can investigate questions in different
settings, applying different methods: cumulative science
![Page 55: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/55.jpg)
Cycloplan (in collab. with Metropolitan Council)Planners can develop ideas informed by usage data (“What if
I add a trail here?”)Planners can share plans with publicPublic can explore plans, give feedback (“How much would
my route be improved with this trail?”)Public can share concerns directly to relevant officials
Participatory Crowdsourcing (in collab. with IBM)Citizens as sensorsContinua of participation; incentives
Models for participation in open content systemsRoles, privileges, processes: Nupedia vs. WikipediaModels for volunteer participation
Initial vs. ongoing
Towards TMSP
![Page 57: If You Build It? Benefits and Costs of Creating Your Own Online Community](https://reader035.vdocuments.net/reader035/viewer/2022062817/56816932550346895de083d6/html5/thumbnails/57.jpg)
The GroupLens Research Group, particularly:John RiedlJoe KonstanReid PriedhorskyDan CosleyKatie Panciera
And:Tom Erikcson, IBM
Me:[email protected]: @lorenterveen
Thanks to…