how wikipedia really works, and what this means for the nature of "truth" amy bruckman...

How Wikipedia Really Works, and What This Means for the Nature of "Truth"Amy BruckmanAssociate Professor

Great Uncle Oscar’s Wikipedia Page

• Oscar Brodney – Born February 18th, 1907– Hollywood screenwriter– Nominated for an Oscar

• Screenplay for The Glenn Miller Story– Other credits:

• Harvey• Tammy and the Bachelor• Francis the Talking Mule• Abbott & Costello’s Mexican Hayride• Etc.

– I have edited his Wikipedia page• On my watchlist

Outline

• Epistemology overview– The social construction of knowledge

• How reliable is Wikipedia?• Who contributes to Wikipedia & why?• Wikis for writing-to-learn

– Science Online (Andrea Forte)

• Conclusion: a teachable moment

Understanding Wikipedia

• How many people have ever used Wikipedia?

• Have you edited Wikipedia?• Do you have a watch list?• “The problem with Wikipedia is that it only

works in practice. In theory, it can never work.” (New York Times, 4/23/07)

What Makes Wikis Unique?

• English Wikipedia is the #8 most popular website (as of 4/18/11)– Ten Wikipedias have over 500,000 articles

• In order: English, German, French, Italian, Polish, Dutch, Portugese, Russian, Spanish, Japanese

• Constructionist learning (Papert, Resnick)– Learning by working on personally meaningful projects

– Learning through design and construction activities

– Low barrier to entry

– Easy learning curve

– No ceiling

• Extremely light weight– Small differences in accessibility change user behavior

• Collaboration on a large-scale

The Source People Use First

• Thanks to Google’s page rank algorithm, Wikipedia often pops up first

• Source most people use first• Is this a good thing?

Crisis in Epistemology

• People don’t know what to believe• They are asking critical questions about

their sources… sometimes

What is “Truth”?

• Epistemology is “the study or a theory of the nature and grounds of knowledge especially with reference to its limits and validity” (Webster’s Online)

• Spectrum from:– Objectivism

• The world exists and is knowable through our senses.

– Subjectivism• We are limited by our subjective perceptions.

– You may not really be there.

The Sane Middle Ground

• Pragmatic Realism (Hilary Putnam)– The would is only accessible through our

subjective perceptions

– But there’s a strange correlation between our perceptions

– That’s because the world is actually there

How Do We Resolve Conflict about What is True?• Sociology of science (Woolgar, Latour)

says:– Knowledge is socially constructed

– Our best guess at reality is what we agree is true

• The more people agree, the more sure we are

– Peer review expresses this process• Scientists form a knowledge-building community

Wikipedia Supports Knowledge-Building Discourse• How reliable is an article?

– How many people have edited it?

– How many people watch it?

– Spans a spectrum from total trash to much more reliable than refereed journal articles

• Postive example: biography articles of new supreme court nominee, new pope

• Negative example: the Massachusetts State House

Evaluations

• Comparison of Wikipedia and Brittanica by Jim Giles (Nature)– Average Wikipedia article has 4 errors

– Average Brittanica article has 3 errors

• Most vandalism is reverted in second (Viegas et al)

• “Stable versions” in use in German Wikipedia, being tested on English Wikipedia– Changes not visible to unlogged-in users til screened

by a human for obvious vandalism

WikiTrust

• How can users know what is reliable?

• Research by Adler, Chatterjee, de Alfaro, et. al.

• Orange text is less reliable– Prime minister’s name is

changed. “Fjog” is Danish for “fool.”

“Assigning Trust to Wikipedia Content,” Proceedings of WikiSym 2008

Historical Biography

• Historian Roy Rosenzweig compares Wikipedia, Encarta, American National Biography Online (scholarly)– Sample of 52 people in ANBO

• 50% in Wikipedia• 20% in Encarta

– Articles are longer in ANBO• Wikipedia articles are 25% as long as ANBO• Encarta articles are 25% as long as Wikipedia

• Errors:– ANBO: 1; Wikipedia: 4; Encarta: 4Source: “Can History Be Open Source?” by Roy Rosenzweig. The Journal of

American History Volume 93, Number 1 (June, 2006): 117-46. (Available online.)

Differences of Style

• All three show biases in how much coverage each subject gets– Wikipedia on Woodrow Wilson: 3200 words; Isaac

Asimov: 3500

• Compare ANBO and Wikipedia on Lincoln– Both accurate– ANBO has elegant prose, richer contextualization,

confident judgment– Wikipedia’s neutral point of view policy (NPOV) leads

to waffling– Wikipedia is more factualist– Wikipedia has fun, colorful details

Emphasis on References

• Statements without references are deleted quickly

• Online references are checked more closely than print ones

• Loops implying a reference where there is none do happen– Wikipedia info has no cite– Journalist cites Wikipedia article– Someone fixing Wikipedia cites journalist

• Reference syntax is awkward– We created a tool to help: ProveIt

ProveIt

Why Do People Contribute?

• Interviews with 21 “Wikipedians”• Becoming a part of Wikipedia is a process of:

– Legitimate peripheral participation (Lave & Wenger), in a

– Knowledge-building community (Scardamalia & Bereiter)

“Becoming Wikipedian,” Andrea Forte, Susan Bryant et. al. (Group 2005)

Legitimate Peripheral Participation (LPP)

• Lave and Wenger (1991)– Example: tailors in West Africa– Start by sweeping floor

• Legitimate: floor needs to be swept• Peripheral: watching activity around them

– When someone finally says “sew this seam,” they’re ready

• Seen it over and over

• Real-world learning is often more like this than like school

Becoming Wikipedian

• Typical progression:– Start with one edit– Get a watch list– Begin to care about the site as a whole

• LPP in a knowledge-building community• Activity Theory

– Transformation of tools, rules, division of labor, object, community

Power and Authority on Wikipedia• It is NOT a free for all

– How it really works matters

• Interview study with 11 people in administrative roles on Wikipedia– Nature of power and

authority– How conflicts are

resolved

Brett Favre: Profootballreference.com lists Farve as having 8224 career passing attempts, while the official Packers website and NFL.com list him as having 8223. An edit war ensues over the 1 attempt leading to an editor getting indefinitely banned. Sockpuppeting followed, including "aging" accounts to circumvent semi-protection. All over 1 passing attempt... In a 16 year hall of fame career. His name is still spelled weird.

Forte & Bruckman “Scaling Consensus: Increasing Decentralization in Wikipedia Governance” (HICSS 2008)

Increasing Decentralization

• Policy – Creation

• Main policy creation slowing• Moving into WikiProjects

– Interpretation & enforcement• Jimmy --> ArbCom --> Admins• 1785 admins (as of 4/18/11)• Complicated process

– Example: British climatologist William Connelley» Broke rules» Penalty from ArbCom: limited to one revert per day» Penalty not enforced by Admins

• Becoming an Admin– Differs by language– Criteria getting harder

How is Wikipedia Organized?

• Wikipedia governance is accomplished by articulating social norms as policies and guidelines

• Processes for defining and enforcing policies and guidelines are becoming increasingly decentralized

• Wikipedia can’t function as one community—discrete sub-communities have emerged: WikiProjects

Decentralization: WikiProjects

• Group that focuses on a particular content area

• Case Study: WikiProject Military History– Gather subject-matter experts in one place for

networking and collaboration

– Establish subject-specific guidelines

– Support review processes that aren’t possible site-wide

– Encourage contributions • People know their stubs will be nurtured, their articles

protected

– Does not provide mediation for disputes• Lack of local enforcement a potential problem

Deviant Behavior: The Experiences of Banned Users

• Measurements of author contributions suggest that banned Wikipedia users sometimes have a history of good contributions (Adler et. al., 2008)

• What’s going on here? • Ongoing study examines the experiences

of people who have been banned from the site

Emergent Themes

• Perception among banned users is that the experience of editing Wikipedia has changed– Early on, contributing content was sufficient to be a

valued community member– Increasingly, community members’ goals need to

include negotiating complicated social structures – Maintaining social ties is now an important

component of being a Wikipedia contributor

Science Online: Motivation

• What if we created a version of Wikipedia written by high-school students?– Focus is on science

• PhD work of Andrea Forte– MLIS UT Austin

– Assistant Professor, Drexel University

Wiki as a Construction Kit

• Constructing text is a powerful learning activity – Writing-to-learn

(Scardamalia and Bereiter, Emig, Britton, etc.)

• We can design environments that support specific writing activities

• Design challenges

– Support critical citation – media literacy skills

– Make it fit in the classroom

Three Classroom Studies

• AY 2005: Public Policy• AY 2007: Environmental Science (public school)• AY 2008: Biology (private school)• Software improvements to MediaWiki, based on pilot

study findings:– Add support for citations

• MediaWiki mods and Firefox plugin– Teacher tools

Constructionist Learning at Work

Good News & Bad News

• Constructionism predicts precisely the kind of engagement and learning that we saw when students wrote on the wiki– The creation and sharing of a personally meaningful public artifact

as a learning activity leads to deep engagement

• Problematic fit with AP exam“They’re student-made so there could be a lot more information on

the wiki than we actually need to know for the test.” – Sylvia

“The level of thinking that I guess I had them do and work on some of those is probably deeper than the curriculum requires for the assessment.” – Mr. Grant

• Contrast public vs. private school students

Crisis in Epistemology

• How do we know if something is “true”?• One answer: degree of review

– Highly edited Wikipedia article is arguably more reliable than a refereed journal article

• Experts don’t agree• Students are confused• Real answer is multi-dimensional

– How accurate is Wikipedia? Well, which article?

A Teachable Moment

• Role for librarians in bringing clarity• Critical moment where the nature of

proof is at issue– It’s always been an issue– Now everyone is listening

Acknowledgments

• ELC students:– Betsy DiSalvo, Casey Fiesler, Andrea Forte (Drexel

University), Kurt Luther, Sarita Yardi– Jordan Patton, Vanessa Larco

• ELC sponsors:– IBM, Intel, Microsoft, Pitney Bowes, Ricoh – The National Science Foundation– US Department of Education

• For more info:– My research group: http://www.cc.gatech.edu/elc/– ProveIt: http://www.cc.gatech.edu/elc/proveit– My blog: http://nextbison.wordpress.com– Wikimedia Foundation FAQ for librarians:

http://outreach.wikimedia.org/wiki/FAQ_For_Librarians