common crawl: an open repository of web data
DESCRIPTION
Talk given by Lisa Green from the Common Crawl Foundation at the Hadoop User Group UK meetup on 10 October in LondonTRANSCRIPT
What Does The Data World
Mean to Society?Lisa Green
1 October 2012
London HUG
Lisa Green10 October 2012
Common Crawl : An Open Repository
of Web Data
Photo license: Public Domain Origin: http://en.wikipedia.org/wiki/File:Floppy_disk_2009_G1.jpg
Photo license: CC-BY-SA Origin: http://en.wikipedia.org/wiki/File:Wikimedia_Foundation_Servers-8055_08.jpg
Image license: CC-BY Origin: http://en.wikipedia.org/wiki/File:Internet_map_1024.jpg
Still NascentStill Nascent• Even cheaper storage• Even cheaper compute• Education• Open Data
Still Nascent• Even cheaper storage• Even cheaper compute• Education
Still Nascent• Even cheaper storage• Even cheaper compute
Still Nascent• Even cheaper storage
Image license: CC-BY Credit: NASA, ESA, and the Hubble Heritage Team (STScI/AURA)
Proprietary
Commercial
Gratis
Libre
Progress
Insight
Analysis
Data
Gil Elbaz
Common Crawl Data
• ~8 Billion web pages • ~120 TB• 2008-2012• ARC files, JSON metadata, text files• Available to anyone
ARC Files - Raw Content
Metadata• Status information• HTTP response code• File names & offsets of ARC files• HTML title• HTML meta tags• RSS/Atom information• All anchors/hyperlinks
Text Files - Text Only
http://commoncrawl.org/get-started
http://webdatacommons.org
Change between 2010 and 2012• URLs with embedded data +6%• Microdata +14%• RDFa +26%
• 22% of Web pages contain Facebook URLs• 8% of Web pages implement Open Graph tags
A corpus of anchortext-WikipediaConcept-Count from the CommonCrawl dataset, to benefit
research on WSD, NLP and IR.
Explicit Topic Modeling:Given a concept (represented as a wikipedia page), it can tell what are the most common terms people use to describe the concept.
Given a sentence, it can help identify entities (person, location, organization) in the sentence and map them onto Wikipedia concepts.
http://wikientities.appspot.com
Mapping French websites related to Open Data
Other Use Examples
• Apache Giraph Testing• Maplight• Tineye• Factual• Sentiment Analysis Projects
In Development
• N-gram and Link Graph Extracts• Pig Reader• More Frequent Full Crawls• Focused Subset Crawls at High Frequency• Open Educational Resources
What Does The Data World
Mean to Society?Lisa Green
1 October 2012
Lisa Green
@commoncrawl@boudicca
Thank YouLondon HUG