Crowdsourcing Approaches to Big Data Curation - Rio Big Data Meetup

Download Crowdsourcing Approaches to Big Data Curation - Rio Big Data Meetup

Post on 23-Jul-2015



Data & Analytics

16 download

Embed Size (px)


  • Crowdsourcing Approaches to Big Data Cura5on

    Edward Curry Insight Centre for Data Analy5cs,

    University College Dublin

  • Take Home

    Algorithms Humans Better Data Data

  • Talk Overview

    Part I: Mo4va4on Part II: Data Quality And Data Cura4on Part III: Crowdsourcing Part IV: Case Studies on Crowdsourced Data Cura4on

    Part V: SeBng up a Crowdsourced Data Cura4on Process

    Part VI: Linked Open Data Example Part IIV: Future Research Challenges

  • PART I

  • BIG Big Data Public Private Forum


    Overall objective

    Bringing the necessary stakeholders into a self-sustainable industry-led initiative, which will greatly contribute to

    enhance the EU competitiveness taking full advantage of Big Data technologies.

    Work at technical, business and policy levels, shaping the future through the positioning of IIM and Big Data

    specifically in Horizon 2020.

    BIG Big Data Public Private Forum

  • BIG Big Data Public Private Forum


    Health Public Sector Finance & Insurance

    Telco, Media& Entertainment

    Manufacturing, Retail, Energy,


    Needs Offerings

    Value Chain

    Technical Working Groups

    Industry Driven Sectorial Forums

    Data Acquisition

    Data Analysis

    Data Curation

    Data Storage

    Data Usage

    Structured data Unstructured data Event processing Sensor networks Protocols Real-time Data streams Multimodality

    Stream mining Semantic analysis Machine learning Information extraction

    Linked Data Data discovery Whole world semantics

    Ecosystems Community data analysis

    Cross-sectorial data analysis

    Data Quality Trust / Provenance Annotation Data validation Human-Data Interaction

    Top-down/Bottom-up Community / Crowd Human Computation Curation at scale Incentivisation Automation Interoperability

    In-Memory DBs NoSQL DBs NewSQL DBs Cloud storage Query Interfaces Scalability and Performance

    Data Models Consistency, Availability, Partition-tolerance

    Security and Privacy Standardization

    Decision support Prediction In-use analytics Simulation Exploration Visualisation Modeling Control Domain-specific usage

  • BIG Big Data Public Private Forum


  • BIG Big Data Public Private Forum


    Key Trends Lower usability barrier for data tools Blended human and algorithmic data processing for coping with

    for data quality Leveraging large communities (crowds) Need for semantic standardized data representation Significant increase in use of new data models (i.e. graph)

    (expressivity and flexibility)

    Much of (Big Data) technology is evolving evolutionary

    But business processes change must be revolutionary

    Data variety and verifiability are key opportunities

    Long tail of data variety is a major shift in the data landscape

    The Data Landscape Lack of Business-driven Big

    Data strategies Need for format and data

    storage technology standards Data exchange between

    companies, institutions, individuals, etc.

    Regulations & markets for data access

    Human resources: Lack of skilled data scientists

    Biggest Blockers

    Technical White Papers available on:

  • The Internet of Everything: Connecting the Unconnected

  • Earth Science Systems of Systems

  • Ci5zen Sensors

    humans as ci,zens on the ubiquitous Web, ac,ng as sensors and sharing their observa,ons and views

    Sheth, A. (2009). Ci4zen sensing, social signals, and enriching human experience. Internet Compu,ng, IEEE, 13(4), 87-92.

    Air Pollution

  • Citizens as Sensors


  • The Problems with Data

    Knowledge Workers need: Access to the right data Confidence in that data

    Flawed data effects 25% of critical data in worlds top companies

    Data quality role in recent financial crisis: Asset are defined differently

    in different programs

    Numbers did not always add up

    Departments do not trust each others figures

    Figures not worth the pixels they were made of

  • What is Data Quality?

    Desirable characteristics for information resource Described as a series of quality dimensions: n Discoverability & Accessibility: storing and classifying in

    appropriate and consistent manner

    n Accuracy: Correctly represents the real-world values it models n Consistency: Created and maintained using standardized

    definitions, calculations, terms, and identifiers

    n Provenance & Reputation: Track source & determine reputation Includes the objectivity of the source/producer Is the information unbiased, unprejudiced, and impartial? Or does it come from a reputable but partisan source?

    Wang, R. and D. Strong, Beyond Accuracy: What Data Quality Means to Data Consumers. Journal of Management Information Systems, 1996. 12(4): p. 5-33.

  • Data Quality


    APNR iPod Nano Red 150

    APNS iPod Nano Silver 160

    150 5

    Source A

    Source B

    Schema Difference?

    Data Developer


    iPod Nano




    iPod Nano



    iPod Nano IPN890 150


    Value Conflicts? Entity Duplication?

    Data Steward

    Business Users


    Technical Domain (Technical)


  • What is Data Curation?

    n Digital Curation Selection, preservation, maintenance, collection,

    and archiving of digital assets

    n Data Curation Active management of data over its life-cycle

    n Data Curators Ensure data is trustworthy, discoverable, accessible,

    reusable, and fit for use Museum cataloguers of the Internet age

  • Related Activities

    n Data Governance/ Master Data Management Convergence of data quality, data management,

    business process management, and risk management

    Part of overall data governance strategy for organization

    n Data Curator = Data Steward

    n DO

  • Types of Data Curation

    n Multiple approaches to curate data, no single correct way Who?

    Individual Curators Curation Departments Community-based Curation

    How? Manual Curation (Semi-)Automated Sheer Curation

  • Types of Data Curation Who?

    n Individual Data Curators Suitable for infrequently changing small quantity

    of data (

  • Types of Data Curation Who?

    n Curation Departments Curation experts working with subject matter

    experts to curate data within formal process Can deal with large curation effort (000s of records)

    n Limitations Scalability: Can struggle with large quantities of

    dynamic data (>million records)

    Availability: Post-hoc nature creates delay in curated data availability

  • Types of Data Curation - Who?

    n Community-Based Data Curation Decentralized approach to data curation Crowd-sourcing the curation process

    Leverages community of users to curate data Wisdom of the community (crowd) Can scale to millions of records

  • Types of Data Curation How?

    n Manual Curation Curators directly manipulate data Can tie users up with low-value add activities

    n (Sem-)Automated Curation Algorithms can (semi-)automate curation

    activities such as data cleansing, record duplication and classification

    Can be supervised or approved by human curators

  • Types of Data Curation How?

    n Sheer curation, or Curation at Source Curation activities integrated in normal workflow

    of those creating and managing data

    Can be as simple as vetting or rating the results of a curation algorithm

    Results can be available immediately

    n Blended Approaches: Best of Both Sheer curation + post hoc curation department Allows immediate access to curated data Ensures quality control with expert curation

  • Data Quailty

    Data Curation Example

    Profile Sources

    Define Mappings

    Cleans Enrich

    De-duplicate Define Rules

    Curated Data

    Data Developer

    Data Curator

    Data Governance

    Business Users


    Product Data Product Data

  • Data Curation

    n Pros Can create a single version of truth Standardized information creation and management Improves data quality

    n Cons Significant upfront costs and efforts Participation limited to few (mostly) technical experts Difficult to scale for large data sources

    Extended Enterprise e.g. partner, data vendors Small % of data under management (i.e. CRM, Product, )

  • The New York Times

    100 Years of Expert Data Curation

  • The New York Times

    n Largest metropolitan and third largest newspaper in the United States

    n q Most popular newspaper

    website in US

    n 100 year old curated repository defining its participation in the emerging Web of Data

  • The New York Times

    n Data curation dates back to 1913 Publisher/owner Adolph S. Ochs decided to

    provide a set of additions to the newspaper

    n New York Times Index Organized catalog of articles titles and summaries

    Containing issue, date and column of article Categorized by subject and names Introduced on quarterly then annual basis

    n Transitory content of newspaper became important source of searchable historical data Often used to settle historical debates

  • The New York Times

    n Index Department was created in 1913 Curation and cataloguing of NYT resources

    Since 1851 NYT had low quality index for internal use

    n Developed a comprehensive catalog using a controlled vocabulary Covering subjects, personal names,

    organizations, geographic locations and titles of creative works (books, movies, etc), linked to articles and their summaries

    n Current Index Dept. has ~15 people

  • The New York Times

    n Challenges with consistently and accurately classifying news articles over time Keywords expressing subjects may show some

    variance due to cultural or legal constraints

    Identities of some entities, such as organizations and places, changed over time

    n Controlled vocabulary grew to hundreds of thousands of categories Adding complexity to classification process

  • The New York Times

    n Increased importance of Web drove need to improve categorization of online content

    n Curation carried out by Index Department Library-time (days to weeks) Print edition can handle next-day index

    n Not suitable for real-time online publishing needed a same-day index

  • The New York Times

    n Introduced two stage curation process Editorial staff performed best-effort semi-

    automated sheer curation at point of online pub. Several hundreds journalists

    Index Department follow up with long-term accurate classification and archiving

    n Benefits: Non-expert journalist curators provide instant

    accessibility to online users

    Index Department provides long-term high-quality curation in a trust but verify approach

  • NYT Curation Workflow

    Curation starts with article getting out of the newsroom

  • NYT Curation Workflow

    Member of editorial staff submits article to web-based rule based information extraction system (SAS Teragram)

  • NYT Curation Workflow

    Teragram uses linguistic extraction rules based on subset of Index Depts controlled vocab.

  • NYT Curation Workflow

    Teragram suggests tags based on the Index vocabulary that can potentially describe the content of article

  • NYT Curation Workflow

    Editorial staff member selects terms that best describe the contents and inserts new tags if necessary

  • NYT Curation Workflow

    Reviewed by the taxonomy managers with feedback to editorial staff on classification process

  • NYT Curation Workflow

    Article is published online at

  • NYT Curation Workflow

    At later stage article receives second level curation by Index Dept. additional Index tags and a summary

  • NYT Curation Workflow

    Article is submitted to NYT Index

  • The New York Times

    n Early adopter of Linked Open Data (June 09)

  • The New York Times

    n Linked Open Data @ Subset of 10,000 tags from index vocabulary Dataset of people, organizations & locations

    Complemented by search services to consume data about articles, movies, best sellers, Congress votes, real estate,

    n Benefits Improves traffic by third party data usage Lowers development cost of new applications

    for different verticals inside the website E.g. movies, travel, sports, books


  • Introduction to Crowdsourcing

    n Coordinating a crowd (a large group of workers)to do micro-work (small tasks) that solves problems (that computers or a single user cant)

    n A collection of mechanisms and associated methodologies for scaling and directing crowd activities to achieve goals

    n Related Areas Collective Intelligence Social Computing Human Computation Data Mining

    A. J. Quinn and B. B. Bederson, Human computation: a survey and taxonomy of a growing field, in Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems, 2011, pp. 14031412.

  • When Computers Were Human

    n Maskelyne 1760 Used human computers

    to created almanac of moon positions

    Used for shipping/navigation

    Quality assurance Do calculations twice Compare to third verifier

    D. A. Grier, When Computers Were Human, vol. 13. Princeton University Press, 2005.

  • When Computers Were Human

  • Human Visual perception Visuospatial thinking Audiolinguistic ability Sociocultural


    Creativity Domain knowledge

    Machine Large-scale data


    Collecting and storing large amounts of data

    Efcient data movement Bias-free analysis

    Human vs Machine Affordances

    R. J. Crouser and R. Chang, An affordance-based framework for human computation and human-computer collaboration, IEEE Trans. Vis. Comput. Graph., vol. 18, pp. 28592868, 2012.

  • When to Crowdsource a Task?

    n Computers cannot do the task

    n Single person cannot do the task

    n Work can be split into smaller tasks

  • Platforms and Marketplaces

  • Types of Crowds

    n Internal corporate communities Taps potential of internal workforce Curate competitive enterprise data that will

    remain internal to the company May not always be the case e.g. product technical

    support and marketing data

    n External communities Public crowd-souring market places Pre-competitive communities

  • Generic Architecture


    Platform/Marketplace (Publish Task, Task Management)






  • Mturk Workflow


  • Crowdsourced Data Curation


    DQ Rules & Algorithms

    Entity Linking Data Fusion

    Relation Extraction

    Human Computation

    Relevance Judgment

    Data Verification Disambiguation

    Clean Data Internal Community - Domain Knowledge - High Quality Responses - Trustable

    Web of Data


    Textual Content

    Programmers Managers

    External Crowd - High Availability - Large Scale - Expertise Variety

  • Examples of CDM Tasks

    n Understanding customer sentiment for launch of new product around the world.

    n Implemented 24/7 sentiment analysis system with workers from around the world.

    n 90% accuracy in 95% on content

    n Categorize millions of products on eBays catalog with accurate and complete attributes

    n Combine the crowd with machine learning to create an affordable and flexible catalog quality system

  • Examples of CDM Tasks

    n Natural Language Processing Dialect Identification, Spelling Correction, Machine

    Translation, Word Similarity

    n Computer Vision Image Similarity, Image An...


View more >